# End-to-end smoke

A walkthrough of every operator-facing surface. Each section is
self-contained — run them in any order. Good as a release smoke test
(mark each section ✓ / ✗ in your release notes) and as a guided tour.

## Nomenclature

| Term | What it means |
| --- | --- |
| Provider | A recipe for reaching compute — an SSH config, an AWS region + credentials, a Slurm cluster. |
| Instance | One live cloud resource (EC2 VM, GCE VM, Kube pod) launched from a Provider. |
| Daemon | A `nymph` process registered with the control plane. One Worker row. |
| Mode | A named RunConfig — runner, image, env, resources. |
| Queue | A named pipe of pending Invocations that daemons subscribe to. |
| Invocation | One unit of work dispatched to a daemon. |
| ExecJob | An ad-hoc shell command run on a daemon, outside the Invocation path. |

## 1. Provider lifecycle

```bash
lakeshore providers add my-test --launcher SSH \
  --kwarg host=my-laptop.local --kwarg user=$USER
lakeshore providers list                       # appears
lakeshore providers show my-test               # full config
lakeshore providers update my-test --kwarg port=2222
lakeshore providers show my-test               # port=2222 reflected
lakeshore providers remove my-test             # soft delete
```

**Expect:** every step exits 0; `show` reflects the latest state.

The name is a **positional**, and the flag is `--launcher` — one of
`SSH`, `SLURM`, `EC2`, `GCE`, `Kube`, capitalized exactly like that.
There is no `--name` and no `--type`. `--launcher` is immutable:
supplying it to `providers update` is an error.

## 2. Soft delete, restore, and hide

```bash
lakeshore providers remove my-test         # soft delete
lakeshore providers list                   # not shown
lakeshore providers list --deleted         # shown with [D <ts>]
lakeshore providers list --all             # hidden + deleted

# Restore over HTTP
curl -X POST -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test/restore"

# Hide / unhide over HTTP
curl -X PATCH -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' -d '{"hidden":true}' \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test"
lakeshore providers list --hidden

# Or do all of it interactively
lakeshore providers edit
```

**Expect:** a soft delete renames the row with a ` [deleted <iso>]`
suffix so the `(namespace, name)` unique index stays intact; `restore`
matches both the plain and the tombstoned name. Hidden rows survive but
stay out of default listings. `providers edit` refetches between actions
so the visible state always reflects what just happened.

## 3. Cloud-config auto-detect

Pre: `~/.aws/credentials` and/or a gcloud configuration.

```bash
lakeshore aws discover             # profiles from ~/.aws
lakeshore aws discover --json
lakeshore gcp discover             # gcloud configurations
lakeshore slurm discover           # /etc/slurm/slurm.conf + sinfo
lakeshore kube discover            # kubectl config view
lakeshore docker discover          # docker context ls

lakeshore providers add aws-dev --from-aws-profile sandbox
lakeshore providers add gcp-dev --from-gcloud-config research
lakeshore providers show aws-dev
```

**Expect:** every `discover` is read-only and makes no cloud API calls —
`--json` is their only flag. `--from-aws-profile` implies
`--launcher EC2` and `--from-gcloud-config` implies `--launcher GCE`, so
you do not pass `--launcher` alongside them.

## 4. SSH discover and the interactive picker

Pre: at least two entries in `~/.ssh/config`.

```bash
lakeshore ssh discover                         # table + footer legend
lakeshore ssh discover --filter new            # only entries not on the server
lakeshore ssh discover --show-stripped         # per-entry stripped directives
lakeshore ssh upload                           # interactive multi-select
lakeshore ssh upload my-dev-box --interactive  # picker even with an alias
lakeshore ssh upload --all                     # non-interactive, everything
lakeshore ssh probe my-dev-box --timeout 5
lakeshore ssh status --history 3
```

**Expect:** `--filter` takes `all` | `new` | `existing` and exits 2 on
anything else. Local-only directives are stripped silently, and the
picker groups rows into "new" and "existing".

## 5. Daemon install over SSH

Pre: an SSH alias you can reach, and `LAKESHORE_ADMIN_TOKEN` exported.

```bash
lakeshore daemon install <ssh-alias> --label <name>
lakeshore daemon list                    # <name> appears, state=active
lakeshore daemon show <id>               # version, runners, capabilities
lakeshore daemon list --watch            # live view, 2 s interval
```

**Expect:** the CLI SSHes in, drops the binary and config, and waits for
the Worker row to register. `--no-wait` skips the wait — there is no
positive `--wait` on `install`. `--binary <path>` rsyncs a local nymph
instead of downloading; `--slurm` wraps the launch in an `sbatch`
heredoc.

Note `--keep-alive-s <n>` here (`-1` = pool, `0` = single job, `N` =
seconds), which is spelled `--keep-alive <seconds>` on `daemon launch`
and `keep_alive_s` in YAML.

## 6. Cloud launch

```bash
lakeshore daemon launch --provider my-ec2 --name gpu --count 2 \
  --queue training --runner docker --wait
lakeshore daemon launch-log <id> --tail 50
lakeshore providers instances my-ec2
lakeshore providers instances --all
lakeshore providers terminate my-ec2 i-0abc...
```

**Expect:** `--name gpu --count 2` produces labels `gpu-00`, `gpu-01` and
sets the cloud Name tag to match. `launch-log` fetches the cloud serial
console (EC2 only today). `daemon launch` reads project defaults from
`.lakeshore` / `.lakeshore.local` — pass `--no-config` to ignore them.

## 7. Kill, cleanup, hibernate, reset

```bash
lakeshore daemon kill <id>                        # also terminates the instance
lakeshore daemon kill <id> --no-terminate         # keep the instance
lakeshore daemon kill --prefix gpu- --yes         # bulk by label prefix

lakeshore daemon cleanup --older-than 2h --dry-run
lakeshore daemon cleanup --older-than 2h
lakeshore daemon cleanup --older-than 5d --yes

lakeshore daemon hibernate <id> --until +30m
lakeshore daemon reset <id>
```

**Expect:** `kill` **terminates the cloud instance by default** —
`--no-terminate` is the opt-out. `cleanup` defaults to `24h` and caps at
`168h` (one week); values outside that range are a 400. `--until`
accepts a relative duration (`+30s`, `+5m`, `+1h`, `+3d`) or an ISO-8601
timestamp; a past deadline is a 400. `reset` queues a `reset_backoff`
command so a daemon stuck in exponential backoff retries promptly.

## 8. Queues

```bash
lakeshore queues add smoke --kind fifo --elasticity fixed
lakeshore queues ls
lakeshore queues show smoke
lakeshore queues patch smoke --description "release smoke"
lakeshore queues stats smoke
lakeshore queues events smoke --verbose
lakeshore queues drain smoke
lakeshore queues unarchive smoke
lakeshore queues rm smoke --force
```

**Expect:** `ls` and `rm` are the irregular verbs — `list` and `remove`
do not exist in this group. The `default` queue is auto-created on first
list and refuses both `archive` and `rm` with a 409.

## 9. Jobs

```bash
lakeshore jobs list
lakeshore jobs list --state queued --state running
lakeshore jobs show <id>
lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue smoke
```

**Expect:** a queued invocation flips to `killed`; a **running** one
returns 409 with the claiming worker id, because there is no daemon-side
cancel signal in the wire protocol yet. `bulk-cancel` touches only
`state=queued` rows.

## 10. Exec on a daemon

```bash
# Pinned to one daemon. Flags come BEFORE the daemon arg; no `--`.
lakeshore daemon exec <daemon> nvidia-smi
lakeshore daemon exec --timeout 30 --workdir /tmp <daemon> ls -la

# Queue-routed — the control plane picks an eligible member.
lakeshore exec --queue training 'python -c "print(1+1)"'

# Pipe a local Python script to a remote python3 -
lakeshore run ./train.py --daemon <id> --env SEED=1
```

**Expect:** both exec commands use pass-through options, so CLI flags
must precede the command and no `--` separator is used. `lakeshore exec`
with no `--queue` pops an interactive picker. `lakeshore run` is
Python-only: it pipes the file bytes to a remote `python3 -` and writes
nothing to disk on the daemon.

## 11. Secrets, storage, and code

```bash
cat ~/.aws/keypair.json | lakeshore secrets add aws-lab --kind aws_keypair
lakeshore secrets list
lakeshore secrets show aws-lab                 # metadata only, never plaintext
lakeshore secrets rotate aws-lab --from-file ./new.json

lakeshore storage add smoke-bucket --kind s3 \
  --bucket my-smoke-bucket --region us-east-1 --creds aws-lab
lakeshore storage presign smoke-bucket hello.txt --put
lakeshore storage credentials smoke-bucket --env
lakeshore storage remove smoke-bucket

lakeshore code push --storage code-staging
lakeshore code list
```

**Expect:** `secrets add --kind` is required and accepts `ssh_key`,
`aws_keypair`, `gcp_sa_json`, `opaque`; with no `--from-file`, the
plaintext is read from stdin. Plaintext is AES-256-GCM encrypted and is
never returned by any GET. `code push` defaults to the storage named
`code-staging`.

## 12. Tunnels and mounts (metadata only)

```bash
lakeshore tunnels add ny-lab --kind wireguard --config-file ~/wg/ny-lab.conf
lakeshore tunnels list
lakeshore providers update bos14 --tunnel ny-lab
lakeshore providers update bos14 --tunnel ""
lakeshore tunnels remove ny-lab

lakeshore mounts add data --kind nfs --kwarg server=10.0.0.1 --kwarg path=/data
lakeshore mounts list
lakeshore mounts update data --kwarg options=ro
lakeshore mounts remove data
```

**Expect:** both round-trip cleanly with no side effects on running
daemons. Neither is activated by the runner yet — see
[Tunnels](/get-started/tunnels.md) and [Mounts](/get-started/mounts.md).

## 13. Admin tokens and namespaces

Requires `LAKESHORE_ADMIN_TOKEN`; the group exits 1 without it.

```bash
lakeshore admin tokens create ci-runner
lakeshore admin tokens ls
lakeshore admin tokens revoke <id>

lakeshore admin ns create scratch
lakeshore admin ns ls
lakeshore admin ns rm scratch
```

**Expect:** the minted plaintext (`dlk_…`) is shown exactly once.
`tokens list` ≡ `ls`; `namespace` ≡ `ns`, with `ls` and `rm` aliases.
Revocation is a tombstone — the row stays listable and the auth hook
returns 403.

## 14. Shell completion

```bash
echo 'eval "$(lakeshore completion bash)"' >> ~/.bashrc
exec bash
lakeshore                                   # top-level verbs
lakeshore providers                         # subcommands
lakeshore providers add --from-aws-profile  # AWS profiles from ~/.aws
lakeshore daemon kill                       # live worker ids
lakeshore providers show                    # live provider names
```

**Expect:** `completion` accepts `bash`, `zsh`, or `fish` and exits 2 on
anything else. Server-backed values are cached for 30 s under
`$XDG_CACHE_HOME/lakeshore/`; local lookups (SSH aliases, AWS profiles,
gcloud configs, kube and docker contexts) are instant and uncached.

## 15. Dashboard

```bash
lakeshore tui                       # aliases: dashboard, top
lakeshore tui --tab queues --interval 2
```

**Expect:** `--tab` takes `workers`, `queues`, `invocations`, or
`storage`; an unknown value warns and falls back to `workers`. It exits 2
with a friendly message when stdout is not a TTY or no control plane is
configured.

> **Note:** The CLI defines no `.version()`, so `--version` is rejected as an unknown
> option. Check the installed version with `npm ls -g @dreamlake/lakeshore`.

## Read next

- [CLI](/cli.md) — the full command reference.
- [CLI examples](/cli/examples.md) — runnable snippets per verb group.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — what the daemon does
  between install and kill.
- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — per-launcher configuration.