DreamLake
  1. Get Started
  2. CLI
  3. Python
  4. Nymph
  5. API
  6. Admin

End-to-end smoke

A walkthrough of every operator-facing surface. Each section is self-contained — run them in any order. Good as a release smoke test (mark each section ✓ / ✗ in your release notes) and as a guided tour.

Nomenclature

TermWhat it means
ProviderA recipe for reaching compute — an SSH config, an AWS region + credentials, a Slurm cluster.
InstanceOne live cloud resource (EC2 VM, GCE VM, Kube pod) launched from a Provider.
DaemonA nymph process registered with the control plane. One Worker row.
ModeA named RunConfig — runner, image, env, resources.
QueueA named pipe of pending Invocations that daemons subscribe to.
InvocationOne unit of work dispatched to a daemon.
ExecJobAn ad-hoc shell command run on a daemon, outside the Invocation path.

1. Provider lifecycle

bash
lakeshore providers add my-test --launcher SSH \
  --kwarg host=my-laptop.local --kwarg user=$USER
lakeshore providers list                       # appears
lakeshore providers show my-test               # full config
lakeshore providers update my-test --kwarg port=2222
lakeshore providers show my-test               # port=2222 reflected
lakeshore providers remove my-test             # soft delete

Expect: every step exits 0; show reflects the latest state.

The name is a positional, and the flag is --launcher — one of SSH, SLURM, EC2, GCE, Kube, capitalized exactly like that. There is no --name and no --type. --launcher is immutable: supplying it to providers update is an error.

2. Soft delete, restore, and hide

bash
lakeshore providers remove my-test         # soft delete
lakeshore providers list                   # not shown
lakeshore providers list --deleted         # shown with [D <ts>]
lakeshore providers list --all             # hidden + deleted

# Restore over HTTP
curl -X POST -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test/restore"

# Hide / unhide over HTTP
curl -X PATCH -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' -d '{"hidden":true}' \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test"
lakeshore providers list --hidden

# Or do all of it interactively
lakeshore providers edit

Expect: a soft delete renames the row with a [deleted <iso>] suffix so the (namespace, name) unique index stays intact; restore matches both the plain and the tombstoned name. Hidden rows survive but stay out of default listings. providers edit refetches between actions so the visible state always reflects what just happened.

3. Cloud-config auto-detect

Pre: ~/.aws/credentials and/or a gcloud configuration.

bash
lakeshore aws discover             # profiles from ~/.aws
lakeshore aws discover --json
lakeshore gcp discover             # gcloud configurations
lakeshore slurm discover           # /etc/slurm/slurm.conf + sinfo
lakeshore kube discover            # kubectl config view
lakeshore docker discover          # docker context ls

lakeshore providers add aws-dev --from-aws-profile sandbox
lakeshore providers add gcp-dev --from-gcloud-config research
lakeshore providers show aws-dev

Expect: every discover is read-only and makes no cloud API calls — --json is their only flag. --from-aws-profile implies --launcher EC2 and --from-gcloud-config implies --launcher GCE, so you do not pass --launcher alongside them.

4. SSH discover and the interactive picker

Pre: at least two entries in ~/.ssh/config.

bash
lakeshore ssh discover                         # table + footer legend
lakeshore ssh discover --filter new            # only entries not on the server
lakeshore ssh discover --show-stripped         # per-entry stripped directives
lakeshore ssh upload                           # interactive multi-select
lakeshore ssh upload my-dev-box --interactive  # picker even with an alias
lakeshore ssh upload --all                     # non-interactive, everything
lakeshore ssh probe my-dev-box --timeout 5
lakeshore ssh status --history 3

Expect: --filter takes all | new | existing and exits 2 on anything else. Local-only directives are stripped silently, and the picker groups rows into "new" and "existing".

5. Daemon install over SSH

Pre: an SSH alias you can reach, and LAKESHORE_ADMIN_TOKEN exported.

bash
lakeshore daemon install <ssh-alias> --label <name>
lakeshore daemon list                    # <name> appears, state=active
lakeshore daemon show <id>               # version, runners, capabilities
lakeshore daemon list --watch            # live view, 2 s interval

Expect: the CLI SSHes in, drops the binary and config, and waits for the Worker row to register. --no-wait skips the wait — there is no positive --wait on install. --binary <path> rsyncs a local nymph instead of downloading; --slurm wraps the launch in an sbatch heredoc.

Note --keep-alive-s <n> here (-1 = pool, 0 = single job, N = seconds), which is spelled --keep-alive <seconds> on daemon launch and keep_alive_s in YAML.

6. Cloud launch

bash
lakeshore daemon launch --provider my-ec2 --name gpu --count 2 \
  --queue training --runner docker --wait
lakeshore daemon launch-log <id> --tail 50
lakeshore providers instances my-ec2
lakeshore providers instances --all
lakeshore providers terminate my-ec2 i-0abc...

Expect: --name gpu --count 2 produces labels gpu-00, gpu-01 and sets the cloud Name tag to match. launch-log fetches the cloud serial console (EC2 only today). daemon launch reads project defaults from .lakeshore / .lakeshore.local — pass --no-config to ignore them.

7. Kill, cleanup, hibernate, reset

bash
lakeshore daemon kill <id>                        # also terminates the instance
lakeshore daemon kill <id> --no-terminate         # keep the instance
lakeshore daemon kill --prefix gpu- --yes         # bulk by label prefix

lakeshore daemon cleanup --older-than 2h --dry-run
lakeshore daemon cleanup --older-than 2h
lakeshore daemon cleanup --older-than 5d --yes

lakeshore daemon hibernate <id> --until +30m
lakeshore daemon reset <id>

Expect: kill terminates the cloud instance by default — --no-terminate is the opt-out. cleanup defaults to 24h and caps at 168h (one week); values outside that range are a 400. --until accepts a relative duration (+30s, +5m, +1h, +3d) or an ISO-8601 timestamp; a past deadline is a 400. reset queues a reset_backoff command so a daemon stuck in exponential backoff retries promptly.

8. Queues

bash
lakeshore queues add smoke --kind fifo --elasticity fixed
lakeshore queues ls
lakeshore queues show smoke
lakeshore queues patch smoke --description "release smoke"
lakeshore queues stats smoke
lakeshore queues events smoke --verbose
lakeshore queues drain smoke
lakeshore queues unarchive smoke
lakeshore queues rm smoke --force

Expect: ls and rm are the irregular verbs — list and remove do not exist in this group. The default queue is auto-created on first list and refuses both archive and rm with a 409.

9. Jobs

bash
lakeshore jobs list
lakeshore jobs list --state queued --state running
lakeshore jobs show <id>
lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue smoke

Expect: a queued invocation flips to killed; a running one returns 409 with the claiming worker id, because there is no daemon-side cancel signal in the wire protocol yet. bulk-cancel touches only state=queued rows.

10. Exec on a daemon

bash
# Pinned to one daemon. Flags come BEFORE the daemon arg; no `--`.
lakeshore daemon exec <daemon> nvidia-smi
lakeshore daemon exec --timeout 30 --workdir /tmp <daemon> ls -la

# Queue-routed — the control plane picks an eligible member.
lakeshore exec --queue training 'python -c "print(1+1)"'

# Pipe a local Python script to a remote python3 -
lakeshore run ./train.py --daemon <id> --env SEED=1

Expect: both exec commands use pass-through options, so CLI flags must precede the command and no -- separator is used. lakeshore exec with no --queue pops an interactive picker. lakeshore run is Python-only: it pipes the file bytes to a remote python3 - and writes nothing to disk on the daemon.

11. Secrets, storage, and code

bash
cat ~/.aws/keypair.json | lakeshore secrets add aws-lab --kind aws_keypair
lakeshore secrets list
lakeshore secrets show aws-lab                 # metadata only, never plaintext
lakeshore secrets rotate aws-lab --from-file ./new.json

lakeshore storage add smoke-bucket --kind s3 \
  --bucket my-smoke-bucket --region us-east-1 --creds aws-lab
lakeshore storage presign smoke-bucket hello.txt --put
lakeshore storage credentials smoke-bucket --env
lakeshore storage remove smoke-bucket

lakeshore code push --storage code-staging
lakeshore code list

Expect: secrets add --kind is required and accepts ssh_key, aws_keypair, gcp_sa_json, opaque; with no --from-file, the plaintext is read from stdin. Plaintext is AES-256-GCM encrypted and is never returned by any GET. code push defaults to the storage named code-staging.

12. Tunnels and mounts (metadata only)

bash
lakeshore tunnels add ny-lab --kind wireguard --config-file ~/wg/ny-lab.conf
lakeshore tunnels list
lakeshore providers update bos14 --tunnel ny-lab
lakeshore providers update bos14 --tunnel ""
lakeshore tunnels remove ny-lab

lakeshore mounts add data --kind nfs --kwarg server=10.0.0.1 --kwarg path=/data
lakeshore mounts list
lakeshore mounts update data --kwarg options=ro
lakeshore mounts remove data

Expect: both round-trip cleanly with no side effects on running daemons. Neither is activated by the runner yet — see Tunnels and Mounts.

13. Admin tokens and namespaces

Requires LAKESHORE_ADMIN_TOKEN; the group exits 1 without it.

bash
lakeshore admin tokens create ci-runner
lakeshore admin tokens ls
lakeshore admin tokens revoke <id>

lakeshore admin ns create scratch
lakeshore admin ns ls
lakeshore admin ns rm scratch

Expect: the minted plaintext (dlk_…) is shown exactly once. tokens list ≡ ls; namespace ≡ ns, with ls and rm aliases. Revocation is a tombstone — the row stays listable and the auth hook returns 403.

14. Shell completion

bash
echo 'eval "$(lakeshore completion bash)"' >> ~/.bashrc
exec bash
lakeshore <TAB>                                  # top-level verbs
lakeshore providers <TAB>                        # subcommands
lakeshore providers add --from-aws-profile <TAB> # AWS profiles from ~/.aws
lakeshore daemon kill <TAB>                      # live worker ids
lakeshore providers show <TAB>                   # live provider names

Expect: completion accepts bash, zsh, or fish and exits 2 on anything else. Server-backed values are cached for 30 s under $XDG_CACHE_HOME/lakeshore/; local lookups (SSH aliases, AWS profiles, gcloud configs, kube and docker contexts) are instant and uncached.

15. Dashboard

bash
lakeshore tui                       # aliases: dashboard, top
lakeshore tui --tab queues --interval 2

Expect: --tab takes workers, queues, invocations, or storage; an unknown value warns and falls back to workers. It exits 2 with a friendly message when stdout is not a TTY or no control plane is configured.

There is no `lakeshore --version`

The CLI defines no .version(), so --version is rejected as an unknown option. Check the installed version with npm ls -g @dreamlake/lakeshore.

Read next