CLI examples — Daemons, exec, and run

Part of the CLI examples cookbook. Auth setup lives on Auth + setup.

Daemons

A daemon (nymph) is a worker process on a host, managed through the control plane. Two ways to bring one up:

daemon launch — the control plane provisions a fresh cloud host via a registered provider.
daemon install — you already have SSH to a host; the CLI bootstraps nymph on it.

`daemon` is the fleet, `worker` is this machine

lakeshore daemon ... always goes through the control plane. To run a native Python queue worker on your own box, use lakeshore worker start — covered in its own section below. daemon start --local was removed.

`daemon launch` — provision a host via a provider

--name sets both the worker label and the cloud Name tag (visible in the EC2 / GCE console). Omit it and the label falls back to <provider>-<short-ulid>.

bash

lakeshore daemon launch --provider my-ec2 --name ge-debug-1 --wait

daemon launch reads project defaults from .lakeshore and .lakeshore.local, walking up from cwd and stopping at the git root. --no-config skips that discovery entirely.

`daemon launch --count` — a fleet

Launches N workers in parallel. With --count > 1, --name becomes a prefix: fleet-00, fleet-01, … --concurrency bounds the in-flight launches (default 5, sized to the AWS RunInstances rate limit) and retries on throttle. --wait waits for all of them.

bash

lakeshore daemon launch --provider my-ec2 --name fleet --count 4 --wait
lakeshore daemon launch --provider my-ec2 --name fleet --count 20 --concurrency 8 --wait

`daemon launch` — capacity, tags, runners

--tag, --queue, and --runner are each repeatable, and each one replaces the corresponding list from .lakeshore rather than appending to it. The first --runner becomes the default runner.

bash

lakeshore daemon launch \
  --provider my-ec2 \
  --name h100-1 \
  --tag gpu:h100 \
  --tag region:us-east \
  --runner process \
  --capacity '{"cpu":8,"gpu":1}' \
  --keep-alive 600 \
  --wait

--keep-alive <seconds> is the idle policy: 0 = exit after one job, N = exit after N idle seconds, -1 = pool mode (never idle-exit).

`daemon launch --queue` — bind to a queue

Daemons subscribe to queues by name via Worker.queues: string[]. Pass --queue one or more times at launch; omit it entirely and the daemon lands in the default queue (the empty array).

bash

lakeshore daemon launch --provider my-ec2 --queue training-h100 --name train-01 --wait

lakeshore daemon launch --provider my-ec2 \
  --queue training-h100 --queue eval \
  --name multi-01 --wait

`daemon launch --bootstrap-script` / `--setup-script`

--bootstrap-script runs as root before nymph starts (cloud-init). --setup-script is repeatable and queues post-register setup commands, each with refresh_capabilities set; it appends to the setup_scripts list from .lakeshore. --no-setup clears both sources.

bash

cat > /tmp/bootstrap.sh <<'SH'
#!/usr/bin/env bash
set -eux
apt-get update && apt-get install -y htop
SH
lakeshore daemon launch --provider my-ec2 --name ge-debug-1 \
  --bootstrap-script /tmp/bootstrap.sh --wait

Both flags read the file locally and send its content inline in the launch payload.

`daemon install` — bootstrap on a host you can SSH to

Takes an SSH alias. --label defaults to the alias. --tag and --runner are repeatable (--runner defaults to process). --keep-alive-s defaults to -1 (pool mode). The admin token defaults to $LAKESHORE_ADMIN_TOKEN.

bash

lakeshore daemon install my-server --label ge-debug-1
lakeshore daemon install my-server --tag gpu:a100 --runner docker --keep-alive-s 900

`--no-wait`, not `--wait`

daemon install waits for the Worker row by default and exposes only the negation --no-wait. There is no positive --wait on install (there is on launch).

`daemon install --binary` — rsync a local build

Skip the download and rsync a locally-built nymph binary onto the host. Match the host arch. --nymph-base-url overrides the download source when you're not using --binary.

bash

cargo build --release --target x86_64-unknown-linux-gnu --manifest-path nymph/Cargo.toml
lakeshore daemon install my-server \
  --label ge-debug-1 \
  --binary nymph/target/x86_64-unknown-linux-gnu/release/nymph

`daemon install --slurm` — sbatch instead of setsid+nohup

Wrap the nymph startup in an sbatch heredoc so the daemon runs as a SLURM job. Use on login nodes where a detached background process is discouraged.

bash

lakeshore daemon install login.cluster --slurm --slurm-partition gpu

`daemon list` — what's registered

Every daemon in the namespace, with last-seen time, state, and a queues column (— when empty, i.e. the default queue).

bash

lakeshore daemon list
lakeshore daemon list --json | jq '.[] | {id, label, state, lastSeenAt}'

--watch takes an optional value; bare --watch refreshes every 2 seconds. Ctrl-C exits.

bash

lakeshore daemon list --watch        # every 2s
lakeshore daemon list --watch 5      # every 5s

`daemon show` — one daemon, full record

Same --json / --watch [seconds] treatment.

bash

lakeshore daemon show "$D"
lakeshore daemon show "$D" --json | jq .
lakeshore daemon show "$D" --watch

`daemon launch-log` — cloud serial console

When a launched daemon never reaches its first /hello, grab the cloud serial-console output. EC2-only today.

bash

lakeshore daemon launch-log "$D" --tail 200
lakeshore daemon launch-log "$D" --no-latest --json

`daemon setup` — queue a post-register script

Upload a bash script and run it on a daemon that has already registered. --script is required; the content is uploaded inline. --sudo runs it as root, and --refresh-capabilities makes the daemon re-detect its host capabilities afterwards and publish them on the next poll.

bash

cat > /tmp/post.sh <<'SH'
#!/usr/bin/env bash
echo "running on $(hostname)"
nvidia-smi || true
SH
lakeshore daemon setup "$D" --script /tmp/post.sh --name post-install --refresh-capabilities

cat > /tmp/apt-install.sh <<'SH'
#!/usr/bin/env bash
apt-get update && apt-get install -y htop
SH
lakeshore daemon setup "$D" --script /tmp/apt-install.sh --sudo --name apt-htop

Setup commands run serially, in submission order, on the daemon.

`daemon hibernate` — pause polling

Tell a daemon to stop polling until a deadline. The process stays alive but goes idle. --until is required and accepts a relative offset (+30s, +5m, +1h, +3d) or an ISO-8601 timestamp.

bash

lakeshore daemon hibernate "$D" --until +1h
lakeshore daemon hibernate "$D" --until 2026-06-01T00:00:00Z

`daemon reset` — clear poll backoff

Queues a reset_backoff command so a daemon stuck in exponential backoff drops back to retry_min_s on its next poll.

bash

lakeshore daemon reset "$D"

`daemon update` — OTA and queue membership

Rolls a daemon to a nymph version. Omit --version and it uses the namespace's target version.

bash

lakeshore daemon update "$D" --version 0.1.4
lakeshore daemon update --all --version 0.1.4 --yes
lakeshore daemon update --prefix fleet- --version 0.1.4 --yes

--url and --sha256 roll a custom binary — pair with lakeshore nymph push:

bash

PUSH=$(lakeshore nymph push ./nymph --json)
# `.url` in the envelope is server-relative; daemons need the
# fully-qualified address, which the human output prints as `full:`.
lakeshore daemon update --all \
  --url "$LAKESHORE_URL$(jq -r '.url' <<<"$PUSH")" \
  --sha256 "$(jq -r '.sha256' <<<"$PUSH")" \
  --yes

The same verb edits Worker.queues. --add-queue / --remove-queue are repeatable, need a positional <id>, do not require --version, and skip the OTA path entirely.

bash

lakeshore daemon update train-01 --add-queue eval
lakeshore daemon update train-01 \
  --add-queue training-h100 \
  --add-queue cpu-prep \
  --remove-queue scratch

`daemon ota` is a deprecated alias

daemon ota still works as a hidden back-compat alias for daemon update, but it requires --version and carries no help text. Use daemon update.

`daemon target-version` / `set-target-version`

The namespace-wide auto-update target. Daemons whose config sets runtime.auto_update = true converge to it on their own.

bash

lakeshore daemon target-version                   # show
lakeshore daemon set-target-version 0.1.4         # set
lakeshore daemon set-target-version --clear       # unset
lakeshore daemon target-version --json

`daemon kill` — drop a daemon

Hard-deletes the Worker row. The cloud instance is terminated by default; --no-terminate keeps the box and just forgets about it.

bash

lakeshore daemon kill "$D"
lakeshore daemon kill "$D" --no-terminate
lakeshore daemon kill --prefix fleet- --yes        # bulk by label prefix

--prefix is mutually exclusive with the <id> positional.

`daemon cleanup` — drop stale daemons

Bulk-delete daemons that haven't polled recently. --older-than defaults to 24h and accepts 30m, 2h, 5d, or a bare number of hours.

bash

lakeshore daemon cleanup --older-than 5m --dry-run
lakeshore daemon cleanup --older-than 5m --yes
lakeshore daemon cleanup --older-than 24h --yes --json

`daemon status` / `daemon stop` — the local daemon record

These two act on the local daemon state file only (~/.config/dreamlake/daemons/local.json, or $DREAMLAKE_STATE_DIR), not on the fleet.

stop sends SIGTERM and waits up to --timeout seconds (default 30). --force sends SIGKILL instead.

bash

lakeshore daemon status
lakeshore daemon stop                # SIGTERM, wait 30s
lakeshore daemon stop --timeout 5
lakeshore daemon stop --force        # SIGKILL

`worker` — a native worker on this machine

worker start supervises a native Python queue worker in the foreground; worker once drains a single job and exits. Both take the identical flag set.

bash

lakeshore worker start --queue default
lakeshore worker start --queue default --url http://localhost:8080 --token "$TOKEN"
lakeshore worker once --queue default
lakeshore worker start --queue default --dry-run    # print the child argv and exit

Under the hood the CLI spawns python -u -m dreamlake.lakeshore.daemon. Flags you omit are deliberately not placed on the child's argv, so the Python module's own LAKESHORE_* env fallbacks stay authoritative:

Flag	Env fallback	Default
`--queue`	`LAKESHORE_QUEUE`	`default`
`--root`	`LAKESHORE_ROOT`	cwd
`--url`	`LAKESHORE_URL`	unset → a local SQLite plane under `LAKESHORE_HOME` (`~/.lakeshore`)
`--namespace`	`LAKESHORE_NAMESPACE`	—
`--token`	`LAKESHORE_CLIENT_TOKEN`	—
`--runtime-policy`	`LAKESHORE_RUNTIME_POLICY`	`strict` \| `minor` \| `off`
`--python`	`LAKESHORE_PYTHON`	`python3`

Plus --module, --env K=V (repeatable), --allow-pickled, and --kill-timeout <seconds> (default 15). The token is passed to the child through the environment, never on argv.

`worker` ignores your saved login

lakeshore worker deliberately does not consult ~/.config/lakeshore/auth.yml. The plane a worker drains has to come from --url / LAKESHORE_URL and --token / LAKESHORE_CLIENT_TOKEN. Running lakeshore auth login is not sufficient setup for a worker.

Compose — bring up a whole group

The five compose verbs are top-level commands that all read a single lakeshore.yaml from the current directory (-f to point elsewhere; there is no upward walk in v1).

bash

lakeshore up --plan                 # show the plan, no side effects
lakeshore up                        # reconcile
lakeshore up gpu --wait             # only the `gpu` daemon group, wait for active
lakeshore ps                        # daemons owned by this file
lakeshore status                    # current vs desired drift (read-only)
lakeshore logs gpu --follow         # multiplexed console output; --poll-ms defaults to 5000
lakeshore down                      # drain (default)
lakeshore down --terminate --prune-queues

--json is available on up, down, ps, and status. See Compose for the file schema.

Exec and run

Two surfaces for one-off work. exec runs bash; run ships a Python script.

Both use Commander's passThroughOptions(): CLI flags come first, everything after the daemon argument is the remote command, and there is no -- separator.

`exec` — top-level, with a picker

bash

lakeshore exec nvidia-smi                          # interactive daemon picker
lakeshore exec --queue training-h100 nvidia-smi    # skip the picker; the CP routes to a queue member

`daemon exec` — an explicit daemon

<daemon> accepts a worker id, a label, or a machine_id.

bash

lakeshore daemon exec "$D" nvidia-smi
lakeshore daemon exec "$D" python -c 'import torch; print(torch.cuda.is_available())'
lakeshore daemon exec --id "$D" hostname     # treat the arg as a literal worker id

`daemon exec` — env, stdin, workdir

--env is repeatable and merges on top of the daemon's own environment. --stdin takes a path, or - to forward this CLI's stdin.

bash

lakeshore daemon exec \
  --env CUDA_VISIBLE_DEVICES=0 \
  --env BATCH=64 \
  "$D" python train.py

lakeshore daemon exec --stdin input.json "$D" jq .results
echo '{"a":1}' | lakeshore daemon exec --stdin - "$D" jq .a
lakeshore daemon exec --workdir /tmp "$D" pwd

The remote command runs under bash -c — not a login shell, so /etc/profile.d and ~/.bashrc are not sourced.

`daemon exec` — timeouts

--timeout kills the remote process (SIGTERM, then SIGKILL five seconds later; the wire reports exit code -2 for a timeout and -1 for a spawn failure or cancellation). --wait is how long the CLI long-polls for the result, default 600 seconds. They are independent.

bash

lakeshore daemon exec --timeout 30 "$D" sleep 120
lakeshore daemon exec --wait 1800 --timeout 1500 "$D" python long_train.py

`daemon exec --json` — the structured envelope

The envelope carries stdout and stderr base64-encoded as stdout_b64 / stderr_b64.

bash

lakeshore daemon exec --json "$D" uname -a \
  | jq '{exit_code, stdout: (.stdout_b64 | @base64d)}'

`run <script.py>` — ship a Python script

Reads the local file and pipes its bytes as stdin to a remote python3 - over the exec channel. Nothing is written to disk on the daemon. A non-.py extension only warns.

bash

cat > /tmp/hello.py <<'PY'
import platform, sys
print("hello from", platform.node(), "python", sys.version_info[:3])
PY
lakeshore run /tmp/hello.py --daemon "$D"
lakeshore run /tmp/hello.py                  # omit --daemon for the picker

bash

lakeshore run /tmp/hello.py \
  --daemon "$D" \
  --env DEBUG=1 \
  --timeout 60 \
  --python /opt/conda/bin/python

`run` is narrower than `daemon exec`

run accepts only --daemon, --env, --timeout, --python, and --json. There is no --workdir and no --stdin — the script body is the stdin. Arguments after <script> are currently ignored. Use daemon exec when you need those knobs.

CLI examples — Daemons, exec, and run

Daemons

daemon launch — provision a host via a provider

daemon launch --count — a fleet

daemon launch — capacity, tags, runners

daemon launch --queue — bind to a queue

daemon launch --bootstrap-script / --setup-script

daemon install — bootstrap on a host you can SSH to

daemon install --binary — rsync a local build

daemon install --slurm — sbatch instead of setsid+nohup

daemon list — what's registered

daemon show — one daemon, full record

daemon launch-log — cloud serial console

daemon setup — queue a post-register script

daemon hibernate — pause polling

daemon reset — clear poll backoff

daemon update — OTA and queue membership

daemon target-version / set-target-version

daemon kill — drop a daemon

daemon cleanup — drop stale daemons

daemon status / daemon stop — the local daemon record

worker — a native worker on this machine

Compose — bring up a whole group

Exec and run

exec — top-level, with a picker

daemon exec — an explicit daemon

daemon exec — env, stdin, workdir

daemon exec — timeouts

daemon exec --json — the structured envelope

run <script.py> — ship a Python script

Read next

`daemon launch` — provision a host via a provider

`daemon launch --count` — a fleet

`daemon launch` — capacity, tags, runners

`daemon launch --queue` — bind to a queue

`daemon launch --bootstrap-script` / `--setup-script`

`daemon install` — bootstrap on a host you can SSH to

`daemon install --binary` — rsync a local build

`daemon install --slurm` — sbatch instead of setsid+nohup

`daemon list` — what's registered

`daemon show` — one daemon, full record

`daemon launch-log` — cloud serial console

`daemon setup` — queue a post-register script

`daemon hibernate` — pause polling

`daemon reset` — clear poll backoff

`daemon update` — OTA and queue membership

`daemon target-version` / `set-target-version`

`daemon kill` — drop a daemon

`daemon cleanup` — drop stale daemons

`daemon status` / `daemon stop` — the local daemon record

`worker` — a native worker on this machine

`exec` — top-level, with a picker

`daemon exec` — an explicit daemon

`daemon exec` — env, stdin, workdir

`daemon exec` — timeouts

`daemon exec --json` — the structured envelope

`run <script.py>` — ship a Python script