# CLI examples — Daemons, exec, and run

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives
on [Auth + setup](/cli/examples/auth.md).

## Daemons

A **daemon** (nymph) is a worker process on a host, managed through the
control plane. Two ways to bring one up:

1. `daemon launch` — the control plane provisions a fresh cloud host
   via a registered provider.
2. `daemon install` — you already have SSH to a host; the CLI
   bootstraps nymph on it.

> **Note:** `lakeshore daemon ...` always goes through the control plane. To run a
> native Python queue worker on your own box, use `lakeshore worker
> start` — covered in its own section below. `daemon start --local` was
> removed.

### `daemon launch` — provision a host via a provider

`--name` sets both the worker label and the cloud Name tag (visible in
the EC2 / GCE console). Omit it and the label falls back to
`<provider>-<short-ulid>`.

```bash
lakeshore daemon launch --provider my-ec2 --name ge-debug-1 --wait
```

`daemon launch` reads project defaults from `.lakeshore` and
`.lakeshore.local`, walking up from cwd and stopping at the git root.
`--no-config` skips that discovery entirely.

### `daemon launch --count` — a fleet

Launches N workers in parallel. With `--count > 1`, `--name` becomes a
prefix: `fleet-00`, `fleet-01`, … `--concurrency` bounds the in-flight
launches (default 5, sized to the AWS `RunInstances` rate limit) and
retries on throttle. `--wait` waits for **all** of them.

```bash
lakeshore daemon launch --provider my-ec2 --name fleet --count 4 --wait
lakeshore daemon launch --provider my-ec2 --name fleet --count 20 --concurrency 8 --wait
```

### `daemon launch` — capacity, tags, runners

`--tag`, `--queue`, and `--runner` are each repeatable, and each one
**replaces** the corresponding list from `.lakeshore` rather than
appending to it. The first `--runner` becomes the default runner.

```bash
lakeshore daemon launch \
  --provider my-ec2 \
  --name h100-1 \
  --tag gpu:h100 \
  --tag region:us-east \
  --runner process \
  --capacity '{"cpu":8,"gpu":1}' \
  --keep-alive 600 \
  --wait
```

`--keep-alive <seconds>` is the idle policy: `0` = exit after one job,
`N` = exit after N idle seconds, `-1` = pool mode (never idle-exit).

### `daemon launch --queue` — bind to a queue

Daemons subscribe to queues by name via `Worker.queues: string[]`. Pass
`--queue` one or more times at launch; omit it entirely and the daemon
lands in the default queue (the empty array).

```bash
lakeshore daemon launch --provider my-ec2 --queue training-h100 --name train-01 --wait

lakeshore daemon launch --provider my-ec2 \
  --queue training-h100 --queue eval \
  --name multi-01 --wait
```

### `daemon launch --bootstrap-script` / `--setup-script`

`--bootstrap-script` runs as root **before** nymph starts (cloud-init).
`--setup-script` is repeatable and queues post-register setup commands,
each with `refresh_capabilities` set; it appends to the
`setup_scripts` list from `.lakeshore`. `--no-setup` clears both
sources.

```bash
cat > /tmp/bootstrap.sh <<'SH'
#!/usr/bin/env bash
set -eux
apt-get update && apt-get install -y htop
SH
lakeshore daemon launch --provider my-ec2 --name ge-debug-1 \
  --bootstrap-script /tmp/bootstrap.sh --wait
```

Both flags read the file locally and send its content inline in the
launch payload.

### `daemon install` — bootstrap on a host you can SSH to

Takes an SSH alias. `--label` defaults to the alias. `--tag` and
`--runner` are repeatable (`--runner` defaults to `process`).
`--keep-alive-s` defaults to `-1` (pool mode). The admin token defaults
to `$LAKESHORE_ADMIN_TOKEN`.

```bash
lakeshore daemon install my-server --label ge-debug-1
lakeshore daemon install my-server --tag gpu:a100 --runner docker --keep-alive-s 900
```

> **Warning:** `daemon install` waits for the Worker row by default and exposes only
> the negation `--no-wait`. There is no positive `--wait` on `install`
> (there is on `launch`).

### `daemon install --binary` — rsync a local build

Skip the download and rsync a locally-built nymph binary onto the host.
Match the host arch. `--nymph-base-url` overrides the download source
when you're not using `--binary`.

```bash
cargo build --release --target x86_64-unknown-linux-gnu --manifest-path nymph/Cargo.toml
lakeshore daemon install my-server \
  --label ge-debug-1 \
  --binary nymph/target/x86_64-unknown-linux-gnu/release/nymph
```

### `daemon install --slurm` — sbatch instead of setsid+nohup

Wrap the nymph startup in an `sbatch` heredoc so the daemon runs as a
SLURM job. Use on login nodes where a detached background process is
discouraged.

```bash
lakeshore daemon install login.cluster --slurm --slurm-partition gpu
```

### `daemon list` — what's registered

Every daemon in the namespace, with last-seen time, state, and a
`queues` column (`—` when empty, i.e. the default queue).

```bash
lakeshore daemon list
lakeshore daemon list --json | jq '.[] | {id, label, state, lastSeenAt}'
```

`--watch` takes an **optional** value; bare `--watch` refreshes every 2
seconds. Ctrl-C exits.

```bash
lakeshore daemon list --watch        # every 2s
lakeshore daemon list --watch 5      # every 5s
```

### `daemon show` — one daemon, full record

Same `--json` / `--watch [seconds]` treatment.

```bash
lakeshore daemon show "$D"
lakeshore daemon show "$D" --json | jq .
lakeshore daemon show "$D" --watch
```

### `daemon launch-log` — cloud serial console

When a launched daemon never reaches its first `/hello`, grab the
cloud serial-console output. EC2-only today.

```bash
lakeshore daemon launch-log "$D" --tail 200
lakeshore daemon launch-log "$D" --no-latest --json
```

### `daemon setup` — queue a post-register script

Upload a bash script and run it on a daemon that has already
registered. `--script` is required; the content is uploaded inline.
`--sudo` runs it as root, and `--refresh-capabilities` makes the daemon
re-detect its host capabilities afterwards and publish them on the next
poll.

```bash
cat > /tmp/post.sh <<'SH'
#!/usr/bin/env bash
echo "running on $(hostname)"
nvidia-smi || true
SH
lakeshore daemon setup "$D" --script /tmp/post.sh --name post-install --refresh-capabilities

cat > /tmp/apt-install.sh <<'SH'
#!/usr/bin/env bash
apt-get update && apt-get install -y htop
SH
lakeshore daemon setup "$D" --script /tmp/apt-install.sh --sudo --name apt-htop
```

Setup commands run **serially**, in submission order, on the daemon.

### `daemon hibernate` — pause polling

Tell a daemon to stop polling until a deadline. The process stays alive
but goes idle. `--until` is required and accepts a relative offset
(`+30s`, `+5m`, `+1h`, `+3d`) or an ISO-8601 timestamp.

```bash
lakeshore daemon hibernate "$D" --until +1h
lakeshore daemon hibernate "$D" --until 2026-06-01T00:00:00Z
```

### `daemon reset` — clear poll backoff

Queues a `reset_backoff` command so a daemon stuck in exponential
backoff drops back to `retry_min_s` on its next poll.

```bash
lakeshore daemon reset "$D"
```

### `daemon update` — OTA and queue membership

Rolls a daemon to a nymph version. Omit `--version` and it uses the
namespace's target version.

```bash
lakeshore daemon update "$D" --version 0.1.4
lakeshore daemon update --all --version 0.1.4 --yes
lakeshore daemon update --prefix fleet- --version 0.1.4 --yes
```

`--url` and `--sha256` roll a custom binary — pair with
`lakeshore nymph push`:

```bash
PUSH=$(lakeshore nymph push ./nymph --json)
# `.url` in the envelope is server-relative; daemons need the
# fully-qualified address, which the human output prints as `full:`.
lakeshore daemon update --all \
  --url "$LAKESHORE_URL$(jq -r '.url' <<<"$PUSH")" \
  --sha256 "$(jq -r '.sha256' <<<"$PUSH")" \
  --yes
```

The same verb edits `Worker.queues`. `--add-queue` / `--remove-queue`
are repeatable, need a positional `<id>`, do **not** require
`--version`, and skip the OTA path entirely.

```bash
lakeshore daemon update train-01 --add-queue eval
lakeshore daemon update train-01 \
  --add-queue training-h100 \
  --add-queue cpu-prep \
  --remove-queue scratch
```

> **Warning:** `daemon ota` still works as a hidden back-compat alias for
> `daemon update`, but it requires `--version` and carries no help text.
> Use `daemon update`.

### `daemon target-version` / `set-target-version`

The namespace-wide auto-update target. Daemons whose config sets
`runtime.auto_update = true` converge to it on their own.

```bash
lakeshore daemon target-version                   # show
lakeshore daemon set-target-version 0.1.4         # set
lakeshore daemon set-target-version --clear       # unset
lakeshore daemon target-version --json
```

### `daemon kill` — drop a daemon

Hard-deletes the Worker row. **The cloud instance is terminated by
default**; `--no-terminate` keeps the box and just forgets about it.

```bash
lakeshore daemon kill "$D"
lakeshore daemon kill "$D" --no-terminate
lakeshore daemon kill --prefix fleet- --yes        # bulk by label prefix
```

`--prefix` is mutually exclusive with the `<id>` positional.

### `daemon cleanup` — drop stale daemons

Bulk-delete daemons that haven't polled recently. `--older-than`
defaults to `24h` and accepts `30m`, `2h`, `5d`, or a bare number of
hours.

```bash
lakeshore daemon cleanup --older-than 5m --dry-run
lakeshore daemon cleanup --older-than 5m --yes
lakeshore daemon cleanup --older-than 24h --yes --json
```

### `daemon status` / `daemon stop` — the local daemon record

These two act on the **local** daemon state file only
(`~/.config/dreamlake/daemons/local.json`, or `$DREAMLAKE_STATE_DIR`),
not on the fleet.

`stop` sends SIGTERM and waits up to `--timeout` seconds (default 30).
`--force` sends SIGKILL instead.

```bash
lakeshore daemon status
lakeshore daemon stop                # SIGTERM, wait 30s
lakeshore daemon stop --timeout 5
lakeshore daemon stop --force        # SIGKILL
```

## `worker` — a native worker on this machine

`worker start` supervises a native Python queue worker in the
foreground; `worker once` drains a single job and exits. Both take the
identical flag set.

```bash
lakeshore worker start --queue default
lakeshore worker start --queue default --url http://localhost:8080 --token "$TOKEN"
lakeshore worker once --queue default
lakeshore worker start --queue default --dry-run    # print the child argv and exit
```

Under the hood the CLI spawns `python -u -m dreamlake.lakeshore.daemon`.
Flags you omit are deliberately **not** placed on the child's argv, so
the Python module's own `LAKESHORE_*` env fallbacks stay authoritative:

| Flag                | Env fallback              | Default                                            |
| ------------------- | ------------------------- | -------------------------------------------------- |
| `--queue`           | `LAKESHORE_QUEUE`         | `default`                                          |
| `--root`            | `LAKESHORE_ROOT`          | cwd                                                |
| `--url`             | `LAKESHORE_URL`           | unset → a local SQLite plane under `LAKESHORE_HOME` (`~/.lakeshore`) |
| `--namespace`       | `LAKESHORE_NAMESPACE`     | —                                                  |
| `--token`           | `LAKESHORE_CLIENT_TOKEN`  | —                                                  |
| `--runtime-policy`  | `LAKESHORE_RUNTIME_POLICY`| `strict` \| `minor` \| `off`                       |
| `--python`          | `LAKESHORE_PYTHON`        | `python3`                                          |

Plus `--module`, `--env K=V` (repeatable), `--allow-pickled`, and
`--kill-timeout <seconds>` (default 15). The token is passed to the
child through the environment, never on argv.

> **Warning:** `lakeshore worker` deliberately does **not** consult
> `~/.config/lakeshore/auth.yml`. The plane a worker drains has to come
> from `--url` / `LAKESHORE_URL` and `--token` / `LAKESHORE_CLIENT_TOKEN`.
> Running `lakeshore auth login` is not sufficient setup for a worker.

## Compose — bring up a whole group

The five compose verbs are **top-level** commands that all read a
single `lakeshore.yaml` from the current directory (`-f` to point
elsewhere; there is no upward walk in v1).

```bash
lakeshore up --plan                 # show the plan, no side effects
lakeshore up                        # reconcile
lakeshore up gpu --wait             # only the `gpu` daemon group, wait for active
lakeshore ps                        # daemons owned by this file
lakeshore status                    # current vs desired drift (read-only)
lakeshore logs gpu --follow         # multiplexed console output; --poll-ms defaults to 5000
lakeshore down                      # drain (default)
lakeshore down --terminate --prune-queues
```

`--json` is available on `up`, `down`, `ps`, and `status`. See
[Compose](/get-started/compose.md) for the file schema.

## Exec and run

Two surfaces for one-off work. `exec` runs bash; `run` ships a Python
script.

Both use Commander's `passThroughOptions()`: **CLI flags come first,
everything after the daemon argument is the remote command, and there
is no `--` separator.**

### `exec` — top-level, with a picker

```bash
lakeshore exec nvidia-smi                          # interactive daemon picker
lakeshore exec --queue training-h100 nvidia-smi    # skip the picker; the CP routes to a queue member
```

### `daemon exec` — an explicit daemon

`<daemon>` accepts a worker id, a label, or a `machine_id`.

```bash
lakeshore daemon exec "$D" nvidia-smi
lakeshore daemon exec "$D" python -c 'import torch; print(torch.cuda.is_available())'
lakeshore daemon exec --id "$D" hostname     # treat the arg as a literal worker id
```

### `daemon exec` — env, stdin, workdir

`--env` is repeatable and merges on top of the daemon's own
environment. `--stdin` takes a path, or `-` to forward this CLI's
stdin.

```bash
lakeshore daemon exec \
  --env CUDA_VISIBLE_DEVICES=0 \
  --env BATCH=64 \
  "$D" python train.py

lakeshore daemon exec --stdin input.json "$D" jq .results
echo '{"a":1}' | lakeshore daemon exec --stdin - "$D" jq .a
lakeshore daemon exec --workdir /tmp "$D" pwd
```

The remote command runs under `bash -c` — **not** a login shell, so
`/etc/profile.d` and `~/.bashrc` are not sourced.

### `daemon exec` — timeouts

`--timeout` kills the **remote** process (SIGTERM, then SIGKILL five
seconds later; the wire reports exit code `-2` for a timeout and `-1`
for a spawn failure or cancellation). `--wait` is how long the **CLI**
long-polls for the result, default 600 seconds. They are independent.

```bash
lakeshore daemon exec --timeout 30 "$D" sleep 120
lakeshore daemon exec --wait 1800 --timeout 1500 "$D" python long_train.py
```

### `daemon exec --json` — the structured envelope

The envelope carries stdout and stderr base64-encoded as `stdout_b64` /
`stderr_b64`.

```bash
lakeshore daemon exec --json "$D" uname -a \
  | jq '{exit_code, stdout: (.stdout_b64 | @base64d)}'
```

### `run <script.py>` — ship a Python script

Reads the local file and pipes its bytes as stdin to a remote
`python3 -` over the exec channel. Nothing is written to disk on the
daemon. A non-`.py` extension only warns.

```bash
cat > /tmp/hello.py <<'PY'
import platform, sys
print("hello from", platform.node(), "python", sys.version_info[:3])
PY
lakeshore run /tmp/hello.py --daemon "$D"
lakeshore run /tmp/hello.py                  # omit --daemon for the picker
```

```bash
lakeshore run /tmp/hello.py \
  --daemon "$D" \
  --env DEBUG=1 \
  --timeout 60 \
  --python /opt/conda/bin/python
```

> **Note:** `run` accepts only `--daemon`, `--env`, `--timeout`, `--python`, and
> `--json`. There is no `--workdir` and no `--stdin` — the script body
> *is* the stdin. Arguments after `<script>` are currently ignored. Use
> `daemon exec` when you need those knobs.

## Read next

- [Daemon launch](/nymph/daemons.md) — the provider-driven path in detail.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — install → list → kill → cleanup.
- [Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md)
- [Daemon protocol](/nymph/protocol.md) — the exec / poll / ack wire format.
- [Compose](/get-started/compose.md) — the `lakeshore.yaml` schema.