# DreamLake — Full documentation

> Lakeshore runs your Python functions on remote GPUs — decorate a function, call it, and Lakeshore provisions the host, ships your code, executes it, and streams the result back.

Generated from https://lakeshore.dreamlake.ai. 79 pages.

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths

# Happy Paths — Operator

Five paths across the operator surface: bringing the stack up locally,
putting a daemon on a remote host, declarative cluster management, cloud
provisioning, and watching it all from the dashboard.

These are the paths that exercise *everything together*. If they are
green, the system can be trusted with real workloads.

## The list

| # | Path | Verifies |
| - | ---- | -------- |
| 03 | [Hello, local stack](/admin/happy-paths/03-hello-local-stack.md) | Control plane + Mongo + nymph on `localhost` via mprocs |
| 08 | [Bootstrap a daemon on a remote host](/admin/happy-paths/08-bootstrap-daemon-remote.md) | `lakeshore daemon install` onto a bare SSH host |
| 12 | [Compose a cluster declaratively](/admin/happy-paths/12-compose-cluster.md) | `lakeshore up` / `down` against a `lakeshore.yaml` |
| 13 | [Cloud provider launch](/admin/happy-paths/13-cloud-provider-launch.md) | EC2 / GCE / Kube provisioning through the control plane |
| 14 | [Dashboard observability](/admin/happy-paths/14-dashboard-observability.md) | Reactive view of an in-flight invocation |

## Cross-surface prerequisites

| For path | You also need green |
| -------- | ------------------- |
| 08 | [07 · SSH provider smoke](/cli/happy-paths/07-ssh-provider-smoke.md) |
| 12 | A provider registered, plus its credentials as secrets |
| 13 | Cloud credentials stored as a Lakeshore secret |
| 14 | A submitted invocation — pick any from [Python SDK](/python-sdk/happy-paths.md) |

## The cascade

`03` is the foundation — everything else assumes you can stand the stack
up. `08` proves daemon-on-remote bootstrap, which the cloud paths later
automate. `12` is the declarative story. `13` is the most complex and the
only one that spends real money, so verify it last, not first. `14` ties
it together by showing the system from the user's reactive view.

→ Switch surface: [Python SDK](/python-sdk/happy-paths.md) · [CLI](/cli/happy-paths.md)

---

Source: https://lakeshore.dreamlake.ai/cli/examples

# CLI examples

Shell-by-shell walkthroughs of common operator tasks, grouped by what
you're trying to do. Start with the auth page — the rest assume the
shell setup it describes.

| Page | Covers |
| ---- | ------ |
| [Auth + setup](/cli/examples/auth.md) | One-time login, shell environment, `$D` seeding |
| [Providers + discover](/cli/examples/providers-and-discover.md) | Register cloud, SSH, SLURM, Kube targets; the local discovery helpers |
| [Queues + jobs](/cli/examples/queues-and-jobs.md) | Named work queues and inspecting invocations |
| [Daemons + exec/run](/cli/examples/daemons-and-exec.md) | Worker daemons, compose, and one-off command execution |
| [Storage, code, mounts](/cli/examples/storage-code-mounts.md) | S3 storage entries, code archives, mount declarations |
| [Secrets, modes, tunnels, config, nymph](/cli/examples/secrets-modes-tunnels-config-nymph.md) | The long tail of management commands |

If you want a guided narrative instead, see the
[CLI happy paths](/cli/happy-paths.md).

## Conventions used across these pages

- Names for providers, secrets, modes, tunnels, mounts, storage
  entries, and queues all validate against `^[a-z0-9][a-z0-9-]*$`.
- Every action calls `process.exit(rc)`. `2` is the conventional "bad
  usage / not configured" code and `1` is the failure code, so these
  blocks are safe under `set -e`.
- `--json` is available on most read commands and is the intended
  scripting surface; the table output is for humans and not stable.

## What's not on these pages

- **Provider-driven dispatch from Python** — the `@dls.udf` decorator,
  modes, capacity matching — lives on [`/python-sdk`](/python-sdk.md). The
  CLI's role there is to **register** the targets; the Python API runs
  the jobs.
- **Lower-level daemon internals** — capability publishing, hibernate
  semantics, OTA self-replace — are documented at
  [`/nymph/daemons`](/nymph/daemons.md) and
  [`/nymph/protocol`](/nymph/protocol.md).
- **The `dreamlake` group** — a separate data plane with its own
  servers, its own credential store (`~/.dreamlake/`), and its own
  connection flags. It is not covered by this cookbook.

## Read next

- [`/cli`](/cli.md) — the overview and the full command surface.
- [`/cli/tui`](/cli/tui.md) — the interactive dashboard over the same data.
- [`/python-sdk`](/python-sdk.md) — what you can launch from Python.
- [`/nymph/daemon-lifecycle`](/nymph/daemon-lifecycle.md) — install → list → kill → cleanup.
- [`docs.dreamlake.ai/lakeshore/providers`](https://docs.dreamlake.ai/lakeshore/providers) — the EC2 / GCE / SLURM / Kube / SSH launchers.
- [`/cli/completion`](/cli/completion.md) — the full completion installation guide.

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths

# Happy Paths — CLI

Five paths covering the CLI surface end-to-end: the auth wire, command
execution, provider invocation, storage round-trip, and the token
lifecycle.

Walk top-down. The CLI is the user-facing entry point for almost every
Lakeshore action — these paths are the ones that fail loudest when
something breaks.

## The list

| # | Path | Verifies |
| - | ---- | -------- |
| 02 | [Hello, hosted CP](/cli/happy-paths/02-hello-hosted-cp.md) | `auth login` + `auth status` against a real control plane |
| 04 | [Smoke-exec cascade](/cli/happy-paths/04-smoke-exec-cascade.md) | The whole `daemon exec` surface — stdout, stderr, exit codes, env, stdin, timeout, long-poll |
| 07 | [SSH provider smoke](/cli/happy-paths/07-ssh-provider-smoke.md) | `ssh upload` / `providers add` + `providers test` against a real host |
| 09 | [Storage round-trip](/cli/happy-paths/09-storage-round-trip.md) | `storage add` + `presign` + `credentials` against a real bucket |
| 15 | [Token lifecycle](/cli/happy-paths/15-token-lifecycle.md) | `admin tokens create` + `auth login` against a token-enforced control plane |

## Cross-surface prereqs

| For path | You also need green |
| -------- | ------------------- |
| 04 | [03 · Hello, local stack](/admin/happy-paths/03-hello-local-stack.md) (Operator) |
| 07 | An SSH-reachable host |
| 09 | S3/R2 credentials registered as a Lakeshore secret |
| 15 | A control plane with `LAKESHORE_ADMIN_TOKEN` set (not open mode) |

## The cascade

`02` is the wire test against the hosted control plane — it proves auth
works. `04` exercises the entire `daemon exec` contract in one shell
script and bails on the first failure, which makes it cheap and
comprehensive. `07` validates the provider abstraction without
involving a daemon. `09` covers the storage record plus the presign and
STS credential paths. `15` is the auth-mode story: admin token → client
token → SDK.

→ Switch surface: [Python SDK](/python-sdk/happy-paths.md) · [Operator](/admin/happy-paths.md)

---

Source: https://lakeshore.dreamlake.ai/cli/installation

# Install the CLI

```bash
npm i -g @dreamlake/lakeshore
```

> **Note:** On macOS: `brew install node && npm i -g @dreamlake/lakeshore`

The package is ESM. The `lakeshore` bin prefers the compiled
`dist/cli/index.js`; in a source checkout without a build it falls back
to running `src/cli/index.ts` directly (under Bun) or via the `tsx` ESM
loader (under Node).

Verify the install with `lakeshore --help`. There is **no**
`lakeshore --version` — the program never registers one, so Commander
rejects it as an unknown option. Use `npm ls -g @dreamlake/lakeshore`
to see which version you have.

## Authenticate

Your admin will give you a **token** and a **namespace** (e.g. `fortyfive`).

```bash
lakeshore auth login \
  --server https://api.lakeshore.dreamlake.ai \
  --namespace <your-namespace> \
  --token <your-token>
```

`login` validates the token by calling
`GET <server>/v1/namespaces/<ns>/whoami` with a Bearer header before it
saves anything. On success it writes YAML to
`$XDG_CONFIG_HOME/lakeshore/auth.yml` (default
`~/.config/lakeshore/auth.yml`) with mode `600`:

```yaml file="~/.config/lakeshore/auth.yml"
server: https://api.lakeshore.dreamlake.ai
namespace: your-namespace
token: dlk_xxxxxxxx…
```

> **Warning:** `--namespace` must match the namespace the token was minted for.
> Without it the CLI defaults to `default`, and the control plane
> rejects the call because the token does not grant access to that
> namespace.

Omit `--token` to be prompted interactively. If `LAKESHORE_ADMIN_TOKEN`
is exported, `login` instead mints a fresh per-namespace token on the
server for you and saves that; `--name <label>` labels it.

Verify with:

```bash
lakeshore auth status
```

This reloads the file and re-validates against `whoami`, printing the
config path, server, namespace, token prefix, last-used timestamp, and
`authMode`.

## Env vars that change where commands point

| Variable              | Read by                     | Effect                                                                     |
| --------------------- | --------------------------- | -------------------------------------------------------------------------- |
| `LAKESHORE_URL`       | every command **except** `auth` | Control-plane base URL. When set, the CLI sends **no bearer token** and ignores the saved login. |
| `LAKESHORE_NAMESPACE` | every command except `auth` | Namespace used alongside `LAKESHORE_URL`. Defaults to `default`.            |
| `LAKESHORE_SERVER`    | the `auth` commands only    | Server URL when `--server` is not passed.                                   |
| `LAKESHORE_ADMIN_TOKEN` | `auth login`, `admin`, `nymph`, `daemon install` | Admin bearer. Required for every `admin` subcommand. |

> **Warning:** `LAKESHORE_URL` and `LAKESHORE_SERVER` are not interchangeable, and
> `LAKESHORE_URL` is not a convenience alias for the saved server. In
> `LAKESHORE_URL` mode the CLI deliberately sends no token — which is
> right for an open-mode local stack and wrong against a token-enforced
> deployment. Unset it after `auth login` if you need the token path.

`lakeshore auth logout` deletes the saved file.

## Where to next

| I want to…                      | Page                                                       |
| ------------------------------- | ---------------------------------------------------------- |
| See the whole command surface   | [`lakeshore` CLI](/cli.md)                                    |
| Run my first commands           | [CLI examples · auth](/cli/examples/auth.md)                  |
| Set up shell tab-completion     | [Completion](/cli/completion.md)                              |
| Connect a cloud provider        | [Providers](https://docs.dreamlake.ai/lakeshore/providers) |
| Walk a verified end-to-end flow | [Happy paths](/cli/happy-paths.md)                            |

---

Source: https://lakeshore.dreamlake.ai/dev/happy-paths

The lists below are split by surface so you can focus on the layer
you own: [Python SDK](/python-sdk/happy-paths.md), [CLI](/cli/happy-paths.md),
or [Operator](/admin/happy-paths.md).

# Happy Paths

A **happy path** is the canonical, no-edge-cases flow through a feature
— what a user runs when everything works. For Lakeshore, the paths
below are both:

- **the test plan** — each one is a scripted walkthrough; "green
  across all of them" is the shipping bar; and
- **the implementation roadmap** — when one is red, the work to turn
  it green is the next concrete unit.

## Working through them

- **Walk top-down.** Ordered leaf-nodes-first; each depends on the
  ones above. If `02` is red, don't chase `08` — fix `02` first.
- **One owner, one outcome.** Each page ends with a single
  "I ran it and it worked" verdict.
- **Status drifts.** Re-run regularly; a green path can go red when
  unrelated changes ship.
- **Cheap checkpoint vs. expensive.** Unit-test surface lives at
  [/get-started/status](/get-started/status.md) (runs every commit). Happy paths are the
  expensive checkpoint that ties everything together.

> **Note:** Not the user-facing tutorial set ([Tutorials](/get-started/tutorials.md)). Happy
> paths are project-facing — contributors and operators run them to
> verify the surface is intact. Many duplicate the tutorials on
> purpose, with stricter step-by-step assertions.

## The list

| # | Path | Verifies |
| - | ---- | -------- |
| 01 | [Hello, local Python](/python-sdk/happy-paths/01-hello-local-python.md) | Python SDK install + decorator |
| 02 | [Hello, hosted control plane](/cli/happy-paths/02-hello-hosted-cp.md) | Auth + token wire |
| 03 | [Hello, local stack](/admin/happy-paths/03-hello-local-stack.md) | CP + nymph + CLI on `localhost` |
| 04 | [Smoke-exec cascade](/cli/happy-paths/04-smoke-exec-cascade.md) | The whole `daemon exec` surface |
| 05 | [Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md) | Submit → poll → execute round-trip |
| 06 | [Fan out and gather](/python-sdk/happy-paths/06-fan-out-gather.md) | Parallelism + ordered collection |
| 07 | [SSH provider smoke](/cli/happy-paths/07-ssh-provider-smoke.md) | Provider abstraction (no daemon) |
| 08 | [Bootstrap a daemon on a remote host](/admin/happy-paths/08-bootstrap-daemon-remote.md) | Daemon-on-remote install |
| 09 | [Storage round-trip](/cli/happy-paths/09-storage-round-trip.md) | Presigned URLs against a real bucket |
| 10 | [Code mount](/python-sdk/happy-paths/10-code-mount.md) | Local code → S3 → worker |
| 11 | [Storage access from a UDF](/python-sdk/happy-paths/11-storage-from-udf.md) | Worker fetch from logical storage |
| 12 | [Compose a cluster declaratively](/admin/happy-paths/12-compose-cluster.md) | Declarative `lakeshore up/down` |
| 13 | [Cloud provider launch](/admin/happy-paths/13-cloud-provider-launch.md) | EC2/GCE/Slurm/Kube launch |
| 14 | [Dashboard observability](/admin/happy-paths/14-dashboard-observability.md) | Reactive view of the system |
| 15 | [Token lifecycle](/cli/happy-paths/15-token-lifecycle.md) | Admin → namespace → client auth |

---

Source: https://lakeshore.dreamlake.ai/get-started/architecture

# Architecture

> Understand the three layers, the objects the control plane stores, and
> how an invocation travels from your laptop to a remote worker.

Lakeshore ships as three independently-deployed pieces, each swappable
without touching the others:

- **`lakeshore` CLI** — Node + TypeScript, distributed via npm
  (`@dreamlake/lakeshore`). The `lakeshore` binary registers providers,
  manages secrets / modes / queues / storages / mounts / tunnels, brings
  daemons up, and runs one-off commands against them.
- **Lakeshore control plane** — Node + Fastify 5 + Prisma 5 + Mongo,
  deployed on Heroku. The single brain that owns canonical state and
  hands Invocations to Workers.
- **`nymph` daemon** — Rust binary (the `nymph` submodule —
  `dreamlake-ai/nymph`), the host process that supervises Invocations
  under docker / gvisor / process runners. Closed-source for now; built
  from source.

```
┌─────────────────────────────────────────────────────────────────┐
│  Client surface                                                 │
│   ─ lakeshore CLI         (Node, on npm — operator verbs)       │
│   ─ Python SDK            (dreamlake-lakeshore — @udf, Futures) │
├─────────────────────────────────────────────────────────────────┤
│  Control plane  (Lakeshore, Node + Fastify + Prisma)            │
│                                                                 │
│   Durable store   Mongo    — every model below                  │
│   Hot lane        Redis    — result-chunk delivery, log cache   │
│   Blob store      S3       — spilled payloads, code archives,   │
│                              archived exec logs                 │
│                                                                 │
│   ─ Scheduling   Queue · Invocation · Function · ExecJob        │
│   ─ Compute      Worker · Provider · Mode · QueueServer         │
│   ─ Data         Storage · Mount · CodeRepo                     │
│   ─ Identity     Namespace · Token · EnrollToken · DaemonKey ·  │
│                  Secret · Tunnel                                │
│   ─ Audit        Event · ElasticityEvent · ResultChunk          │
├─────────────────────────────────────────────────────────────────┤
│  Runtime  (Rust daemon, on each compute host)                   │
│   ─ Daemon (nymph)        (long-polls; runs Invocations under   │
│                            process / docker / gvisor / slurm /  │
│                            kube runners)                        │
│   ─ Wire protocol         (msgpack long-poll; daemon-initiated) │
└─────────────────────────────────────────────────────────────────┘
```

`prisma/schema.prisma` defines exactly 21 models. All of them carry a
`namespaceId` except three: `Namespace` itself (it *is* the boundary),
`QueueServer`, and `NymphRelease` (a global binary registry, deduped by
`sha256`).

## Reading the layers

**Client surface** is what you talk to. The CLI is the operator surface:
register providers, create queues, launch daemons, exec commands against
them, inspect jobs. The Python SDK (`dreamlake-lakeshore`) is the
developer surface — the `@udf(queue=...)` decorator, `submit()` returning
a durable invocation id, and `SyncQueue.result(id)`. See
[Python SDK](/python-sdk.md).

**Control plane** is [Lakeshore](/dev/controlplane-internals) — a single
Node.js + TypeScript service. Three stores sit behind it, each with a
distinct job:

- **Mongo (via Prisma)** is the durable record and the source of truth
  for every model on this page. It also carries the dispatch hot path:
  a raw `findAndModify` on the `Invocation` collection —
  `{ state: "queued", "runConfig.queue": … }` sorted by
  `{ priority: -1, submittedAt: 1 }` — atomically flips one row to
  `running` and stamps the `workerId`. That is why v0 needs no Lua
  scripts and no row locks.
- **Redis** is a low-latency delivery channel, *not* the scheduler. It
  carries exec log streams (`exec:<id>:stdout`), per-invocation result
  journals (`invocation:<id>:result`), and the daemon-signature
  replay-nonce store. The `ResultChunk` rows in Mongo are the durable
  truth — trimming Redis never loses a frame, reads just degrade to
  Mongo. (A Redis hot lane for *queue dispatch* is described in the
  schema header as a designed-but-unbuilt v1+ optimization.)
- **S3** holds anything too big to inline: spilled payloads, `CodeRepo`
  archives, and archived exec logs. See
  [Payload store](/get-started/payloads.md) for the concept and
  [`/dev/payload-store`](/dev/payload-store) for the spec.

**Runtime** is what executes your code. The
[daemon](/nymph/daemons.md) — `nymph`, a Rust binary — runs on every
compute host; it speaks the [wire protocol](/nymph/protocol.md) to the
control plane and supervises the Invocation under one of the registered
runner kinds: `process` (alias `subprocess`), `docker`, `gvisor`,
`slurm`, or `kube`. Daemons are **outbound-only**: they long-poll the
control plane and never accept inbound connections, so a compute host
needs egress and nothing else.

> **Warning:** Older notes describe a `nymph relay` — a FIFO ↔ HTTPS forwarder for HPC
> clusters whose compute nodes have no egress. There is no `relay`
> subcommand, no FIFO code, and no `LAKESHORE_RELAY_*` handling in the
> nymph source. `tui` is the binary's only subcommand. Treat
> [Relay config](/dev/config-relay) as a design note, not shipped
> behaviour.

## The object model

Everything the control plane stores is namespaced. Start with the four
you will meet first, then the supporting cast.

### The core four

#### Queue

A **named pipe** carrying pending Invocations, and the first-class
scheduling primitive. Workers subscribe by name —
`Worker.queues: string[]` records which queues a daemon pulls from, and
the *empty array* means the default queue (there is no sentinel string).

A Queue carries three policies:

| Policy | Field | Values |
| --- | --- | --- |
| Scheduling | `kind` | `fifo` · `priority` · `boltzmann` · `filo` |
| Elasticity | `elasticity.kind` | `fixed` · `fully_elastic` · `pool_with_threshold` · `max_count` |
| Admission | `admission` | `max_depth`, `on_full` (`reject` \| `block`), `rate_limit`, `deadline_cutoff_s` |

`kindParams` holds type-specific knobs — `{ temperature: 0.5 }` for
`boltzmann`, `{ priority_field: "p" }` for `priority`.

> **Warning:** All four `kind` values are accepted and stored, but the dispatcher's
> claim always sorts `{ priority: -1, submittedAt: 1 }` regardless of
> kind. `boltzmann` and `filo` do not yet change dispatch order.
> Similarly, `admission` is validated and stored but never enforced at
> admit time.

> **Warning:** The stored values are `fully_elastic`, `pool_with_threshold`, and
> `max_count` — underscores, not hyphens. A hyphenated value is rejected
> by the API. (The `lakeshore queues add --elasticity` flag accepts the
> hyphenated spelling and converts it before the request.)

Lifecycle: the schema declares `active`, `draining`, `paused`, and
`archived`, but only three are reachable through routes — `/drain` →
`draining`, `/archive` → `archived`, `/unarchive` → `active`. Nothing
writes `paused`.

See [Queues](/get-started/queues.md) for the concept page,
[Elasticity](/get-started/elasticity.md) and
[Scaling rules](/get-started/scaling-rules.md) for the autoscaling
behaviour, and [`/dev/queues`](/dev/queues) for the design note.

#### Worker

A daemon process registered with the control plane. Each Worker carries
a stable `machineId` (reported on `/hello` and preserved across
restarts), a queues array, a `runners` array, a `capabilities` object
(runner list, `max_invocations`, workdir, plus the detected CPU/GPU
inventory), and a tag set. Queue membership and elasticity `match`
constraints decide which Invocations a Worker can claim.

Worker state runs `pending` → `setting_up` → `joining` → `active` →
`gone`. `pending` is a row pre-created by `daemons/launch` before the
host boots; `setting_up` means queued setup commands are still draining;
`joining` is set by `/hello`; `active` is the first poll after that.
`draining` is declared in the schema but no route writes it. A seventh
label, `stale`, is **derived at read time** when `lastSeenAt` is older
than 60 s — it is never persisted.

Two fields worth knowing: `hostStatus` tracks the *cloud instance*
lifecycle independently of daemon poll state (reconciled against the
provider's DescribeInstances), and `composeProject` records the
`project:` slug when the Worker was spawned by `lakeshore up` — ad-hoc
daemons leave it null and stay invisible to `lakeshore ps` /
`lakeshore down`.

See [Daemon runtime](/nymph/daemons.md) and
[Daemon lifecycle](/nymph/daemon-lifecycle.md).

#### Invocation

One deferred execution of a Function. Keyed by a **ULID generated by the
producer** — the same id that flows across the wire — with `attempt`,
`priority`, submitted / started / finished timestamps, and a state:

`queued` → `running` → one of `succeeded` · `failed` · `killed` ·
`timeout`.

Arguments and results ride as inline msgpack in `argsBlob` /
`resultBlob`. The model also declares `argsRef` / `resultRef` for
spilling to S3, but no route writes them yet — the
[payload store](/get-started/payloads.md) is wired on the ExecJob path
first. `correlationId`, `parentInvocationId`, and a denormalized
`rootInvocationId` give you the pipeline tree — the root field equals
the invocation's own id when it *is* the root.

There is **no `queueId` column**. Phase A.5 removed it; the queue name
is denormalized onto `runConfig.queue` by `POST /v1/producer/submit`,
and every read path (admin list, `?queue=` filter, the dispatcher's
claim) matches on that embedded path. Absent, it falls back to
`"default"`. `idempotencyKey` and `codeRepoId` exist on the model but
no route writes either one yet.

Inspect via `lakeshore jobs list / show / kill` — see
[Jobs](/get-started/jobs.md).

#### Worker host

The machine the daemon runs on. Brought online either by the control
plane (via a `Provider` row — EC2 / GCE / SLURM / Kube) or by the CLI
bootstrapping nymph onto an existing host over SSH. See
[Daemon launch](/nymph/daemons.md) and
[Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md).

### Compute and placement

| Model | What it is |
| --- | --- |
| **Provider** | Where Worker hosts come from. The API field is `launcher`, and the only accepted values are `SSH`, `SLURM`, `EC2`, `GCE`, `Kube` — capitalized exactly like that. (`launcher` maps to the DB column `Provider.kind`; the API's `kwargs` maps to `Provider.config`.) Credentials ride as `$secret` references. Removal is a soft delete — the row is renamed with a ` [deleted <iso>]` suffix and can be restored. |
| **Mode** | A named compute config: `backend`, `runner`, `image`, resources, env, timeout, and an optional Provider link. The route does **not** validate `backend` or `runner`; it defaults them to `"fabric"` and `"process"` and otherwise stores any string. Managed via `lakeshore modes ...`; `.dreamrc` works as a local fallback. |
| **QueueServer** | A dispatch tier — `edge`, `region`, or `central` — with a parent/child self-relation, and the one model that is not namespaced. **Schema-only: no code reads it**, and the only writes to `Worker.serverNodeId` are literal `null`. Do not plan around the tier hierarchy yet. |
| **ExecJob** | An ad-hoc command run on a Worker, separate from the Invocation path. Status runs `pending` → `running` → `done` \| `timeout` \| `failed`. Delivered through the same poll loop as a `kind="exec"` command. Exit code `-1` maps to `failed`, `-2` to `timeout`. |
| **Function** | The callable itself, content-addressed by `(module, qualname, version)` where `version` is the sha256 of the source. Captures the signature and, optionally, the full source for dashboard display. |

### Data and code movement

| Model | What it is |
| --- | --- |
| **Storage** | A registered S3-compatible bucket (`s3`) or scoped prefix (`s3-prefix`). The bucket / prefix / endpoint triple is **immutable after creation** — repointing would silently break every reference, so you create a new entry instead. Omitting `endpoint` means AWS; set it for Ceph, MinIO, R2, or B2. See [Storages](/get-started/storages.md). |
| **Mount** | A shared filesystem — one of exactly ten kinds: `nfs`, `samba`, `ftp`, `sftp`, `s3`, `s3fs`, `google_drive`, `dropbox`, `bind`, `configmap` — that the runner attaches into the job's workdir. **Metadata only today**: the server stores and validates the config, but runner-side activation is a daemon follow-up. See [Mounts](/get-started/mounts.md). |
| **CodeRepo** | A content-addressed snapshot of a git tree, deduped by `(namespace, remote, commit)`. Dirty trees get a synthetic commit hash and a `refs/snapshots/run-<id>` ref. The archive lands in S3 or GitHub; the row maps `(remote, commit)` → archive location so later invocations skip the upload. |

### Identity and secrets

| Model | What it is |
| --- | --- |
| **Namespace** | The tenancy boundary — every user-owned collection scopes by `namespaceId`. Namespaces sharing an `orgId` can see each other's providers and queues. Also carries `targetNymphVersion`, the OTA version daemons converge to. |
| **Token** | A per-namespace bearer credential for the CLI and SDKs. Plaintext looks like `dlk_<base64url-32-bytes>`, is shown once, and is never stored; Mongo keeps `sha256(plaintext)` plus an 8-char display prefix. Revocation is a tombstone, so the listing can still show it. |
| **EnrollToken** | A one-shot, TTL-bounded credential consumed exactly once at `/v1/daemon/hello` to enroll a new daemon into a namespace. Plaintext prefix is `dle_` — deliberately distinct from the CLI's `dlk_`. Default TTL 1 hour, hard cap 30 days. |
| **DaemonKey** | A per-host Ed25519 identity; the private key never leaves the host. Rotation runs `active` → `retiring` → `revoked`, with an overlap window so in-flight requests signed by the old key don't fail mid-rotation. See [Key rotation](/nymph/key-rotation.md). |
| **Secret** | Encrypted credential material (SSH keys, AWS keypairs, GCP service-account JSON, opaque bytes), stored as AES-256-GCM ciphertext. Plaintext never touches disk or Mongo. Referenced elsewhere by `$secret` markers. See [Auth and secrets](/api/auth-and-secrets.md). |
| **Tunnel** | A network tunnel (currently WireGuard only) the launcher brings up before reaching a Provider. Metadata-only today. See [Tunnels](/get-started/tunnels.md). |

### Audit and observability

| Model | What it is |
| --- | --- |
| **Event** | The tailable log, keyed by queue *name*, not id. The schema documents `invocation.*`, `pool.*`, and `queue.state` kinds, but the only kinds any code writes today are `invocation.progress`, `invocation.log`, and `invocation.event` — all from `POST /v1/daemon/event`. |
| **ElasticityEvent** | One row per autoscaler decision (`scale_up`, `scale_down`, `tick_noop`) with the human-readable reason and before/after signal snapshots, so you can reconstruct what the controller saw. `tick_noop` is hidden unless you pass `?verbose=true`. |
| **ResultChunk** | The durable journal of streamed result frames, one row per frame, ordered by a per-invocation `seq` cursor. Readers resume with `?since=<last seq>`. |
| **NymphRelease** | An operator-uploaded nymph binary (`lakeshore nymph push`), so debug fleets can OTA-update without the full release pipeline. See [`/dev/nymph-push`](/dev/nymph-push). |

## How dispatch flows today

```
operator                control plane                  worker host
────────                ─────────────                  ───────────

lakeshore daemon launch
  --queue training-h100 ───▶ Worker row created
                             Provider boots host

                             /hello (EnrollToken ───▶ Daemon enrols,
                              consumed, DaemonKey       registers
                              registered)               queues=["training-h100"]
                                       │
                                       ▼
                                /poll ─┘  ◀── long-poll

lakeshore exec / run
  ───▶ Invocation enqueued ───▶ Queue("training-h100")
                                       │
                                       └─ poll response  ───▶ Daemon runs
                                                                command in
                                                                runner
                                                                (process /
                                                                 docker / gvisor)
                                  ◀── /result-chunk (streamed frames)
                                  ◀── /event         (progress, logs)
                                  ◀── /ack           (terminal result)
```

The daemon-facing surface is eight POST endpoints, all under
`/v1/daemon/` and all msgpack:

| Endpoint | Purpose |
| --- | --- |
| `/hello` | Enrol: consume an EnrollToken, register the DaemonKey, upsert the Worker row. |
| `/poll` | The long-poll. Returns commands + `target_nymph_version`. |
| `/ack` | Terminal outcome of one invocation. |
| `/event` | Progress / log / event rows. |
| `/keys` | Key rotation, signed by the outgoing key. |
| `/result-chunk` | One frame of a streamed result journal. |
| `/exec/:exec_id/result` | Terminal outcome of one ExecJob. |
| `/exec/:exec_id/chunk` | One buffered slice of exec stdout. |

Three things to notice:

1. The CLI never talks to a worker directly. Every operator action
   (`exec`, `run`, `daemon update`, `daemon kill`) goes through the
   control plane.
2. The worker never accepts an inbound connection. It long-polls, and
   control commands ride *back* on the poll response. The complete set
   of command kinds is `run`, `setup`, `exec`, `hibernate`,
   `ota_update`, `reset_backoff`, and `rotate_key` — note the
   underscores on the last three. Only `run` comes from claiming an
   invocation; the others short-circuit the claim loop.
3. Only `/poll`, `/ack`, and `/event` are covered by the signature
   pre-handler. `/hello` and `/keys` verify their own signatures
   in-route, and the two exec callbacks plus `/result-chunk` are
   authenticated by the unguessable id in the URL.

The Python SDK ([`/python-sdk`](/python-sdk.md)) produces the same
Invocation envelope and rides the same dispatch path.

## Agents — not yet modeled

> **Warning:** An **agent** is a long-lived caller: it holds a session, issues many
> invocations over its life, and receives a streamed channel of messages
> rather than one terminal result. That shape does not fit the Invocation
> model, which is request/response and dies at the end.
> 
> Nothing in the schema models this yet. The design — an
> `external | managed` split mirroring Providers, explicit lifecycle
> states, and a persistent platform-owned workdir that composes with git
> mounts — is being worked out in
> [Agent lifecycle](https://docs.dreamlake.ai/dev/agent-lifecycle) and
> [DreamLake ↔ Lakeshore federation](/dev/dreamlake-federation).
> Treat both as forward-looking, not as documentation of shipped behaviour.

## State machines at a glance

| Object | States | Notes |
| --- | --- | --- |
| **Queue** | `active` · `draining` · `archived` | `paused` is in the enum but unreachable. |
| **Worker** | `pending` · `setting_up` · `joining` · `active` · `gone` | `draining` is declared but never written; `stale` is derived at read time from `lastSeenAt`. |
| **Invocation** | `queued` · `running` · `succeeded` · `failed` · `killed` · `timeout` | The last four are the terminal set. |
| **ExecJob** | `pending` · `running` · `done` · `timeout` · `failed` | |
| **DaemonKey** | `active` · `retiring` · `revoked` | `active` and `retiring` both verify; `revoked` is a hard 403. |
| **Worker host** (cloud) | `starting` · `running` · `stopping` · `stopped` · `terminated` · `unknown` | Null for local/SSH daemons. |

## Glossary

| Term | Definition |
| --- | --- |
| **Queue** | The unified scheduling primitive — a named pipe of pending Invocations. Workers subscribe by name; an empty `queues` array means the default queue. |
| **Worker** | A daemon process registered with the control plane. Carries `queues: string[]`, `runners`, and a capabilities object. |
| **Invocation** | One deferred execution, keyed by a producer-generated ULID. Inspect via `lakeshore jobs ...`. |
| **Function** | The callable, content-addressed by `(module, qualname, sha256(source))`. |
| **Provider** | A registered launch target the control plane uses to boot hosts. `launcher` is one of `SSH`, `SLURM`, `EC2`, `GCE`, `Kube`. |
| **Mode** | A server-stored named compute config: backend + runner + image + resources + env, with an optional Provider link. Managed via `lakeshore modes ...`; `.dreamrc` is a local fallback. |
| **QueueServer** | A dispatch tier (`edge` / `region` / `central`). Schema-only today — no code reads it. |
| **Daemon** | `nymph` (Rust binary, repo `dreamlake-ai/nymph`). Long-running host process that runs and monitors Invocations. Its only subcommand is `nymph tui`; bare `nymph` runs the daemon. |
| **Runner** | The execution sandbox the daemon uses for one Invocation: `process` (alias `subprocess`), `docker`, `gvisor`, `slurm`, or `kube`. An unknown kind fails the invocation rather than falling back. |
| **Storage** | A registered S3-compatible bucket or prefix. Used for artifact upload, code shipping, and large-payload spill. Its S3 target is immutable after creation. |
| **Mount** | A registry entry for a shared filesystem the runner attaches into the workdir. Metadata-only today. |
| **CodeRepo** | A versioned code snapshot pushed via `lakeshore code push`, deduped by `(namespace, remote, commit)`. Daemons pull it at invocation time so your working tree arrives without manual rsync. |
| **Namespace** | The tenancy boundary. Namespaces sharing an `orgId` can see each other's providers and queues. |
| **Lakeshore** | The control plane. Single Node + TS + Fastify + Prisma service over Mongo, with Redis for result streaming and S3 for blobs. |

## Read next

| I want to… | Page |
| --- | --- |
| Learn the queue model | [Queues](/get-started/queues.md) |
| Understand autoscaling | [Elasticity](/get-started/elasticity.md) |
| See all CLI commands | [CLI](/cli.md) |
| Try runnable examples | [CLI examples](/cli/examples.md) |
| Launch daemons via providers | [Daemon launch](/nymph/daemons.md) |
| Understand the wire protocol | [Daemon protocol](/nymph/protocol.md) |
| Explore the control plane internals | [Lakeshore](/dev/controlplane-internals) |
| Read the queue design note | [Queue design](/dev/queues) |
| Read the payload store design | [Payload store](/dev/payload-store) |

---

Source: https://lakeshore.dreamlake.ai/get-started/tutorials

# Tutorials

> Hands-on walkthroughs covering the core Lakeshore workflow. Each one is
> self-contained — work through them in order or jump to the one that
> matches what you need.

| I want to… | Tutorial | Time |
| --- | --- | --- |
| Run my first queue round-trip | [Hello UDF](/get-started/tutorials/hello-udf.md) | ~5 min |
| Scale to many invocations in parallel | [Fan out](/get-started/tutorials/fan-out.md) | ~10 min |
| Declare a cluster in YAML | [Compose a cluster](/get-started/tutorials/compose.md) | ~10 min |
| Exercise every operator surface | [End-to-end smoke](/get-started/tutorials/end-to-end.md) | ~30 min |

Start with **Hello UDF** if this is your first time with the Python SDK —
it runs with zero infrastructure. Move to **Fan out** once a single
invocation is comfortable. **End-to-end smoke** is the operator
checklist: providers, daemons, queues, jobs, and completion.

## Runnable examples

The full set of runnable examples lives in the
[`lakeshore-examples`](https://github.com/dreamlake-ai/lakeshore-examples)
repo. Each directory is self-contained; scripts carry PEP 723 inline
metadata so `uv run <script>.py` resolves their dependencies.

| Example | What it covers |
| --- | --- |
| [`01-hello-ssh`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/01-hello-ssh) | SSH provider basics |
| [`02-hello-local-python`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/02-hello-local-python) | Local function dispatch |
| [`05-pipeline-iteration`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/05-pipeline-iteration) | Chained stages over one plane |
| [`08-queues-cpu-vs-gpu`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/08-queues-cpu-vs-gpu) | Mixed CPU + GPU fleet from one `lakeshore.yaml` |
| [`09-hello-queue-udf`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/09-hello-queue-udf) | Queue-bound `@udf`, submit → id → result |
| [`10-fan-out-gather`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/10-fan-out-gather) | Fan-out, streaming, and a two-stage pipeline |
| [`14-compose-cluster`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/14-compose-cluster) | A 30-machine compose cluster with mounts and elasticity |

Older examples — EC2 launch, GPU clusters, code mounts, storage access,
compose runtime — live under
[`_legacy/`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/_legacy)
and predate the current SDK surface. Read them for shape, not for
copy-paste.

## Related happy paths

The [CLI](/cli/happy-paths.md), [Python SDK](/python-sdk/happy-paths.md), and
[admin](/admin/happy-paths.md) sections each carry numbered happy paths — a
narrower "run these commands, expect this output" format that pairs well
with these tutorials.

---

Source: https://lakeshore.dreamlake.ai

# Lakeshore

Lakeshore runs Python functions somewhere other than the machine in
front of you. You decorate a function with `@udf`, call it, and the
result comes back — Lakeshore packs the arguments, hands the work to a
queue, and a worker somewhere else runs it and returns the value.

You'd reach for it when the work doesn't fit locally: a GPU job, a long
sweep, a fan-out of identical calls. You write Python; Lakeshore owns
the queueing, the transport, and the worker side.

> **Note:** Material that describes Lakeshore *as part of DreamLake* has moved to the
>   **Lakeshore tab** at
>   [docs.dreamlake.ai/lakeshore](https://docs.dreamlake.ai/lakeshore) — the
>   provider pages, the Python architecture overview, and the newer pages on
>   declaring access, mounting storage, host setup and queues.
> 
>   This site remains the reference for the SDK, the CLI, the daemon, and the
>   control-plane API. Old URLs redirect, so nothing you have bookmarked breaks.

## Install

Pick the surface you are starting from — each page below covers the
authentication and first-run check for that one.

| You are… | Install | Then read |
| --- | --- | --- |
| Writing Python | `pip install dreamlake-lakeshore` (0.3.7) | [Python SDK](/python-sdk.md) |
| Operating a fleet | `npm i -g @dreamlake/lakeshore` (0.2.0) | [CLI installation](/cli/installation.md) |
| Bringing up a compute host | normally installed *through* the CLI — `lakeshore daemon install <ssh-alias>` | [Daemon installation](/nymph/installation.md) |

The CLI is optional for the local tier and required once you manage
providers, daemons, or a hosted control plane. See
[Release notes](/cli/release-notes.md) for the CLI changelog.

The CLI and the SDK each need a **server URL**, a **namespace**, and a
**token**. Get them from whoever runs your control plane, or stand one
up yourself with [Admin setup](/admin/setup).

## Two tiers, one decorator

A bare `@udf` is a **local** call — an ordinary in-process function call
with a run context wrapped around it. Nothing touches the network, and
the HTTP/msgpack stack is never even imported.

```python file="hello.py"
import dreamlake.lakeshore as dls

@dls.udf
def stats(xs: list[float]) -> dict:
    return {"n": len(xs), "mean": sum(xs) / len(xs)}

print(stats([1.0, 2.0, 3.0]))     # {'n': 3, 'mean': 2.0} — runs here
```

Adding `queue=` makes the same function **remote**. A plain call now
submits to that queue and blocks for the result; `.submit()` returns the
invocation id instead, so you can walk away and reconnect later.

```python file="remote.py"
import dreamlake.lakeshore as dls

@dls.udf(queue="cpu")
def double(x: int) -> int:
    return x * 2

print(double(21))                 # 42 — submit, wait, unpack
inv_id = double.submit(21)        # a durable string id; no Future type
```

Point the SDK at a control plane with `LAKESHORE_URL` (plus
`LAKESHORE_CLIENT_TOKEN` and `LAKESHORE_NAMESPACE` when the plane
requires auth). With `LAKESHORE_URL` unset, submits go to a local
SQLite plane under `LAKESHORE_HOME` (default `~/.lakeshore`) — handy for
trying the remote tier without any server at all.

Something has to drain the queue. The simplest drainer is a native
Python worker on your own machine:

```bash
lakeshore worker start --url http://localhost:8080 --queue cpu
```

> **Note:** `lakeshore worker` deliberately does not read the credentials written by
> `lakeshore auth login`. The plane a worker drains must come from `--url`
> or `LAKESHORE_URL`, so a stray login can never redirect a worker.

## The three pieces

```text
your process                 control plane                  worker host
────────────                 ─────────────                  ───────────

 @udf call    ──submit──▶   queue: "cpu"     ◀──poll──   nymph daemon
 result       ◀──result──   invocation done   ──ack───▶   nymph daemon
```

| Piece | What it is | Where it runs |
| --- | --- | --- |
| `lakeshore` CLI | Node binary on npm. Providers, secrets, queues, daemons. | Your laptop |
| Control plane | Fastify 5 + Prisma + MongoDB. Owns the registry and the dispatch loop. | A server (Heroku today) |
| `nymph` daemon | Rust binary. Long-polls the control plane, runs invocations. | The compute host |

Every wire call is daemon-initiated and outbound. The control plane
never opens a connection to a worker, so a worker needs no inbound port,
no port forward, and no firewall change.

## Where to go next

| You want to… | Start here |
| --- | --- |
| Run your first function | [Quick start](/get-started/quick-start.md) |
| Install the CLI and connect a provider | [CLI installation](/cli/installation.md) |
| Understand what's running where | [Architecture](/get-started/architecture.md) |
| Follow a longer tutorial | [Tutorials](/get-started/tutorials.md) |
| Understand queues and scheduling | [Queues](/get-started/queues.md) |
| Run a fleet daemon on a remote host | [Install a remote daemon](/nymph/install-remote-daemon.md) |

---

Source: https://lakeshore.dreamlake.ai/nymph/installation

# Install nymph

`nymph` is the Lakeshore worker daemon: a single static Rust binary that
long-polls the control plane, runs the work it is handed, and reports
back. It needs no inbound port — everything is outbound HTTPS.

Once installed, running the bare binary (no subcommand) *is* the daemon.
The only subcommand is [`nymph tui`](/nymph/status-tui.md).

## One-liner

```bash
curl -fsSL https://dreamlake.ai/dreamlake/nymph/install.sh | bash
```

`scripts/install.sh` does exactly four things:

1. Maps `uname -s` → `unknown-linux-gnu` / `apple-darwin` and `uname -m`
   → `x86_64` / `aarch64`. Any other OS or arch is a hard error.
2. Downloads `${BASE_URL}/${VERSION}/nymph-${arch}-${os}.tar.gz` and its
   `.sha256` sidecar.
3. Verifies the checksum with `sha256sum -c` (or `shasum -a 256 -c` on
   macOS).
4. Untars and moves `nymph` into `/usr/local/bin`, escalating with
   `sudo` only when that directory is not writable.

| Flag / env       | Default                                                            | Meaning                             |
| ---------------- | ------------------------------------------------------------------ | ----------------------------------- |
| `--version <v>`  | `latest`                                                           | Release directory to pull from.     |
| `--dest <dir>`   | `/usr/local/bin`                                                   | Install directory.                  |
| `NYMPH_BASE_URL` | `https://pub-c0109e197b4a4d1abe5884ac4dd3a023.r2.dev/dreamlake/nymph` | Release bucket base URL. |

```bash
curl -fsSL https://dreamlake.ai/dreamlake/nymph/install.sh | bash -s -- --version 0.1.4
curl -fsSL https://dreamlake.ai/dreamlake/nymph/install.sh | bash -s -- --dest ~/.local/bin
```

> **Warning:** The install script installs a binary and nothing else — there is no
> `--service` flag and no `NYMPH_PREFIX` variable. Running nymph under a
> service manager is covered in [Run it as a service](#run-it-as-a-service)
> below.

### Published targets

CI builds on a `v*` tag push for exactly three targets:

- `x86_64-unknown-linux-gnu`
- `aarch64-unknown-linux-gnu`
- `aarch64-apple-darwin`

Intel macOS (`x86_64-apple-darwin`) was dropped from CI. Each target
ships a `nymph-<target>.tar.gz` (what `install.sh` consumes) *and* a raw
`nymph-<target>` binary (what the OTA path consumes), each with a
`.sha256` sidecar, under `dreamlake/nymph/<version>/` in the release
bucket.

## Build from source

```bash
git clone https://github.com/dreamlake-ai/nymph.git
cd nymph
cargo build --release      # target/release/nymph
```

The crate is `publish = false` — it is never on crates.io. The release
profile is size-tuned (`opt-level = "z"`, fat LTO, symbols stripped,
`panic = "abort"`).

The `ota-self-replace` feature is **on by default**: after a successful
[OTA update](/nymph/protocol.md#ota_update) the daemon `exec()`s into the
freshly installed binary. Build with `--no-default-features` to install
the new binary but keep the old process running until you restart it
yourself.

## Point it at a control plane

The daemon reads one TOML file at boot and never re-reads it. With no
`--config`, it tries these in order:

1. `/etc/dreamlake/udf-daemon.toml`
2. `$HOME/.config/dreamlake/udf-daemon.toml`
3. built-in defaults (`http://localhost:8080`, `process` runner)

A minimal working config:

```toml file="/etc/dreamlake/udf-daemon.toml"
[server]
url = "https://api.lakeshore.dreamlake.ai"
token_file = "/etc/dreamlake/token"
namespace = "default"

[daemon]
label = "bos14-login"
tags = ["gpu:h100"]

[runtime]
workdir = "/var/lib/dreamlake/udf"
runners = ["process"]
default_runner = "process"
keep_alive_s = -1
```

> **Warning:** The top-level config struct is `#[serde(deny_unknown_fields)]`. A stray
> section — `[relay]`, or a bare top-level key like `server_url = "…"` —
> fails the parse and the daemon refuses to boot. The nine valid sections
> are `[server]`, `[daemon]`, `[runtime]`, `[poll]`, `[log]`,
> `[shutdown]`, `[mounts]`, `[introspect]`, `[identity]`. Unknown keys
> *inside* a valid section are silently ignored.

Then start it:

```bash
nymph                                    # uses the default config search path
nymph --config /etc/dreamlake/udf-daemon.toml
```

### Daemon flags and env vars

| Flag              | Env var             | Default     | Meaning                                          |
| ----------------- | ------------------- | ----------- | ------------------------------------------------ |
| `--config <path>` | `LAKESHORED_CONFIG` | search path | Path to `udf-daemon.toml`.                        |
| `--server <url>`  | `LAKESHORE_SERVER`  | from config | Control-plane base URL.                           |
| `--token <tok>`   | `LAKESHORE_TOKEN`   | from config | Bootstrap bearer used on `/hello`.                |
| `--queue <name>`  | `DREAMLAKE_QUEUE`   | `default`   | Queue to subscribe to. A trailing `*` is a prefix match. |
| `--worker-id <id>`| —                   | derived     | Override the provisional `{machine_id}-{ULID}` id. |

Two more env vars affect logging only: `RUST_LOG` (full `EnvFilter`
syntax, wins over `[log] level`) and `LAKESHORE_LOG_FORMAT` (`human` or
`json`, wins over `[log] format`). See
[Telemetry and introspection](/nymph/telemetry.md).

> **Note:** It is `LAKESHORED_CONFIG`, not `LAKESHORE_CONFIG`. The queue override is
> `DREAMLAKE_QUEUE`, not `LAKESHORE_QUEUE`.

## Verify it registered

```bash
lakeshore daemon list
```

The daemon posts `/v1/daemon/hello` on boot, then long-polls. It shows
up as `joining` on the hello and flips to `active` on its first poll.
See [Daemon lifecycle](/nymph/daemon-lifecycle.md) for the full operator
surface.

## Run it as a service

`install.sh` does not write a unit file. Two supported paths:

- **Control-plane launch.** `lakeshore daemon launch` renders a
  cloud-init script that installs the binary at `/opt/dreamlake/bin/nymph`,
  writes `/etc/dreamlake/udf-daemon.toml`, and installs + enables a
  `dreamlake-nymph.service` systemd unit (`Restart=always`,
  `KillMode=mixed`, `TimeoutStopSec=30`, logs appended to
  `/var/log/dreamlake/nymph.log`). See
  [Daemon lifecycle → launch](/nymph/daemon-lifecycle.md#launch-through-a-provider).
- **SSH bootstrap.** `lakeshore daemon install <alias>` starts the
  daemon under `setsid nohup` (or `sbatch` with `--slurm`) in the SSH
  user's `$HOME` — no systemd, no sudo. See
  [Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md).

For a hand-rolled unit, mirror the launch path: run as a non-root user
that owns the binary and the workdir, `KillMode=mixed` so `SIGTERM`
reaches the daemon directly (it does its own
[graceful drain](/nymph/daemons.md#shutdown-and-drain)), and a
`TimeoutStopSec` comfortably above `[shutdown] grace_s`.

## Where to next

| I want to…                                   | Page                                                              |
| -------------------------------------------- | ----------------------------------------------------------------- |
| Bootstrap nymph on a host over SSH            | [Bootstrap a remote daemon](/nymph/install-remote-daemon.md)         |
| Understand the daemon's runtime and runners   | [Daemons](/nymph/daemons.md)                                          |
| Manage daemons from the CLI                   | [Daemon lifecycle](/nymph/daemon-lifecycle.md)                        |
| Read the wire protocol                        | [Daemon protocol](/nymph/protocol.md)                                 |
| Turn on keypair auth                          | [Daemon identity and key rotation](/nymph/key-rotation.md)            |
| Watch a local daemon live                     | [`nymph tui`](/nymph/status-tui.md)                                   |
| Push a debug build without a release          | [Nymph push (dev)](/dev/nymph-push)                                |

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths

# Happy Paths — Python SDK

Five paths covering the SDK surface end to end, from the leaf-node
"does the decorator work" through "your local code runs on a remote
worker that reads storage it holds no credentials for."

Walk top-down. Some paths depend on CLI or Operator paths being green
first — noted on each page.

## The list

| # | Path | Verifies |
| - | ---- | -------- |
| 01 | [Hello, local Python](/python-sdk/happy-paths/01-hello-local-python.md) | SDK install + `@udf` decorator (no network) |
| 05 | [Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md) | Submit → claim → execute → result round-trip |
| 06 | [Fan out and collect](/python-sdk/happy-paths/06-fan-out-gather.md) | Parallelism + ordered collection by invocation id |
| 10 | [Code mount](/python-sdk/happy-paths/10-code-mount.md) | Local git tree → S3 → worker import |
| 11 | [Storage access from a UDF](/python-sdk/happy-paths/11-storage-from-udf.md) | Reading and writing a named storage entry with no static credentials |

## Cross-surface prereqs

| For path | You also need green |
| -------- | ------------------- |
| 05, 06 (remote plane), 10, 11 | [03 · Hello, local stack](/admin/happy-paths/03-hello-local-stack.md) (Operator) |
| 10, 11 | [09 · Storage round-trip](/cli/happy-paths/09-storage-round-trip.md) (CLI) |

## The cascade

`01` is the leaf — no infrastructure, no network. Pass that first, then
`05` gets the submit-and-wait round-trip working (on the local SQLite
plane it needs no server either). `06` proves parallelism and wants two
workers. `10` adds the code-archive path; `11` adds the object-storage
path. Both of those need a control plane. Once all five are green, the
SDK is solid.

For a faster smoke test with no infrastructure at all, run the SDK
repo's `examples/00_hello_udf.py` through `examples/06_dynamic_graph.py`
— they all use `Dispatch(":memory:")`.

→ Switch surface: [CLI](/cli/happy-paths.md) · [Operator](/admin/happy-paths.md)

---

Source: https://lakeshore.dreamlake.ai/cli/examples/auth

# CLI examples — Auth and setup

The first stop in the [CLI examples cookbook](/cli/examples.md). Run the
shell setup block once per shell, then save a token with `auth login`
so the rest of the cookbook is paste-ready.

## Setup (run once per shell)

```bash
# Where the control plane lives. Override for staging / prod.
export LAKESHORE_URL=http://localhost:8080
export LAKESHORE_NAMESPACE=default

# Seed a daemon id into $D for examples that take an <id>. Once any
# daemon is registered, this picks the first one. Re-run after launch.
D=$(lakeshore daemon list --json | jq -r '.[0].id // empty')
echo "using daemon: $D"
```

`$D` is referenced in any block that needs a daemon id. If `daemon list`
is empty when you start, jump to
[Daemons](/cli/examples/daemons-and-exec.md), bring one up, then re-run
the `D=` line.

> **Warning:** With `LAKESHORE_URL` exported, `resolveServer()` takes that path and
> **does not read the saved auth file** — it sends no bearer token at
> all, so any `auth login` token is ignored. That is correct for an
> open-mode local control plane and wrong for a token-enforced
> deployment. For the latter, either skip the `LAKESHORE_URL` export and
> rely on the saved login, or `unset LAKESHORE_URL` after running
> `lakeshore auth login`.

## Auth (one-time login)

The `auth` commands are the only ones that read `LAKESHORE_SERVER`.
Their server precedence is `--server` → `LAKESHORE_SERVER` → the saved
auth file → `http://localhost:8080`.

### `auth login` — save a token

Prompts for a bearer token on a TTY (omit `--token`), validates it
against `GET <server>/v1/namespaces/<ns>/whoami`, then writes
`~/.config/lakeshore/auth.yml` with mode 600.

```bash
lakeshore auth login --server "$LAKESHORE_URL" --namespace default
lakeshore auth login --server "$LAKESHORE_URL" --namespace default --token dlk_xxxx
```

Nothing is saved if `whoami` rejects the token.

### `auth login` — non-interactive (admin-mint via env)

When `LAKESHORE_ADMIN_TOKEN` is exported and `--token` is absent, the
CLI mints a fresh per-namespace token on the server and saves that.
`--name` sets the label base; the CLI appends a timestamp so re-running
doesn't collide with the unique-name constraint.

```bash
LAKESHORE_ADMIN_TOKEN=changeme lakeshore auth login \
  --server "$LAKESHORE_URL" \
  --namespace default \
  --name laptop
```

### `auth status` — show and revalidate

Prints the config path, server, namespace, token prefix and label, last
used timestamp, and `authMode`, after re-hitting `whoami`. Exits 1 when
there is no saved auth or the revalidation fails.

```bash
lakeshore auth status
lakeshore auth status --server https://api.lakeshore.dreamlake.ai --namespace other-ns
```

```text
Auth OK.
  config path: /Users/you/.config/lakeshore/auth.yml
  server:      http://localhost:8080
  namespace:   default
  token:       dlk_abcd... (laptop-2026-01-02T03-04-05)
  lastUsedAt:  2026-01-02T03:05:11.000Z
  authMode:    token
```

`authMode` is one of `admin`, `token`, or `open`. `open` means the
control plane is running without token enforcement.

### `auth logout` — drop the saved token

Deletes `~/.config/lakeshore/auth.yml`. The next CLI invocation needs
either `LAKESHORE_URL` set (open mode) or another `auth login`.

```bash
lakeshore auth logout
```

## Minting tokens for other people

Only an admin can. `lakeshore admin ...` hard-fails with exit 1 unless
`LAKESHORE_ADMIN_TOKEN` is set.

```bash
export LAKESHORE_ADMIN_TOKEN=<admin-token>

lakeshore admin tokens create alice        # label defaults to <$USER>-<YYYY-MM-DD>
lakeshore admin tokens list                # alias: ls
lakeshore admin tokens revoke <token-id>

lakeshore admin namespace create team-a    # group alias: ns
lakeshore admin namespace list             # alias: ls
lakeshore admin namespace delete team-a    # alias: rm
```

`tokens create` prints the plaintext **once** along with a ready-made
`lakeshore auth login` line for the recipient. None of the `admin`
subcommands take flags.

## Read next

- [Installation](/cli/installation.md) — install plus the full env-var table.
- [Providers + discover](/cli/examples/providers-and-discover.md) — the next step.
- [Auth and secrets](/api/auth-and-secrets.md) — the server-side model.

---

Source: https://lakeshore.dreamlake.ai/get-started/queues

# Queues

A **Queue** is the unified scheduling primitive in Lakeshore. It is a
named pipe of pending work; daemons subscribe to it by name; the control
plane hands each queued Invocation to whichever subscribed daemon claims
it first.

There is exactly one primitive at the scheduling layer. Workers carry a
`queues: string[]` array recording which queues they pull from, and the
**empty array means the default queue**. Everything else — elasticity,
admission, work-stealing — is a policy field on the queue row, not a
second primitive.

```
       producer                control plane                  workers
       ────────                ─────────────                  ───────

   submit(invocation)
   queue=training-h100  ───▶   ┌──────────────┐
                               │   training-  │  ◀── poll ──  daemon-A (queues=["training-h100"])
                               │     h100     │
                               │              │  ◀── poll ──  daemon-B (queues=["training-h100","eval"])
                               └──────────────┘
                               ┌──────────────┐
                               │     eval     │  ◀── poll ──  daemon-B
                               └──────────────┘
                               ┌──────────────┐
                               │   default    │  ◀── poll ──  daemon-C (queues=[])
                               └──────────────┘
```

Queue names must match `^[a-z0-9][a-z0-9-]*$`.

## What is on a queue row

| Field | What it holds |
| --- | --- |
| `kind` | Scheduling kind — `fifo` (default), `priority`, `boltzmann`, `filo`. |
| `kindParams` | Kind-specific knobs, e.g. `{ temperature: 0.5 }` or `{ priority_field: "p" }`. |
| `elasticity` | The autoscaling spec. See [Elasticity](/get-started/elasticity.md). |
| `admission` | `{ max_depth?, on_full, rate_limit?, deadline_cutoff_s? }` where `on_full` is `reject` or `block`. |
| `providerRef` | The Provider the elasticity controller launches new daemons under. |
| `daemonTemplate` | `{ instance_type?, image_id?, runner? }` used when the controller launches. |
| `state` | `active`, `draining`, or `archived`. |

> **Warning:** All four `kind` values are accepted and stored, but the dispatcher's
> claim always sorts `{ priority: -1, submittedAt: 1 }` regardless of the
> queue's kind. `boltzmann` and `filo` do not change dispatch order today,
> and `priority` is really "priority field on the invocation, FIFO within
> a tier" — it is not derived from `kindParams`. Likewise `admission` is
> validated and stored but **not enforced** at admit time.

## Membership

A daemon's queue membership lives on the **worker** row, not the queue
row — `Worker.queues: string[]` is the source of truth. `queues show`
walks the workers and reports who is pulling from a queue right now.

**Default-queue membership is the empty array.** A daemon launched
without `--queue` lands in the default queue. There is no sentinel
string written into `Worker.queues` by the system. (The membership
lookup does also accept a literal `"default"` entry if you put one
there, but the explicit add/remove routes reject `default` — the empty
array is the contract.)

## Lifecycle

```bash
lakeshore queues drain training-h100      # stop admitting; finish in-flight
lakeshore queues archive training-h100    # soft-delete, keep the row
lakeshore queues unarchive training-h100  # back to active (reverses both)
lakeshore queues rm training-h100         # hard delete
```

`rm` refuses with a 409 while the queue has members; `--force` cascades
by scrubbing the name out of every `Worker.queues` array first.

The schema also declares a `paused` state, but no route can set it —
`active`, `draining`, and `archived` are the three reachable values.

## The default queue

You do not have to create it. The control plane materialises a `default`
row on the first `queues ls` or `queues show default` in a namespace,
with `kind: "fifo"` and the description "Default queue (auto-created).
Cannot be archived or deleted." It is a reserved name: `archive` and
`rm` both return 409.

```bash
lakeshore daemon launch --provider my-ec2 --name scratch --wait
lakeshore daemon show scratch     # queues: []  — i.e. the default queue
```

## Organization visibility

When namespaces are grouped by a shared `orgId`, `queues ls` aggregates
queues from every namespace in the org and stamps a `namespace` field on
each row so you can tell who owns it. Two namespaces in one org may hold
same-named queues; they are distinct rows scoped to their own namespace.

A token scoped to one namespace in the org also authorises requests
against sibling namespaces — put the sibling slug in the URL path:

```bash
curl -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/bob/queues/inference
```

> **Note:** `--dreamrc <path>` is the CLI's only program-level option. The namespace
> the CLI targets comes from `LAKESHORE_NAMESPACE` (when `LAKESHORE_URL`
> is set) or from `~/.config/lakeshore/auth.yml`. To reach a sibling
> namespace, either re-login against it or call the HTTP route directly.

## A worked example

Spin up two queues — one for CPU pre-processing, one for GPU training —
and a few daemons for each.

```bash
# Create the queues. Both fixed for now.
lakeshore queues add cpu-prep      --kind fifo     --elasticity fixed
lakeshore queues add training-h100 --kind priority --elasticity fixed

# Launch daemons into each. --queue is repeatable.
lakeshore daemon launch --provider my-ec2 --queue cpu-prep      --count 4 --name prep  --wait
lakeshore daemon launch --provider my-gpu --queue training-h100 --count 2 --name train --wait

# What's where?
lakeshore queues ls
lakeshore queues ls --state active
lakeshore daemon list
lakeshore queues show training-h100    # members + full queue row
lakeshore queues stats training-h100   # daemon_count / utilization / pending
```

Edit membership on a running daemon — no relaunch needed. The daemon
picks the change up on its next poll:

```bash
lakeshore daemon update train-00 --add-queue eval
lakeshore daemon update train-00 --remove-queue eval
```

Patch a queue in place:

```bash
lakeshore queues patch training-h100 --max-depth 1000 --on-full reject
lakeshore queues patch training-h100 --description "h100 production"
```

## CLI surface

The `queues` group uses two irregular verbs — `ls` (not `list`) and `rm`
(not `remove`). Everything else in the CLI uses the regular forms.

| Command | Notable flags |
| --- | --- |
| `queues add <name>` | `--description`, `--kind`, `--provider`, `--elasticity`, `--min`, `--max`, `--threshold`, `--instance-type`, `--image-id`, `--runner`, `--max-depth`, `--on-full`, `--json` |
| `queues ls` | `--state <active\|draining\|paused\|archived>`, `--json` |
| `queues show <name>` | `--json` |
| `queues patch <name>` | `--description`, `--provider` (pass `""` to clear), `--min`, `--max`, `--threshold`, `--max-depth`, `--on-full`, `--json` |
| `queues archive` / `unarchive` / `drain` `<name>` | `--json` |
| `queues rm <name>` | `--force` |
| `queues events <name>` | `--verbose`, `--limit <n>` (default 50, max 200), `--json` |
| `queues stats <name>` | `--json` |

Note that `patch` cannot change `--kind` or `--elasticity <kind>`. To
change the elasticity policy itself, PATCH the queue's `elasticity`
object over HTTP, or recreate the queue.

## Shared pools

Multiple queues can share one worker fleet by carrying the same `pool`
name inside their `elasticity` object. Workers in the pool can claim
work from any member queue, and the controller scales the pool as a
single unit.

`pool`, `match`, `steal_delay_s`, and `cost_weight` have **no CLI
flags** — set them by PATCHing the queue's `elasticity` object directly:

```bash
curl -X PATCH \
  -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"elasticity":{"kind":"fully_elastic","max":8,"pool":"mixed"}}' \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/$NS/queues/cpu-prep
```

See [Elasticity — shared pools](/get-started/elasticity.md#shared-pools)
for the semantics.

## HTTP

```
POST   /v1/namespaces/:ns/queues
GET    /v1/namespaces/:ns/queues[?state=…]
GET    /v1/namespaces/:ns/queues/:name
PATCH  /v1/namespaces/:ns/queues/:name
POST   /v1/namespaces/:ns/queues/:name/archive
POST   /v1/namespaces/:ns/queues/:name/unarchive
POST   /v1/namespaces/:ns/queues/:name/drain
DELETE /v1/namespaces/:ns/queues/:name[?force=true]
GET    /v1/namespaces/:ns/queues/:name/events[?verbose=true&limit=…]
GET    /v1/namespaces/:ns/queues/:name/stats
POST   /v1/namespaces/:ns/workers/:id/queues
```

An unknown namespace on any of these is a 404 — the queue routes do not
auto-create namespaces the way the secrets / providers / modes routes do.

## Read next

- [Elasticity](/get-started/elasticity.md) — the four scaling policies you
  can declare on a queue.
- [Scaling rules](/get-started/scaling-rules.md) — the rule book the
  controller follows on every tick.
- [Jobs](/get-started/jobs.md) — inspecting the invocations queued here.
- [Compose](/get-started/compose.md) — declarative queues + daemons in
  `lakeshore.yaml`.
- [Python SDK](/python-sdk.md) — the producer side; `@udf(queue="…")` lands
  on the queues you create here.
- [`/dev/queues`](/dev/queues) — the developer-side design note.

---

Source: https://lakeshore.dreamlake.ai/get-started/quick-start

# Quick start

Run a Python function through a queue and get the result back. The first
half needs nothing but Python; the second half points the same code at a
control plane.

To host your own control plane, follow [Setting Up Lakeshore Service](https://docs.dreamlake.ai/lakeshore/setting-up-lakeshore-service).

## Before you begin

- **Python 3.11+** — `python --version`
- Nothing else, for Part 1.

> **Note:** A bare `@udf` runs in-process, and queue-bound UDFs fall back to a local
> SQLite dispatch plane under `~/.lakeshore` whenever `LAKESHORE_URL` is
> unset. Steps 1–4 need nothing installed but Python; a control plane only
> enters the picture at Step 5.

## Step 1 — Install the SDK

```bash
pip install dreamlake-lakeshore
```

```python
import dreamlake.lakeshore as dls
```

Imports are lazy: a purely local run never pulls in httpx or msgpack.

## Step 2 — A local function

A bare `@udf` is a plain in-process call. No queue, no worker, no
network:

```python file="local.py"
import dreamlake.lakeshore as dls

@dls.udf
def add(a: int, b: int) -> int:
    return a + b

print(add(2, 3))    # 5
```

## Step 3 — A queue round-trip, with zero infrastructure

Bind the function to a queue and it submits instead of running. The
handle you get back is the **invocation id** — a plain string, not a
Future.

```python file="hello_udf.py"
import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

def main():
    dp = dls.Dispatch(":memory:")             # the minimal dispatch plane
    q = dls.SyncQueue("hello", dispatch=dp)

    inv = add.submit(2, 3, _dispatch=dp)      # -> "01JZ…"
    print("invocation:", inv)

    dls.run_worker(q, once=True)              # claim + execute, in-process
    print("result:", q.result(inv, timeout=10))

if __name__ == "__main__":
    main()
```

```bash
python hello_udf.py
```

```text
invocation: 01JZ…
result: 5
```

`dls.Dispatch` is a single SQLite store — no control plane, no daemon,
no network. Swap it for a real plane later and none of the decorated
code changes.

## Step 4 — Fan out

There is no `gather()` helper, because there is no Future type. A list
comprehension is the spawn and a loop is the barrier:

```python file="fan_out.py"
import asyncio, threading
import dreamlake.lakeshore as dls

@dls.udf(queue="fan-out-demo")
def square(x: int) -> int:
    return x * x

async def fan_out(n: int, dp) -> list[int]:
    q = dls.Queue("fan-out-demo", dispatch=dp)          # the async surface
    ids = [square.submit(i, _dispatch=dp) for i in range(n)]
    return await asyncio.gather(*(q.result(i, timeout=30) for i in ids))

def main():
    dp = dls.Dispatch(":memory:")
    done = threading.Event()
    threading.Thread(
        target=dls.run_worker,
        args=(dls.SyncQueue("fan-out-demo", dispatch=dp),),
        kwargs={"stop": done.is_set},
        daemon=True,
    ).start()

    print(asyncio.run(fan_out(10, dp)))
    done.set()

if __name__ == "__main__":
    main()
```

`asyncio.gather` over the awaitable `Queue.result` is the join. Results
come back in submission order.

## Step 5 — Point it at a control plane

Two things change: where the dispatch plane lives, and who runs the
worker.

### Authenticate

```bash
lakeshore auth login --server https://api.lakeshore.dreamlake.ai --namespace <your-namespace>
```

That writes `~/.config/lakeshore/auth.yml` (mode 600) with `server`,
`namespace`, and `token`. In a notebook, use environment variables
instead:

```python
import os
os.environ["LAKESHORE_URL"] = "https://api.lakeshore.dreamlake.ai"
os.environ["LAKESHORE_NAMESPACE"] = "<your-namespace>"
os.environ["LAKESHORE_CLIENT_TOKEN"] = "<your-token>"
```

With `LAKESHORE_URL` set, the SDK's default dispatch plane becomes
`HttpDispatch` against that control plane, and your code needs no
`_dispatch=` argument at all.

### Run a worker

```bash
lakeshore worker start --queue hello --url https://api.lakeshore.dreamlake.ai --token <token>
```

> **Warning:** `lakeshore daemon …` manages the Rust fleet daemon, which supervises
> shell commands and invocations from the control plane. Queue-bound
> Python UDFs need `lakeshore worker start`, which spawns
> `python -m dreamlake.lakeshore.daemon`. The worker deliberately ignores
> saved `auth login` credentials — pass `--url` / `--token`, or set
> `LAKESHORE_URL` and `LAKESHORE_CLIENT_TOKEN`.

### Submit

```python
import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

inv = add.submit(2, 3)                 # uses LAKESHORE_URL / auth.yml
q = dls.SyncQueue("hello")
print(q.result(inv, timeout=60.0))     # 5
```

## Step 6 — The CLI for fleet management

The SDK submits work. The CLI manages the fleet:

```bash
lakeshore queues ls           # queues in your namespace
lakeshore daemon list         # registered daemons
lakeshore jobs list           # invocations the control plane is tracking
```

See [CLI installation](/cli/installation.md) to install and log in.

## Where to next

| I want to… | Page |
| --- | --- |
| Understand `@udf` in depth | [`@udf` decorator](/python-sdk/udf.md) |
| Work through it as a tutorial | [Hello UDF](/get-started/tutorials/hello-udf.md) |
| Understand queues | [Queues](/get-started/queues.md) |
| Move data in and out | [Storages](/get-started/storages.md) |
| Declare a cluster in YAML | [Compose](/get-started/compose.md) |
| Set up cloud providers | [Providers](https://docs.dreamlake.ai/lakeshore/providers) |
| Auto-scale workers | [Elasticity](/get-started/elasticity.md) |
| Full CLI reference | [CLI](/cli.md) |

---

Source: https://lakeshore.dreamlake.ai/get-started/tutorials/hello-udf

# Hello UDF

> **Example code:** [`09-hello-queue-udf`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/09-hello-queue-udf)

Decorate a Python function with `@udf`, bind it to a queue, submit it,
and get the result back by invocation id. Part 1 needs nothing but
Python; Part 2 points the same code at a control plane.

## Prerequisites

| Requirement | Why |
| --- | --- |
| Python 3.11+ | `dreamlake-lakeshore` requires 3.11 or later. |
| `pip install dreamlake-lakeshore` | Ships the `@udf` decorator, queues, and the local dispatch plane. |

Verify the install:

```bash
python -c "import dreamlake.lakeshore as dls; print(dls.__version__)"
```

## Step 1 — Write a `@udf` function

Create `hello_udf.py`:

```python file="hello_udf.py"
import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b
```

Two things to notice:

1. `queue=` takes a **string**, not a `Queue` object. The decorator does
   not create the queue and does not talk to a server at import time.
2. Because the UDF is queue-bound, a plain call `add(2, 3)` **submits
   and waits**. To get the id without waiting, call `add.submit(2, 3)`.

## Step 2 — Run it with zero infrastructure

Add a `main()` that stands up the minimal dispatch plane, submits, drains
one job in-process, and reads the result:

```python file="hello_udf.py"
import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

def main():
    dp = dls.Dispatch(":memory:")             # SQLite plane; no CP, no daemon
    q = dls.SyncQueue("hello", dispatch=dp)

    inv = add.submit(2, 3, _dispatch=dp)
    print("invocation:", inv)

    executed = dls.run_worker(q, once=True)   # claim + execute, one job
    print("executed:", executed)

    print("result:", q.result(inv, timeout=10))

if __name__ == "__main__":
    main()
```

```bash
python hello_udf.py
```

```text
invocation: 01JZ…
executed: 1
result: 5
```

`2 + 3 = 5`. If the last line prints, the submit → claim → execute →
return round-trip works.

## Step 3 — The id is the handle

There is no Future type. `submit()` returns a plain string — persist it,
log it, hand it to another process. Anything that can reach the same
dispatch plane re-attaches with it:

```python
q.result(inv, timeout=30.0)     # blocking fetch, from anywhere
q.cancel(inv)                   # ask the plane to drop it
for frame in q.stream(inv):     # journal frames, with a reconnect cursor
    ...
```

## Step 4 — The local escape hatch

`.local()` bypasses the queue and runs in-process, which is what you
want in unit tests and while debugging:

```python
print(add.local(2, 3))     # 5 — plain value, no plane involved
```

A bare `@dls.udf` (no `queue=`) behaves this way on every call.

## Step 5 — Against a real control plane

Two things change: the dispatch plane, and who runs the worker.

```bash
lakeshore auth login --server https://api.lakeshore.dreamlake.ai --namespace <your-namespace>
lakeshore queues add hello --kind fifo
```

Start a Python worker on a host that can import your module:

```bash
lakeshore worker start --queue hello \
  --url https://api.lakeshore.dreamlake.ai --token <token>
```

> **Warning:** `lakeshore daemon …` manages the Rust fleet daemon, which runs shell
> commands and control-plane invocations. Queue-bound Python UDFs are
> drained by `lakeshore worker start`, which spawns
> `python -m dreamlake.lakeshore.daemon`. The worker deliberately ignores
> saved `auth login` credentials — pass `--url` / `--token` explicitly, or
> export `LAKESHORE_URL` and `LAKESHORE_CLIENT_TOKEN`.

Then drop the `_dispatch=` arguments. With `LAKESHORE_URL` set (or an
`auth.yml` present), the SDK's default plane is already `HttpDispatch`:

```python
inv = add.submit(2, 3)
q = dls.SyncQueue("hello")
print(q.result(inv, timeout=60.0))
```

## Step 6 — Look at it from the CLI

```bash
lakeshore jobs list --queue hello
lakeshore jobs show <id>
lakeshore queues show hello        # queue row + current members
```

## Step 7 — Clean up

```bash
lakeshore queues rm hello
```

`rm` refuses with a 409 while the queue still has members; add `--force`
to scrub the name out of every `Worker.queues` array first.

## What you learned

- `@udf(queue="…")` binds a function to a queue **by name**.
- `submit()` returns a durable invocation id — a string, not a Future.
- `q.result(id, timeout=…)` fetches the value from any process holding
  the id.
- `.local(...)` always runs in-process.
- `dls.Dispatch` makes the whole loop runnable with no infrastructure.

## Read next

- [Fan out](/get-started/tutorials/fan-out.md) — dispatch N invocations and
  join with `asyncio.gather`.
- [`@udf` decorator](/python-sdk/udf.md) — the full decorator surface.
- [Queue API](/python-sdk/queue.md) — `Queue`, `SyncQueue`, streaming, and
  the worker verbs.
- [Queues](/get-started/queues.md) — the operator side of the queue you just
  created.

---

Source: https://lakeshore.dreamlake.ai/python-sdk

# Python SDK

> Decorate a function, call it, and get the result back — Lakeshore handles the rest.

## Install

The distribution is `dreamlake-lakeshore` (hyphen); the import path is
`dreamlake.lakeshore` (dot). `import lakeshore` is not a thing —
`dreamlake` is an implicit namespace package, so there is no top-level
`lakeshore` module.

```bash
pip install dreamlake-lakeshore
uv add dreamlake-lakeshore        # or with uv
```

```python
import dreamlake.lakeshore as dls
```

Python 3.11 or newer. The runtime dependencies are small and pure-Python:
`cloudpickle`, `httpx`, `msgpack`, `params-proto`, `python-ulid`,
`pyyaml`, `typing-extensions`.

Verify the install, and read the version from `importlib.metadata` —
`dreamlake.lakeshore.__version__` is a hardcoded `"0.0.0"` placeholder
that does not track releases:

```bash
python -c "import dreamlake.lakeshore as dls; print(dls.SyncQueue)"
python -c "from importlib.metadata import version; print(version('dreamlake-lakeshore'))"
```

### Extras

| Extra | Pulls in |
| --- | --- |
| `test` | `pytest`, `pytest-xdist` |
| `dev` | `test` plus `build`, `ruff` |
| `server` | nothing today — a reserved, deliberately empty extra |
| `all` | `test` + `server` + `dev` |

```bash
pip install "dreamlake-lakeshore[dev]"
```

> **Warning:** `pyproject.toml` declares no `[project.scripts]`. There is no
> `lakeshore-daemon` command — it was retired in 0.3.6. Run a worker with
> `python -m dreamlake.lakeshore.daemon`, or through the TypeScript CLI
> (`lakeshore worker start`, npm `@dreamlake/lakeshore`). See
> [Dispatch planes](/python-sdk/dispatch.md#the-worker-entrypoint).

## One pattern, end to end

```python
import dreamlake.lakeshore as dls

@dls.udf(queue="compute")
def square(x: int) -> int:
    return x * x

q = dls.SyncQueue("compute")
ids = [square.submit(i) for i in range(8)]        # durable string ids
results = [q.result(i, timeout=60.0) for i in ids]
```

The list comprehension is the spawn; the result loop is the barrier.
There is no Future type — a pending job **is** its invocation id, a
plain string you can persist and reconnect with from any process.

> **Note:** The SDK has no `gather`, no `as_completed`, and no `Future` class. Fan-out
> is a comprehension of `submit` calls plus a loop of `q.result`; on the
> async `Queue`, stdlib `asyncio.gather` composes over `q.result` coroutines.

## Data UDFs in one breath

A data UDF reads and writes files through `dls.run` and returns plain
**string keys**. File bytes never ride the msgpack wire — only keys do.

```python
@dls.udf
def splats_to_mesh(splats: str) -> str:
    mesh = extract(dls.run.read(splats))          # key -> local Path
    mesh.save(dls.run.write("process/mesh.ply"))  # key -> local Path, recorded
    return "process/mesh.ply"                     # return your keys

with dls.scope("scenes/0007"):                    # prefix for read/write
    key = splats_to_mesh("source/splats.ply")
```

## Surfaces

| Surface | Read for |
| --- | --- |
| [`@udf` decorator](/python-sdk/udf.md) | One decorator, four body kinds (sync / async / generator / async-gen), the `dls.run` I/O seam, the return-your-keys contract, and function transport. |
| [Queue API](/python-sdk/queue.md) | `SyncQueue` (blocking) and `Queue` (async) — submit / result / call / stream, worker verbs, and `Topic` pub/sub over the frame journal. |
| [Invocation ids](/python-sdk/invocations.md) | The no-Future model: ids as durable handles, reconnect-by-id, fan-out + collect, and live streaming via `q.stream`. |
| [Dispatch planes](/python-sdk/dispatch.md) | The minimal local plane (`Dispatch`) for dev and tests, `HttpDispatch` against the Node control plane, and the `run_worker` loop. |

## What's exported

`dreamlake.lakeshore.__all__` is exactly these 22 names:

```python
from dreamlake.lakeshore import (
    # Functions
    udf, UDF, thunk, Thunk,
    # The dls.run I/O seam
    scope, run, RunContext,
    # Queues
    Queue, SyncQueue, Topic, Job,
    # Dispatch planes
    Dispatch, HttpDispatch,
    # Worker
    run_worker,
    # Storage
    Storage, TemporaryCredentials,
    # .dreamrc config
    DreamRc, Provider, RunConfig, load_config, resolve,
)
```

Imports are lazy (PEP 562): `import dreamlake.lakeshore` loads almost
nothing, and purely local code (bare `@udf`, `dls.scope`, `dls.run`)
never pulls in httpx or msgpack.

> **Warning:** `dls.run` is the I/O proxy for the ambient run context —
> `dls.run.read` / `dls.run.write` / `dls.run.read_all` /
> `dls.run.write_all` / `dls.run.prefix` / `dls.run.root`. It does not run
> anything. The entry points that execute work are `UDF.__call__`,
> `.submit`, `.remote`, `.local`, the queue verbs, and `dls.run_worker`.

## Runnable examples

The SDK repo ships seven zero-infrastructure examples that run on an
in-memory `Dispatch(":memory:")` plane — no control plane, no daemon:

| Example | Shows |
| --- | --- |
| `examples/00_hello_udf.py` | submit → `run_worker(once=True)` → result |
| `examples/01_fan_out.py` | N submits, collect by id |
| `examples/02_pipeline.py` | two stages, the DAG is ordinary Python |
| `examples/03_body_kinds.py` | sync / async / generator / async-generator |
| `examples/04_ids_not_futures.py` | ids as durable handles |
| `examples/05_streams_topics.py` | `q.stream` frames and `Topic` pub/sub |
| `examples/06_dynamic_graph.py` | work discovered at run time |

## Try it end-to-end

Walk the [happy paths](/python-sdk/happy-paths.md) — five flows from "the
decorator works, no network" up to storage access from a worker.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths/01-hello-local-python

# 01 · Hello, local Python

The leaf node. If the SDK is not importable or the decorator is broken,
nothing above this can be green. Two minutes, no infrastructure.

## Exercises

- The Python SDK install path (uv / pip / editable).
- `@dls.udf` semantics — a bare `@udf` (no `queue=`) runs the decorated
  function in-process on a direct call.
- The always-local escape hatch: `train.local(...)` bypasses any queue
  binding, so test code stays stable when the decorator later gains one.

## Requires

- A Python ≥ 3.11 environment with `dreamlake-lakeshore` importable.
- That is it. No control plane, no daemon, no network — the local tier
  does not even import httpx or msgpack.

## Verifies

The SDK as a leaf node. It imports cleanly, the decorator wraps without
error, and the function executes and returns its value verbatim.

## Run

```bash
cd lakeshore-examples/02-hello-local-python
uv run train.py
```

`train.py` carries inline script metadata whose `[tool.uv.sources]` block
pins `dreamlake-lakeshore` to the `../../lakeshore-py` checkout as an
editable install — so this runs the local source tree, not PyPI. To
exercise the published package instead, delete that block from the
script header; the sibling `pyproject.toml` declares
`dreamlake-lakeshore>=0.3.7`.

Plain pip works too:

```bash
pip install -e lakeshore-py     # or: pip install dreamlake-lakeshore
python lakeshore-examples/02-hello-local-python/train.py
```

## Expected output

```text
{'seed': 42, 'loss': 0.6394267984578837}
{'seed': 7, 'loss': 0.32383276483316237}
{'seed': 1, 'loss': 0.13436424411240122}
```

Three dicts, deterministic across runs because the example seeds
`random`. The third line comes from `train.local(seed=1)`.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `ModuleNotFoundError: No module named 'dreamlake'` | The SDK is not in the active environment. Use `uv run --with-editable lakeshore-py ...` or `pip install -e lakeshore-py`. Note the install name is `dreamlake-lakeshore`, the import path `dreamlake.lakeshore`. |
| `TypeError` on the decorator | An SDK version mismatch — the example declares `dreamlake-lakeshore>=0.3.7`. Bump or pin. |
| Different `loss` values | Check that the example still seeds `random` before each call. The shape is what matters. |

## Status

Runs in the example smoke set; refresh `/test-results.json` via
`uv run lakeshore-examples/test_runner.py`.

## Next

→ [02 · Hello, hosted control plane](/cli/happy-paths/02-hello-hosted-cp.md)

---

Source: https://lakeshore.dreamlake.ai/cli

# `lakeshore` CLI

> The operator surface for managing providers, daemons, queues,
> storage, and secrets from your terminal.

`@dreamlake/lakeshore` is an ESM Node package built on
[Commander](https://github.com/tj/commander.js). It ships a single
binary, `lakeshore`.

```bash
npm i -g @dreamlake/lakeshore
lakeshore --help
```

> **Warning:** The program never calls Commander's `.version()`, so `lakeshore
> --version` fails as an unknown option. Read the installed version from
> npm instead: `npm ls -g @dreamlake/lakeshore`. (Shell completion offers
> `--version` as a candidate — that is a stale entry in the generated
> completion tree, not a real flag.)

## Global options

`--dreamrc <path>` is the **only** program-level option. It points the
`.dreamrc` resolver at an explicit file instead of the standard lookup.
Two commands redeclare it locally and fall back to the program-level
value: `providers test` and `providers instances`. No other command
accepts it.

Everything else is per-command. Every command and subcommand responds
to `--help`, and Commander is configured with `showHelpAfterError()`,
so a usage mistake prints the help for the command you got wrong.

## The command surface

### Resource management

| Group        | Verbs                                                            | Notes                                                                                   |
| ------------ | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `providers`  | `add` `list` `show` `update` `remove` `edit` `discover` `test` `instances` `terminate` | Cloud + SSH launch targets. See [Providers + discover](/cli/examples/providers-and-discover.md). |
| `secrets`    | `add` `list` `show` `rotate` `remove`                            | Named credentials. Plaintext flows in only; `list`/`show` are metadata.                 |
| `modes`      | `add` `list` `show` `update` `remove`                            | Server-stored RunConfigs. Override flag is `--field`, not `--kwarg`.                     |
| `tunnels`    | `add` `list` `show` `remove`                                     | WireGuard (`wg-quick`) configs attached to providers.                                    |
| `mounts`     | `add` `list` `show` `update` `remove`                            | Mount declarations — metadata only, the daemon does the mounting.                        |
| `storage`    | `add` `list` `show` `update` `remove` `presign` `credentials`    | S3-compatible object storage. **Singular verb**, unlike its siblings.                    |
| `code`       | `push` `list`                                                    | Git-tree snapshots uploaded to a storage entry.                                          |
| `queues`     | `add` `ls` `show` `patch` `archive` `unarchive` `drain` `rm` `events` `stats` | **Irregular verbs**: `ls` and `rm`, not `list`/`remove`. See [Queues](/get-started/queues.md). |

### Running work

| Command / group | What it does                                                                                                        |
| --------------- | -------------------------------------------------------------------------------------------------------------------- |
| `daemon`        | Fleet management: `launch` `install` `list` `show` `setup` `exec` `kill` `cleanup` `hibernate` `reset` `update` `target-version` `set-target-version` `launch-log` `stop` `status`. |
| `exec`          | Top-level one-off bash on a daemon. Pops a picker, or takes `--queue <name>` and lets the control plane route.       |
| `run`           | Pipes a local **Python** script to a remote `python3 -` over the exec channel.                                       |
| `worker`        | `start` / `once` — supervise a native Python queue worker on **this** machine.                                       |
| `jobs`          | `list` `show` `kill` `bulk-cancel` over dispatched Invocations.                                                      |
| `up` `down` `ps` `status` `logs` | Compose verbs. They read `./lakeshore.yaml` (`-f` to override). See [Compose](/get-started/compose.md). |

> **Note:** There is no `lakeshore compose ...` command. The five verbs sit at the
> top level even though their source lives under `src/cli/compose/`.

### Discovery, config, and operator tooling

| Group / command | What it does                                                                                     |
| --------------- | -------------------------------------------------------------------------------------------------- |
| `ssh`           | `discover` `upload` `probe` `status` over `~/.ssh/config`.                                        |
| `aws` `gcp` `slurm` `kube` `docker` | Each exposes exactly one read-only `discover` subcommand with a single `--json` flag. |
| `auth`          | `login` `status` `logout` — saves a per-namespace token at `~/.config/lakeshore/auth.yml`.        |
| `config`        | `show` `refresh` `source` — diagnostics for the resolved `.dreamrc`.                              |
| `admin`         | `tokens create` / `list` / `revoke`, `namespace create` / `list` / `delete`. Requires `LAKESHORE_ADMIN_TOKEN`. |
| `nymph`         | `push` `list` — upload a hashed nymph binary for debug-fleet OTA. Admin-authed.                   |
| `dreamlake`     | The DreamLake data plane (separate servers, separate credential store) — upload/download/list/vectorize. |
| `tui`           | Interactive terminal dashboard. Aliases `dashboard` and `top`. See [TUI dashboard](/cli/tui.md).     |
| `completion`    | `bash` / `zsh` / `fish` tab-completion. See [Completion](/cli/completion.md).                        |

> Looking for a **runnable cookbook** with one example per verb?
> [`/cli/examples`](/cli/examples.md) is the flat reference — same
> commands, every block copy-paste-runnable once you've done the
> setup at the top.

## How the CLI finds the control plane

Two resolution paths that do **not** agree, which is worth knowing
before you debug an auth problem:

- The `auth` commands resolve the server as `--server` → `LAKESHORE_SERVER`
  → the saved auth file → `http://localhost:8080`.
- **Every other command** goes through `resolveServer()`, which prefers
  `LAKESHORE_URL` (with `LAKESHORE_NAMESPACE`, default `default`) and in
  that mode sends **no bearer token at all** — it deliberately bypasses
  the saved auth file. Without `LAKESHORE_URL` it falls back to the
  saved login, and with neither it fails outright:

```text
server not configured — run `lakeshore auth login --server <url>` or set LAKESHORE_URL
```

So exporting `LAKESHORE_URL` is right for an open-mode local stack and
wrong for a token-enforced deployment.

## Short examples

### Register a provider

`providers add` takes the name as a **positional** and the launcher as
`--launcher` (one of `SSH`, `SLURM`, `EC2`, `GCE`, `Kube`). Names must
match `^[a-z0-9][a-z0-9-]*$`.

```bash
# From a local AWS profile — implies --launcher EC2:
lakeshore aws discover                                  # see what's on this box
lakeshore providers add ec2-prod --from-aws-profile prod

# From a gcloud configuration — implies --launcher GCE:
lakeshore providers add gce-prod --from-gcloud-config prod

# By hand. Dotted --kwarg keys nest.
lakeshore providers add my-ec2 \
  --launcher EC2 \
  --kwarg region=us-east-1 \
  --kwarg aws_credentials.accessKeyId=AKIAEXAMPLE \
  --kwarg aws_credentials.secretAccessKey=secret-example
```

`lakeshore providers list` shows what's registered;
`lakeshore providers edit` opens an interactive hide/unhide/delete/restore
editor.

### Smoke-test a provider end-to-end

`providers test` runs a small Python script through the provider's
launch path. With no `--script` it uses the bundled
`src/cli/scripts/hello.py`, which is stdlib-only on purpose (it prints
hostname, Python version, platform, and whether torch sees a GPU) so it
runs on a bare host.

```bash
lakeshore providers test my-ec2                    # built-in hello
lakeshore providers test my-ec2 --script train.py  # your own
lakeshore providers test my-ec2 --script train.py --timeout 1200
```

SSH providers run directly from your machine; EC2, GCE, and Kube are
server-mediated (the control plane holds the cloud SDKs). SLURM prints
a stub message — that path is not wired yet.

### Bring a daemon up

```bash
# Provider-driven — the control plane provisions a cloud host:
lakeshore daemon launch --provider ec2-prod
lakeshore daemon launch --provider ec2-prod --count 4 --name fleet

# A host you can already SSH to (login node, on-prem box, VM):
lakeshore daemon install my-server
```

`lakeshore daemon list` shows the new rows; `--watch` refreshes (bare
`--watch` means every 2 seconds). See [Daemon launch](/nymph/daemons.md)
for the provider path and
[Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md)
for the SSH path.

> **Note:** `lakeshore daemon ...` manages nymph daemons through the control plane.
> To run a native Python queue worker on your own machine, use
> `lakeshore worker start` (or `worker once` to drain a single job and
> exit). There is no `daemon start --local`.

### Run a one-off command on a daemon

`exec` runs a bash command on a worker and writes stdout/stderr back
verbatim; the CLI exits with the remote command's exit code.

```bash
lakeshore exec nvidia-smi                      # interactive picker
lakeshore daemon exec ge-debug-1 nvidia-smi    # explicit daemon
```

The `<daemon>` argument accepts a worker id, a label (set by
`daemon launch --name`), or the host's `machine_id`. Add `--id` to skip
the name lookup when a label happens to look like a worker id. If a
name matches more than one daemon, the CLI prints the candidates and
asks you to re-run with `--id <worker-id>`.

Both `exec` forms use Commander's `passThroughOptions()`: **CLI flags
go before the daemon argument, everything after it is the remote
command, and no `--` separator is used.**

```bash
cat input.json | lakeshore daemon exec --stdin - ge-debug-1 jq .results

lakeshore daemon exec \
  --env CUDA_VISIBLE_DEVICES=0 \
  --env BATCH=64 \
  --timeout 60 \
  ge-debug-1 python train.py
```

## Read next

- [Installation](/cli/installation.md) — install and authenticate.
- [CLI examples](/cli/examples.md) — the runnable cookbook, one example per verb.
- [TUI dashboard](/cli/tui.md) — `lakeshore tui` and its inline per-item actions.
- [Completion](/cli/completion.md) — tab-completion install and how it works.
- [Python SDK](/python-sdk.md) — what you can launch from the Python side.
- [Daemon launch](/nymph/daemons.md) · [Daemon lifecycle](/nymph/daemon-lifecycle.md)
- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — the EC2 / GCE / SLURM / Kube / SSH launchers.

---

Source: https://lakeshore.dreamlake.ai/cli/examples/providers-and-discover

# CLI examples — Providers and discover helpers

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives
on [Auth + setup](/cli/examples/auth.md).

## Providers

A **provider** is a registered cloud, SLURM, Kube, or SSH target.
Add → list → test → launch is the happy path.

The provider name is a **positional** argument matching
`^[a-z0-9][a-z0-9-]*$`, and the launcher is `--launcher` — one of
`SSH`, `SLURM`, `EC2`, `GCE`, `Kube`. `--launcher` is required unless
you pass one of the `--from-*` flags, which imply it.

> **Warning:** The `lakeshore` README's quick start shows `providers add --name my-aws
> --type aws`. That signature does not exist. Neither `--name` nor
> `--type` is a flag on `providers add`.

### `providers add` — register an EC2 provider by hand

Inline kwargs. Dotted keys nest (so `aws_credentials.accessKeyId`
becomes the nested object the server expects).

```bash
lakeshore providers add my-ec2 \
  --launcher EC2 \
  --kwarg region=us-east-1 \
  --kwarg instance_type=t3.medium \
  --kwarg aws_credentials.accessKeyId=AKIAEXAMPLE \
  --kwarg aws_credentials.secretAccessKey=secret-example
```

### `providers add` — from a local AWS profile

Pre-fills `aws_credentials` and region from the named `~/.aws` profile.
Implies `--launcher EC2`.

```bash
lakeshore providers add ec2-prod --from-aws-profile prod
```

### `providers add` — from a gcloud configuration

Pre-fills `project_id`, region, zone, and a service-account JSON from
the named `gcloud` configuration. Implies `--launcher GCE`.

```bash
lakeshore providers add gce-prod --from-gcloud-config prod
```

### `providers add` — load kwargs from a file

For longer or nested configs. YAML or JSON; the file's top level
becomes the provider's kwargs object.

```bash
cat > /tmp/kube-kwargs.yaml <<'YAML'
context: my-kube-context
namespace: lakeshore-jobs
image: ghcr.io/dreamlake-ai/runner:latest
YAML
lakeshore providers add my-kube --launcher Kube --kwargs-file /tmp/kube-kwargs.yaml
```

### `providers add --dispatch` — direct vs daemon

`--dispatch` is `direct` or `daemon`. Omit it and the launcher's
default applies:

| Launcher | Default dispatch |
| -------- | ---------------- |
| `SSH`    | `direct`         |
| `Kube`   | `direct`         |
| `SLURM`  | `direct`         |
| `EC2`    | `daemon`         |
| `GCE`    | `daemon`         |

A `dispatch` key inside the provider's kwargs overrides the default too.

```bash
lakeshore providers add my-ec2 --launcher EC2 --dispatch direct --kwarg region=us-east-1
```

### `providers list` — table view

The default management view: everything registered in this namespace,
excluding hidden and soft-deleted rows.

```bash
lakeshore providers list
lakeshore providers list --json | jq -r '.[].name'
lakeshore providers list --hidden          # include hidden
lakeshore providers list --deleted         # include soft-deleted
lakeshore providers list --all             # both
```

Soft-deleted rows come back marked `[D <ts>]`.

### `providers show` — pretty-print one

Dumps the full kwargs for a single provider. Takes a name and no flags.

```bash
lakeshore providers show my-ec2
```

### `providers update` — merge in new kwargs

`--kwarg` deep-merges (repeatable); `=null` deletes a key.
`--kwargs-file` replaces the kwargs object **entirely**.

```bash
lakeshore providers update my-ec2 \
  --kwarg instance_type=t3.large \
  --kwarg root_volume_size_gb=100

lakeshore providers update my-ec2 --kwarg root_volume_size_gb=null
```

> **Warning:** Supplying `--launcher` to `providers update` is an error, not a no-op.
> Remove and re-add the provider to change its launcher.

### `providers update --tunnel` — attach a tunnel

Bind a previously registered WireGuard tunnel to this provider. Pass
`--tunnel ""` to clear.

```bash
lakeshore providers update my-ec2 --tunnel wg-vpc
lakeshore providers update my-ec2 --tunnel ""
```

### `providers test` — smoke-test the launch path

Runs a small Python script on the provider end-to-end. With no
`--script` it uses the bundled stdlib-only `hello.py`, which prints the
hostname, Python version, platform, and whether torch sees a GPU.
`--timeout` defaults to 600 seconds.

```bash
lakeshore providers test my-ec2
lakeshore providers test my-ec2 --script train.py --timeout 1200
```

SSH providers run directly from your machine. EC2, GCE, and Kube are
server-mediated — the CLI POSTs to the control plane, which holds the
cloud SDKs and credentials. SLURM prints a stub message; that path is
not wired yet.

### `providers instances` — live instance state

Hits the provider's launcher to list currently-running instances.
`--timeout` defaults to 10 seconds. The name positional is optional
when you pass `--all`.

```bash
lakeshore providers instances my-ec2
lakeshore providers instances --all --timeout 15
```

### `providers terminate` — kill a cloud instance

Two required positionals: provider then instance id. EC2 / GCE only.
This is a provider-side terminate, not a daemon kill.

```bash
lakeshore providers terminate my-ec2 i-0abc1234
```

### `providers edit` — interactive editor

Arrow-key through the provider list and hide / unhide / delete /
restore. Takes no arguments or flags, and expects a TTY.

```bash
lakeshore providers edit
```

### `providers discover` — unified picker

Surfaces local AWS profiles, GCP configs, and SSH hosts that are **not**
yet registered, lets you multi-select, and registers them in one go.

```bash
lakeshore providers discover
lakeshore providers discover --filter prod              # substring pre-filter
lakeshore providers discover --include-existing-ssh     # don't hide already-registered SSH hosts
lakeshore providers discover --page-size 30             # default 15
lakeshore providers discover --yes --filter prod        # skip the confirm
```

### `providers remove` — delete

Drops the provider row. The cloud instances it spawned are unaffected.

```bash
lakeshore providers remove my-ec2
```

## Discover helpers

Local-only sniffers that inspect your machine's config files. None of
them touch the control plane or the cloud — they're inputs to
`providers add` / `providers discover`. Each exposes exactly one
subcommand, `discover`, with exactly one flag, `--json`.

```bash
lakeshore aws discover      # ~/.aws/credentials + ~/.aws/config
lakeshore gcp discover      # `gcloud`, falling back to ~/.config/gcloud
lakeshore slurm discover    # /etc/slurm/slurm.conf + `sinfo`
lakeshore kube discover     # `kubectl config view`
lakeshore docker discover   # `docker context ls`

lakeshore aws discover --json
```

## SSH

`lakeshore ssh` reads `~/.ssh/config` and can register hosts as
providers.

### `ssh discover` — list hosts

`--filter` accepts `all` (default), `new` (not yet on the server), or
`existing`. Anything else exits 2.

```bash
lakeshore ssh discover
lakeshore ssh discover --filter new
lakeshore ssh discover --filter existing
lakeshore ssh discover --show-stripped        # show local-only directives that were dropped
```

### `ssh upload` — register an SSH host as a provider

```bash
lakeshore ssh upload my-server                      # single alias
lakeshore ssh upload my-server --with-key           # also upload the IdentityFile as a secret
lakeshore ssh upload my-server --name prod-box      # provider name (default: the alias)
lakeshore ssh upload --all                          # every host, fingerprint-dedup'd
lakeshore ssh upload                                # no alias → multi-select picker
lakeshore ssh upload my-server --interactive        # force the picker anyway
```

Precedence: `--all` (or the literal alias `--all`) wins and is
non-interactive; otherwise `--interactive` **or** a missing alias opens
the picker; otherwise it's a single-alias upload.

Parsed `~/.ssh/config` values can be overridden per upload with
`--user`, `--host`, `--port`, and `--pem`. `--key-secret <name>` names
the uploaded key secret (default `ssh-<alias>`). `--dispatch` accepts
`direct` or `daemon` and is silently ignored for any other value.

### `ssh probe` — try to connect, record locally

The result is cached at `~/.config/dreamlake/ssh-status.json` and is
**not** sent to the server. `--timeout` is the ssh `ConnectTimeout`,
default 5 seconds.

```bash
lakeshore ssh probe my-server
lakeshore ssh probe my-server --timeout 15
lakeshore ssh probe --all      # the literal string '--all' as the alias probes every host
```

### `ssh status` — show the local probe cache

`--history` defaults to the 3 most recent probes per provider.

```bash
lakeshore ssh status
lakeshore ssh status --name my-server
lakeshore ssh status --history 5
```

## Read next

- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — per-launcher configuration reference.
- [Daemons + exec/run](/cli/examples/daemons-and-exec.md) — launch workers on these providers.

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths/02-hello-hosted-cp

# 02 · Hello, hosted control plane

Two commands prove the CLI can authenticate against a real control
plane, send a bearer token, and read its namespace back. Run this when
you've just installed the CLI and want to confirm the wire works before
anything else. Takes under a minute.

## Before you begin

- Network access to a control plane. The hosted one lives at
  `https://api.lakeshore.dreamlake.ai`.
- A valid client token (`dlk_...`) scoped to at least one namespace. If
  you don't have one yet, see
  [15 · Token lifecycle](/cli/happy-paths/15-token-lifecycle.md).
- `LAKESHORE_URL` **unset**. If it's exported, every non-`auth` command
  will use it and send no token at all, which defeats the point of this
  test.

## Step 1: Log in

```bash
lakeshore auth login \
  --server https://api.lakeshore.dreamlake.ai \
  --namespace <your-namespace> \
  --token <dlk_...>
```

`login` validates by calling `GET /v1/namespaces/<ns>/whoami` with the
Bearer header **before** it writes anything. On success it saves
`~/.config/lakeshore/auth.yml` with mode 600 and prints:

```text
Saved auth to /Users/you/.config/lakeshore/auth.yml
  server:    https://api.lakeshore.dreamlake.ai
  namespace: <your-namespace>
  token:     dlk_abcd...
  authMode:  token
```

## Step 2: Check status

```bash
lakeshore auth status
```

This reloads the file and re-validates it against the same `whoami`
endpoint.

## What success looks like

```text
Auth OK.
  config path: /Users/you/.config/lakeshore/auth.yml
  server:      https://api.lakeshore.dreamlake.ai
  namespace:   <your-namespace>
  token:       dlk_abcd... (laptop-2026-01-02T03-04-05)
  lastUsedAt:  2026-01-02T03:05:11.000Z
  authMode:    token
```

`authMode: token` plus your namespace means the full auth path — TLS,
Bearer header, token-hash lookup, namespace scoping — works end to end.

## If something goes wrong

| Symptom | Likely cause |
| --- | --- |
| `whoami failed (401)` | Token typo or revoked. Re-mint via [15 · Token lifecycle](/cli/happy-paths/15-token-lifecycle.md). |
| `whoami failed (403)` | Token valid but scoped to a different namespace. Check `--namespace`. |
| `ENOTFOUND` / DNS error | Server URL typo, or no internet. |
| `No saved auth.` from `status` | `login` never completed — it does not save when `whoami` rejects the token. |
| `authMode: open` | You hit a control plane running without token enforcement. Fine for local dev, not what you want against the hosted CP. |

## You're done

The CLI can talk to the hosted control plane. Next, run the same
exercise against a local stack:
[03 · Hello, local stack](/admin/happy-paths/03-hello-local-stack.md).

---

Source: https://lakeshore.dreamlake.ai/get-started/status

# Status

Live verification surface for the Lakeshore stack. The JSON behind this
page is produced by `lakeshore-examples/test_runner.py` and served from
`docs/public/test-results.json`. Doc pages can also embed
`` inline to surface "are this feature's tests
green right now?" next to the section it documents.

## Unit-test suites

These are the leaf-node verifications — fast, deterministic, no infra
needed. Click a suite name to jump to its concept docs.

## Example smoke runs

Examples from
[`lakeshore-examples/`](https://github.com/dreamlake-ai/lakeshore-examples).
Most require a running control plane or external infra and are skipped by
default; pass `--enable-cp` to attempt them against a live control plane.

## How to refresh

Run from the workspace checkout that holds `lakeshore-examples/`:

```bash
uv run lakeshore-examples/test_runner.py
```

It writes `lakeshore-examples/results.json` (canonical) and copies the
same payload to `docs/public/test-results.json` (what this page reads).

| Flag | Effect |
| --- | --- |
| `--suites-only` | Skip the example runs. |
| `--examples-only` | Skip the unit-test suites. |
| `--suites <id> …` | Restrict to specific suite ids. |
| `--only <id> …` | Restrict to specific example ids. |
| `--enable-cp` | Also attempt control-plane-dependent examples. |

> **Note:** `test_runner.py` ships with `lakeshore-examples`, which is a sibling
> repo — clone `dreamlake-ai/lakeshore-workspace` to get it alongside
> everything else. The docs site only consumes the JSON it emits.

## Inline usage

In any doc page that documents a specific feature, surface that
feature's test status next to it:

```mdx
The control-plane poll handler is covered by the
`lakeshore-controlplane` suite: 
```

---

Source: https://lakeshore.dreamlake.ai/get-started/tutorials/fan-out

# Fan out

> **Example code:** [`10-fan-out-gather`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/10-fan-out-gather)

The core Lakeshore pattern: spawn many invocations with a Python list
comprehension, then collect the results with stdlib `asyncio`. The
for-loop is the DAG — there is no scheduling DSL to learn.

This builds on [Hello UDF](/get-started/tutorials/hello-udf.md). Work
through that one first if you have not.

> **Note:** The SDK ships no `gather`, `gather_async`, `as_completed`, `map`, or
> `Future`. `submit()` returns a durable invocation id (a string), the
> async `Queue.result(id)` is awaitable, and `asyncio.gather` /
> `asyncio.as_completed` are the joins.

## Step 1 — Define a queue-bound function

```python file="fan_out.py"
import dreamlake.lakeshore as dls

@dls.udf(queue="fan-out-demo")
def square(x: int) -> int:
    return x * x
```

## Step 2 — Fan out with a list comprehension

`submit()` enqueues and returns immediately, so a comprehension over it
dispatches N invocations without blocking:

```python file="fan_out.py"
ids = [square.submit(i, _dispatch=dp) for i in range(n)]
print(f"dispatched {len(ids)} invocations")
```

At this point the plane holds N pending invocations on `fan-out-demo`.
Every worker subscribed to that queue can claim them.

## Step 3 — Join with `asyncio.gather`

The async `Queue.result(id)` is awaitable, so `asyncio.gather` collects
them in **submission order**:

```python file="fan_out.py"
async def fan_out(n: int, dp) -> list[int]:
    q = dls.Queue("fan-out-demo", dispatch=dp)          # the async surface
    ids = [square.submit(i, _dispatch=dp) for i in range(n)]
    return await asyncio.gather(*(q.result(inv, timeout=30) for inv in ids))
```

## Step 4 — Run it, with zero infrastructure

Stand up the minimal plane and a worker thread in the same process:

```python file="fan_out.py"
import asyncio
import threading

import dreamlake.lakeshore as dls

@dls.udf(queue="fan-out-demo")
def square(x: int) -> int:
    return x * x

async def fan_out(n: int, dp) -> list[int]:
    q = dls.Queue("fan-out-demo", dispatch=dp)
    ids = [square.submit(i, _dispatch=dp) for i in range(n)]
    return await asyncio.gather(*(q.result(inv, timeout=30) for inv in ids))

def main(n: int = 10):
    dp = dls.Dispatch(":memory:")
    done = threading.Event()
    worker = threading.Thread(
        target=dls.run_worker,
        args=(dls.SyncQueue("fan-out-demo", dispatch=dp),),
        kwargs={"stop": done.is_set},
        daemon=True,
    )
    worker.start()

    results = asyncio.run(fan_out(n, dp))
    done.set()
    worker.join(timeout=5)

    for i, r in enumerate(results):
        print(f"  square({i}) = {r}")

if __name__ == "__main__":
    main()
```

```bash
python fan_out.py
```

```text
  square(0) = 0
  square(1) = 1
  square(2) = 4
  …
  square(9) = 81
```

`run_worker(q, stop=…)` loops until the `stop` predicate returns true;
`run_worker(q, once=True)` drains exactly one job and returns.

## Step 5 — Process results as they arrive

When some invocations are much slower than others, handle each result
the moment it lands instead of waiting for the slowest. `asyncio.as_completed`
over awaitable results does it:

```python file="fan_out_stream.py"
@dls.udf(queue="fan-out-demo")
def slow_square(x: int, delay: float) -> int:
    import time
    time.sleep(delay)
    return x * x

async def fan_out(n: int, dp) -> None:
    q = dls.Queue("fan-out-demo", dispatch=dp)
    ids = [
        slow_square.submit(i, delay=(n - i) * 0.05, _dispatch=dp)
        for i in range(n)
    ]

    async def one(i: int, inv: str) -> tuple[int, int]:
        return i, await q.result(inv, timeout=30)

    for fut in asyncio.as_completed([one(i, inv) for i, inv in enumerate(ids)]):
        i, r = await fut
        print(f"  completed: slow_square({i}) = {r}")
```

Give the later submits the shorter sleeps and run three worker threads,
and completions arrive in roughly reverse submit order — which is the
point.

## Step 6 — A two-stage pipeline

Chain two fan-outs by feeding stage-1 values straight into stage-2
submits. Each `gather` is a synchronization barrier:

```python file="pipeline.py"
@dls.udf(queue="pipeline-demo")
def double(x: int) -> int:
    return x * 2

@dls.udf(queue="pipeline-demo")
def to_hex(n: int) -> dict:
    return {"n": n, "hex": hex(n)}

async def pipeline(dp) -> list[dict]:
    q = dls.Queue("pipeline-demo", dispatch=dp)

    stage1 = [double.submit(i, _dispatch=dp) for i in range(6)]
    doubled = await asyncio.gather(*(q.result(inv, timeout=30) for inv in stage1))

    stage2 = [to_hex.submit(v, _dispatch=dp) for v in doubled]
    return await asyncio.gather(*(q.result(inv, timeout=30) for inv in stage2))
```

The call graph is Python control flow. No graph definition language, no
edge declarations.

> **Note:** Bare `str` returns are reserved for data-UDF **key manifests** — the
> return-your-keys contract. A compute UDF that produces text should wrap
> it in structured data so it is never mistaken for a storage key.

## Step 7 — Against a real plane

Drop the `_dispatch=` arguments once `LAKESHORE_URL` (or `auth.yml`) is
set, create the queue, and give it real workers:

```bash
lakeshore queues add fan-out-demo --kind fifo

# One process per worker; each claims one invocation at a time.
lakeshore worker start --queue fan-out-demo --url "$LAKESHORE_URL" --token "$TOKEN"
```

To move an already-running fleet daemon onto the queue instead:

```bash
lakeshore daemon update <daemon-id> --add-queue fan-out-demo
```

## Clean up

```bash
lakeshore queues rm fan-out-demo --force
lakeshore queues rm pipeline-demo --force
```

## What you learned

- A list comprehension over `submit()` fans out N invocations without
  blocking.
- `asyncio.gather` over the async `Queue.result` joins in submission
  order; `asyncio.as_completed` yields in arrival order.
- Pipelines are chained fan-outs — the values from one barrier feed the
  next comprehension.
- The whole pattern runs with no infrastructure on `dls.Dispatch`, and
  the code is unchanged against a control plane.

## Read next

- [Invocation ids](/python-sdk/invocations.md) — the no-Future model,
  reconnect-by-id, and live streaming.
- [Queue API](/python-sdk/queue.md) — `Queue` / `SyncQueue` and the worker
  verbs.
- [Compose a cluster](/get-started/tutorials/compose.md) — give the queue a
  real fleet.
- [Queues](/get-started/queues.md) — kinds, membership, and elasticity.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/execution-matrix

# The execution matrix

Python gives you an unusually large number of ways to express "run this" —
sync calls, futures, generators, coroutines, async generators, context
managers, run objects — and the pre-rewrite API had accreted most of them.
This page is the matrix that framed the question before any opinion: **two
axes**, the *form* (the Python execution pattern) and the *level* (what you
are running).

The question as originally posed: *generators, async generators, sync,
async, pipeline run object — what are the different ways we have, for
running the basics, the systems, and the system-level imperative dynamic
graph?*

## Axis 1 — the form

The full vocabulary Python offers, and how you get values out of each:

| Form | Invoke | Get results | Driver | Barrier? |
| --- | --- | --- | --- | --- |
| F1 sync blocking | `y = f(x)` | return value | caller | — |
| F2 deferred handle | `h = f.submit(x)` | resolve `h` later | caller polls | deferred |
| F3 sync generator | `for x in g()` | lazy pull, one at a time | **consumer** (backpressure) | streaming |
| F3′ gen with `.send()` | `x = yield y` | two-way (trampoline / scheduler) | scheduler | streaming |
| F4 async coroutine | `await f(x)` | awaited value | event loop | deferred |
| F5 async generator | `async for x in g()` | lazy pull plus await | consumer plus loop | streaming |
| F6 context manager | `with X() as h:` / `async with` | scoped lifecycle | — | — |
| F7 run/handle object | `r = pipe.submit(...)` | poll status, stream events, await | caller tracks | either |

Each form F1–F7 has its own page in the **Patterns** section, expanded into
runnable examples against the shipped SDK.

## Axis 2 — the level

- **L0 basic** — one UDF, one invocation.
- **L1 systems** — N calls across queues and workers; the scheduling
  substrate.
- **L2 static pipeline** — a fixed multi-stage DAG, knowable at author time
  (`examples/02_pipeline.py`).
- **L3 dynamic imperative graph** — structure discovered *while running*:
  data-dependent fan-out width, conditional branches, rework loops,
  recursion.

## The original grid — drawn against the pre-rewrite API

> **Warning:** Every symbol in the grid below (`apply`, `apply_spawn`, `apply_async`,
>   `gather`, `map`, `as_completed`, `Future`, `QueueFuture`, `Pool`,
>   `rpc_future`, `@pipeline`) was deleted in the native rewrite. The grid is
>   preserved because the *shape* of the analysis — where the mass sat, where
>   the holes were — is what drove the design. For what to write today, skip
>   to the re-grounded grid below.

| | L0 basic | L1 systems / fan-out | L2 static pipeline | L3 dynamic graph |
| --- | --- | --- | --- | --- |
| **F1 sync blocking** | `apply(fn, …)`, `fut.get()`, `@udf` local | `Queue.map(fn, xs)`, `gather([…])` | gather → gather (barriers) | `while` loop plus blocking calls plus requeue (serial) |
| **F2 sync plus future** | `apply_spawn` → `Future`; `@udf(queue=q)` → `QueueFuture` | `[fn(i) for i …]` → `list[Future]`; `as_completed` | a stage-1 Future passed into a stage-2 call (implicit edge) | spawn futures off resolved values; `parent_invocation_id` builds the tree |
| **F3 sync generator** | `for chunk in q.rpc_future(…)` stream | `as_completed` yields by completion; draft `batch(src, n)` | draft `@pipeline` (a generator that yields `to_dataset(…)`) | `.send()`-driven dynamic pull → **GAP** |
| **F4 async coroutine** | `await apply_async`, `await fut` | `await gather_async`, `map_async` | an `await` chain of stages | `asyncio.create_task` dynamic spawn |
| **F5 async generator** | streaming one call via `async for` → **GAP** | `as_completed_async(futs)` | async-gen stages → **GAP** | `async for` over a growing stream → **GAP** |
| **F6 context manager** | `with Queue(…) as q:` | `Pool(size=4)` | — | — |
| **F7 run object** | — | — | **GAP** (`@pipeline` returned the function unchanged) | **GAP** — no run you can poll, stream, or await |

### What the grid showed

- **The legacy API lived almost entirely in F1, F2, and F4** — blocking,
  futures, async. That was roughly fifteen symbols spent on one region:
  `apply` / `spawn` / `call` in sync and async spellings, `gather` / `map` /
  `as_completed` in sync and async spellings, and two Future classes.
- **The dataflow-primitives draft lived in F3** — `batch`, `@pipeline`, all
  generator-shaped. A *different form*, which is why it read so differently
  from the client.
- **The whole F7 row was empty.** `@pipeline` was a stub that returned the
  function unchanged. The draft's `Graph` is an author-time structure, not a
  live handle, and callee-side run telemetry is not a caller-side tracker.
- **Column L3 was only reachable imperatively.** It was real — the
  `parent_invocation_id` tree captured it at runtime — but never a
  first-class object you could hold, stream, or visualize.

Two cross-cutting tensions fall straight out of the grid: **barrier versus
streaming** (F1/F2 gather against F3/F5 generators), and **static versus
dynamic** (L2 knowable at author time against L3 discovered at run time).
The empty F7 row is exactly where those two reconcile — a run object is
*the* thing that can be both streamed and dynamic.

## Re-grounded: the shipped grid (0.3.7)

The rewrite deleted the legacy surface and collapsed most of the form axis
into the function body itself:

| Form | Shipped spelling | Status |
| --- | --- | --- |
| F1 sync | `f(x)` — local tier, or submit-and-wait when a queue is bound | shipped |
| F2 deferred | `inv = f.submit(x)` then `q.result(inv)` | shipped (**ids, not Futures**) |
| F3 generator | generator body; `q.stream(inv, since=cursor)` | shipped |
| F4 async | `async def` body; `dls.Queue`, the async sibling | shipped |
| F5 async-gen | `async def` generator body; `async for … in q.stream(…)` | shipped |
| F6 scope | `with dls.scope("scenes/0007"):` | shipped |
| F7 run handle | *none — there is no pipeline run object* | **GAP — the open seam** |

Three of the original F5 gaps closed because the async generator became a
first-class body kind, detected by `inspect.isasyncgenfunction` at
decoration. Two cells remain open rather than shipped:

- **F3 × L3, the two-way `.send()` trampoline.** The journal is one-way —
  workers append frames, consumers pull them — so there is no way to send a
  value *into* a paused remote body. Locally, plain Python generators do it
  fine.
- **F5 × L3, merging a growing set of streams.** No fan-in verb exists; you
  hand-roll it with a `TaskGroup` and an `asyncio.Queue`. See
  [F5](/python-sdk/architecture/f5-async-generators.md) for the working, clunky version.

The SDK's own matrix marks both of those cells "explore" rather than GAP —
they are reachable by hand today. Only the F7 row is marked GAP outright.
See [F7 — the run object](/python-sdk/architecture/f7-run-object.md).

The canonical, runnable version of this matrix lives in the SDK repo at
`docs/execution-patterns.md`, with explorations `examples/03_body_kinds.py`
through `examples/06_dynamic_graph.py`
([PR #18](https://github.com/dreamlake-ai/dreamlake-lakeshore/pull/18)).

---

Source: https://lakeshore.dreamlake.ai/python-sdk/udf

# `@udf` decorator

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore import udf
```

One decorator. The function's own shape — sync, async, generator, or
async generator — picks the execution shape; there is no mode flag to
forget and no second decorator.

```python
udf(fn=None, *, queue: str | None = None, transport: str = "auto")
```

Those are the only parameters. There is no `mode=`, `name=`, `retries=`,
`timeout=`, `image=`, or `resources=` on `@udf`.

| Form | Plain call does |
| --- | --- |
| `@udf` (bare) | runs the body in-process and returns the result |
| `@udf(queue="gpu")` | submits and waits — equivalent to `.remote(...)` |

## Bare `@udf` — local by default

A plain call runs in-process, exactly like the undecorated function.
No control plane, no daemon, no network — the local tier imports
nothing distributed.

```python
@dls.udf
def stats(a: list[float]) -> dict:
    return {"mean": sum(a) / len(a)}

stats([1.0, 2.0, 3.0])   # {'mean': 2.0}
```

Every plain call runs inside a **fork** of the ambient run context — its
own key ledger — so sequential calls in one `dls.scope` never
cross-contaminate the return-your-keys check.

## The four body kinds

Detected once at decoration with `inspect`; each dispatches naturally:

```python
@dls.udf
def f(x): ...                 # sync       -> the value

@dls.udf
async def g(x): ...           # async      -> coroutine, await it

@dls.udf
def h(x):                     # generator  -> iterator
    yield ...

@dls.udf
async def k(x):               # async-gen  -> async iterator
    yield ...
```

The `UDF` object exposes the detection as `.is_async`, `.is_gen`, and
`.is_asyncgen`, alongside `.queue` and `.transport`.

For a generator body, the run context is installed only around each
`next()` — never while control is back with the consumer — so a yield
cannot leak the call's scope into consumer code.

## Data UDFs — string keys and the `dls.run` seam

A data UDF never sees a bucket, a credential, or an absolute path. It
asks the ambient run context for local paths and returns plain string
**keys**:

```python
@dls.udf
async def splats_to_mesh(splats: str) -> str:
    mesh = extract(dls.run.read(splats))            # key -> local Path
    mesh.save(dls.run.write("process/mesh.ply"))    # key -> local Path
    return "process/mesh.ply"                       # a KEY, not a path
```

Keys are prefix-relative POSIX strings (`"process/mesh.ply"`). An
absolute key is rejected; `../` may climb to a parent scope but never
out of the run root. Keys resolve to `root / prefix / key` on the local
filesystem, and file bytes never ride the msgpack wire — only keys do.

### `dls.run` surface

| Member | Signature | Does |
| --- | --- | --- |
| `run.write` | `(key, *, dir=False) -> Path` | Register an output key; return the local path to write at. Parents created; `dir=True` makes the directory itself. |
| `run.read` | `(key) -> Path` | Resolve an input key to a local path. Raises `KeyError_` if it does not exist. |
| `run.read_all` / `run.write_all` | `(keys) -> same shape` | Shape-mirroring: `str -> Path`, `list -> list`, `dict -> dict`. |
| `run.prefix` / `run.root` | properties | The ambient prefix (`str`) and filesystem root (`Path`). |

> **Warning:** `RunContext` has a `storage` field and a fetch-on-read path, but nothing
> binds a `Storage` to it today — `run_worker` installs a context with
> `storage=None`. In practice a worker's `dls.run.read/write` resolve
> against the worker's own filesystem under its `--root`, so a remote data
> UDF needs a shared or mounted filesystem. The automatic
> upload-recorded-writes side channel is designed but not implemented.

### `dls.scope` — set the prefix

```python
dls.scope(sub: str = "", *, root: Path | str | None = None)
```

```python
with dls.scope("scenes/0007"):                     # prefix = scenes/0007
    key = splats_to_mesh("source/splats.ply")

with dls.scope(scene_root, root="/data"):          # nesting joins prefixes
    for i in range(num_episodes):
        with dls.scope(f"outputs/{i:04d}"):
            ...
```

`root=` re-anchors the filesystem root — use it on the outermost scope
of a local run. Each scope tracks its own writes, so the key contract
stays per-invocation.

## The return-your-keys contract

The contract is **one-directional**: every key written through
`dls.run.write` must appear in the returned manifest. A returned string
that was never written is fine — it may be plain data, or an upstream
key passed through.

Accepted manifest shapes: `str`, `list[str]`, `tuple[str, ...]`, or
`dict[str, str]` (named outputs; the values are the keys). Violations
raise `KeyError_`, a subclass of `ValueError`.

| You did | You get |
| --- | --- |
| wrote via `dls.run.write`, returned a non-manifest value | `KeyError_: UDF wrote N file(s) via dls.run.write (...) but did not return their keys` |
| wrote a key you left out of the manifest | `KeyError_: UDF wrote key(s) it did not return: [...]` |
| returned a key you never wrote | allowed — plain data or a pass-through key |
| returned plain data, wrote nothing | allowed — not every UDF is a data UDF |

A typo'd return is still caught: the correctly written key goes
unreturned and fails.

Multi-output shapes are stdlib primitives:

```python
@dls.udf
def poses_to_views(splats: str, poses: str) -> dict[str, str]:
    rgb = dls.run.write("process/images/rgb", dir=True)
    depth = dls.run.write("process/images/depth", dir=True)
    render(splats, poses, rgb=rgb, depth=depth)
    return {"rgb": "process/images/rgb", "depth": "process/images/depth"}
```

## Streaming bodies

A generator or async generator yields **partial manifests** — each yield
is shipped as one journal frame, and the yields concatenate into THE
manifest for the key contract:

```python
@dls.udf(queue="gpu")
async def traj_to_ego_views(traj: str):
    for i, frame in enumerate(render(dls.run.read(traj))):
        frame.save(dls.run.write(f"images/{i:04d}.png"))
        yield [f"images/{i:04d}.png"]        # one VALUE frame, live
```

If the yields are not key manifests (a plain data stream), the contract
is skipped for that invocation. Consumers iterate the journal while the
job runs — see [streaming](/python-sdk/invocations.md#streaming).

## `@udf(queue=...)` — remote by default

```python
@dls.udf(queue="gpu")
def embed(x: str) -> list[float]: ...

embed("hello")               # plain call = submit + wait for the result
inv = embed.submit("hello")  # non-blocking -> durable invocation id (str)
```

A plain call on a queue-bound UDF submits and waits. An `async def` or
async-generator body returns an awaitable instead of blocking.

> **Warning:** `.remote()` (and therefore a plain call on a queue-bound UDF) fetches
> the result with the dispatch plane's default `timeout=30.0`. For jobs
> that run longer, use `inv = f.submit(...)` and then
> `q.result(inv, timeout=...)`.

### `.submit(*args, **kwargs) -> str`

Returns the durable invocation id immediately. Control kwargs are
**underscore-prefixed** so they cannot collide with your function's own
keyword arguments:

| Kwarg | Does |
| --- | --- |
| `_key` | Idempotency key — resubmitting the same key on the same queue returns the existing invocation id, no duplicate job. |
| `_priority` | Float attached to the row. `fifo` and `filo` claim highest priority first; `boltzmann` samples proportional to `exp(priority / T)`. Default `0.0`. |
| `_dispatch` | Override the dispatch plane (default: the process plane — see [Dispatch planes](/python-sdk/dispatch.md)). |

`submit` on a UDF with no queue binding raises:

```text
TypeError: embed has no queue binding; decorate with @udf(queue=...) or
pass one at submit time
```

Arguments ride `wire.pack` inside an `Envelope` — ndarray, tensor,
image, and tuple all survive the round trip. The ambient
`dls.run.prefix` travels in the envelope, so the worker resolves keys
exactly as you would have locally.

### `.remote(*args, _dispatch=None, **kwargs)`

Submit and wait. Sync and generator bodies block and return the value;
async and async-generator bodies return an awaitable.

### `.local(*args, **kwargs)`

Force the local tier even on a queue-bound UDF — debugging and unit
tests stay plain function calls:

```python
@dls.udf(queue="gpu")
def add(a: int, b: int) -> int:
    return a + b

add.local(2, 3)   # 5 — no queue, no network
```

`.local()` builds a fresh unbound `UDF` around the same function, which
also drops the original `transport=` setting. That is irrelevant for
local execution but worth knowing.

## How the function crosses the wire

`transport=` on the decorator picks the encoding of the callable itself.
Arguments, keyword arguments, results, and stream frames always use the
msgpack/ExtType codec — only the callable is ever pickled.

| `transport` | Ships | Fails when |
| --- | --- | --- |
| `"auto"` (default) | `ref` when the callable is importable, else `pickle` | — |
| `"ref"` | `module:qualname` plus a `src_hash` version stamp | the callable is a lambda, a `<locals>` closure, a `__main__` def, or a callable instance |
| `"pickle"` | a cloudpickle snapshot plus a runtime stamp | the blob exceeds 32 KiB |

Anything else raises
`TransportError: unknown transport '...'; use "ref" | "pickle" | "auto"`.

Two guardrails on the `pickle` path:

- **Size.** `INLINE_PICKLE_LIMIT` is 32 KiB. A bigger blob means the
  callable captured large state; pass the data as arguments instead.
- **Trust.** Workers refuse pickled callables unless started with
  `allow_pickled=True`. `run_worker(allow_pickled=None)` — the default —
  resolves by trust domain: allowed on the local SQLite plane, refused
  on an HTTP plane.

On the `ref` path the worker compares the submitter's `src_hash` with
the source it resolved and fails loud on a mismatch, so a stale worker
checkout cannot silently run different code. The
[`runtime_policy`](/python-sdk/dispatch.md#the-worker-loop) flag
(`"strict" | "minor" | "off"`, default `"minor"`) governs that check.

When the plane exposes a code channel (`HttpDispatch`), a `ref` submit
also pushes the current git tree so a fresh worker can import your
modules — see [code mount](/python-sdk/happy-paths/10-code-mount.md).

## `dls.thunk` — an ad-hoc callable

For a lambda or a closure you did not decorate, `dls.thunk` bundles the
callable with its arguments for by-value submission:

```python
t = dls.thunk(lambda: score(model, val_ds))
inv = q.submit(t)                 # -> durable string id
val = q.result(inv, timeout=60)
```

`thunk(fn, /, *args, **kwargs)` returns a `Thunk`. Only the callable is
pickled; the arguments ride the codec. Passing a `UDF` unwraps to its
inner function. The receiving worker must allow pickled functions.

## Read next

- [Queue API](/python-sdk/queue.md) — the queue the id lives on.
- [Invocation ids](/python-sdk/invocations.md) — results, reconnect-by-id,
  fan-out + collect.
- [Dispatch planes](/python-sdk/dispatch.md) — where submits actually go,
  and how to run a worker.

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths/03-hello-local-stack

# 03 · Hello, local stack

"Your laptop is a Lakeshore." Brings the whole stack up on `localhost`
so you can iterate without touching a hosted control plane.

For a server setup with authentication and a DreamLake connection, follow [Setting Up Lakeshore Service](https://docs.dreamlake.ai/lakeshore/setting-up-lakeshore-service).

## Exercises

- Mongo via `docker compose` (the `db` pane).
- Control-plane boot — `pnpm dev` inside `lakeshore-controlplane`.
- Nymph registration over `POST /v1/daemon/hello`.
- A `lakeshore daemon list` round trip, proving CLI → control plane →
  daemon.

## Requires

- Docker running.
- The container repo cloned with submodules.
- `pnpm`, `uv`, and `mprocs` on `PATH` (`brew install mprocs`).
- `lakeshore-controlplane/.env` present, or the mprocs `lakeshore` pane's
  built-in defaults. The pane sources `.env` when it exists and then
  fills in `DATABASE_URL` /`MONGODB_URI` defaults for anything unset *or
  empty* — the `.env.example` placeholders are empty strings, which would
  otherwise leak through and break Prisma.

## Verifies

The control plane and one Rust daemon are alive and talking, both
reachable from the CLI on `localhost:8080`.

## Run

```bash
git clone --recurse-submodules \
  https://github.com/dreamlake-ai/lakeshore-workspace.git
export LAKESHORE_WORKSPACE=~/lakeshore-workspace

cd $LAKESHORE_WORKSPACE
mprocs
# `db` autostarts. Press `s` on the `lakeshore` pane, then on `nymph`.
```

Then, in another shell:

```bash
export LAKESHORE_URL=http://localhost:8080
lakeshore daemon list
```

> **Warning:** `LAKESHORE_SERVER` is read only by the `lakeshore auth` commands. Every
> other command — including `daemon list` — resolves the control plane
> from `LAKESHORE_URL`, and sends no bearer token when it is set.

## Expected output

```text
✓ 1 daemon in 'default':

label         state   runners  queues  tags        version  seen  machine_id
Datium.local  active  process  —       host=local  0.1.4    2s    Datium.local
```

One row, state `active`, `seen` measured in seconds. `queues` shows `—`
because an empty membership array means the default queue — there is no
literal `"default"` entry.

## Then smoke it

```bash
LAKESHORE_URL=http://localhost:8080 ./scripts/smoke-exec.sh
```

Ten sections covering stdout, stderr separation, exit codes, env
passthrough, stdin, workdir writes, the timeout sentinel, a long-poll
that spans multiple windows, and host Python. It bails on the first
failure and prints the expected-versus-actual pair.

## Alternative: no TUI

```bash
./scripts/dev-up.sh      # background processes; logs in /tmp/lakeshore-dev/
./scripts/dev-down.sh
```

Idempotent — re-running skips anything already up.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `db` pane: "container name already in use" | A stale standalone `dreamlake-udf-mongo` container. Run the `mongo-clear` pane, then restart `db`. |
| Control-plane pane: `DATABASE_URL is required` | `.env` exists but sets `DATABASE_URL=""`, and something overrode the pane's default. Unset it or give it a real value. |
| `nymph` pane fails with `gcc-11 not found` | A stale `CC` export. The pane forces `CC=clang`; if you are running cargo by hand, `unset CC` first. |
| `daemon list` returns no rows | Nymph started but never reached the control plane. Check the nymph pane's `--server` flag and that the control plane is listening on 8080. |
| `daemon list` returns 401 | The control plane has `LAKESHORE_ADMIN_TOKEN` set, so it is not in open mode. Either clear it in `.env` for local dev, or mint a token — see [15 · Token lifecycle](/cli/happy-paths/15-token-lifecycle.md). |
| `daemon list` says "server not configured" | Neither `LAKESHORE_URL` nor a saved `auth login` is present. |

## Status

Manual.

## Next

→ [04 · Smoke-exec cascade](/cli/happy-paths/04-smoke-exec-cascade.md)

---

Source: https://lakeshore.dreamlake.ai/cli/examples/queues-and-jobs

# CLI examples — Queues and jobs

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives
on [Auth + setup](/cli/examples/auth.md).

## Queues

A **queue** is the unified scheduling primitive: a named pipe of
pending Invocations that daemons subscribe to by name. Workers register
themselves into queues via `Worker.queues: string[]`; the empty array
means the **default queue** (there is no sentinel name). See
[Queues](/get-started/queues.md) and [`/dev/queues`](/dev/queues) for the
concept and design pages.

> **Warning:** It is `lakeshore queues ls` and `lakeshore queues rm` — **not** `list`
> and `remove`. Every other group in the CLI uses `list` / `remove`; this
> one doesn't.

The full verb set is `add`, `ls`, `show`, `patch`, `archive`,
`unarchive`, `drain`, `rm`, `events`, `stats`.

### `queues add` — create a queue

The bare minimum is a name (matching `^[a-z0-9][a-z0-9-]*$`).
`--kind` is one of `fifo` (default), `priority`, `boltzmann`, `filo`.

```bash
lakeshore queues add training-h100
lakeshore queues add eval --kind priority --description "evaluation, urgent first"
```

### `queues add` — with an elasticity declaration

`--elasticity` accepts `fixed`, `fully-elastic`, `pool-with-threshold`,
or `max-count`, with `--min`, `--max`, and `--threshold` (0.0–1.0)
supplying its parameters. `--instance-type`, `--image-id`, and
`--runner` seed the daemon template used when the queue scales up.
`--max-depth` plus `--on-full` (`reject` or `block`) bound the backlog.

```bash
lakeshore queues add training-h100 \
  --kind priority \
  --provider my-gpu-provider \
  --elasticity pool-with-threshold \
  --min 2 --max 16 --threshold 0.7 \
  --instance-type p4d.24xlarge \
  --max-depth 1000 --on-full reject
```

> **Warning:** The `--elasticity` flag takes **hyphenated** values
> (`fully-elastic`, `pool-with-threshold`, `max-count`), while the
> `elasticity.kind` field in `lakeshore.yaml` accepts only the
> **underscored** spellings (`fully_elastic`, `pool_with_threshold`,
> `max_count`). They are not interchangeable.

### `queues ls` — list queues

`--state` filters on `active`, `draining`, `paused`, or `archived`.

```bash
lakeshore queues ls
lakeshore queues ls --state active
lakeshore queues ls --json | jq -r '.[].name'
```

The table shows `NAME / KIND / STATE / MEMBERS / PROVIDER /
ELASTICITY / DESCRIPTION`. `MEMBERS` is the count of workers whose
`queues` array contains this queue's name.

### `queues show` — one queue and its members

```bash
lakeshore queues show training-h100
lakeshore queues show training-h100 --json
```

### `queues patch` — mutate a queue in place

Accepts `--description`, `--provider`, `--min`, `--max`,
`--threshold`, `--max-depth`, `--on-full`, and `--json`. Note it does
**not** accept `--kind` or `--elasticity`.

```bash
lakeshore queues patch training-h100 --description "h100 prod"
lakeshore queues patch training-h100 --max 32 --threshold 0.6
lakeshore queues patch training-h100 --max-depth 2000 --on-full block
lakeshore queues patch training-h100 --provider ""    # clear the provider
```

### `queues drain` / `archive` / `unarchive`

`drain` stops admitting new work but lets in-flight invocations finish.
`archive` soft-deletes — the row survives but stops accepting work.
`unarchive` re-activates a drained or archived queue. All three take a
name and `--json`.

```bash
lakeshore queues drain training-h100
lakeshore queues archive eval
lakeshore queues unarchive eval
```

### `queues rm` — delete

Errors if any worker is still bound to the queue. `--force` cascades —
it scrubs the queue name out of every `Worker.queues` array first.

```bash
lakeshore queues rm eval
lakeshore queues rm eval --force
```

### `queues events` — the elasticity audit trail

One row per controller decision. `tick_noop` rows are hidden unless you
pass `--verbose`, which is the quickest way to check whether the
controller is running at all. `--limit` defaults to 50, capped at 200.

```bash
lakeshore queues events training-h100
lakeshore queues events training-h100 --verbose --limit 200
lakeshore queues events training-h100 --json
```

### `queues stats` — current signals

A point-in-time view of what the controller is looking at: daemon
count, utilization, pending invocations, and the last few non-noop
scale actions.

```bash
lakeshore queues stats training-h100
lakeshore queues stats training-h100 --json
```

## Jobs

Invocations tracked by the control plane — the things `@dls.udf` calls
queue up.

### `jobs list` — recent invocations

`--limit` is 1–200 (default 50) and `--offset` defaults to 0.

```bash
lakeshore jobs list
lakeshore jobs list --limit 20 --offset 20
```

### `jobs list` — filters

`--state` is repeatable and accepts `queued`, `running`, `succeeded`,
`failed`, `killed`, `timeout`. `--worker` and `--queue` take ids.

```bash
lakeshore jobs list --state running --state queued
lakeshore jobs list --worker "$D"
lakeshore jobs list --json | jq '.[] | {id, state, worker}'
```

### `jobs show` — one invocation

```bash
JOB=$(lakeshore jobs list --json | jq -r '.[0].id // empty')
lakeshore jobs show "$JOB"
```

### `jobs kill` — cancel one invocation

Queued jobs are marked `killed` immediately. **Running jobs return
409** — to stop in-flight work, kill the process on the daemon
(`lakeshore daemon exec`) or drop the daemon itself.

```bash
lakeshore jobs kill "$JOB"
```

### `jobs bulk-cancel` — cancel everything queued

Only touches rows in state `queued`; running work is untouched.

```bash
lakeshore jobs bulk-cancel
lakeshore jobs bulk-cancel --queue training-h100
lakeshore jobs bulk-cancel --json
```

## Read next

- [Queues](/get-started/queues.md) — the concept.
- [Elasticity](/get-started/elasticity.md) · [Scaling rules](/get-started/scaling-rules.md)
- [Daemons + exec/run](/cli/examples/daemons-and-exec.md) — binding workers to queues.

---

Source: https://lakeshore.dreamlake.ai/cli/tui

# `lakeshore tui`

> A terminal dashboard mirroring the web UI — Workers, Queues,
> Invocations, and Storage in four tabs — where every list row
> advertises, inline, the actions you can take on it.

```bash
lakeshore tui                # opens on the Workers tab
lakeshore dashboard          # alias
lakeshore top                # alias
```

`tui` is a live, read-and-act dashboard built on
[Ink](https://github.com/vadimdemedes/ink). React and Ink are imported
lazily, so the rest of the CLI never pays the cost of loading them.

The command needs an interactive terminal and a configured control
plane, and it checks both **before** rendering anything:

- Under a pipe or in CI (stdout is not a TTY) it exits 2 with a message
  pointing you at the non-interactive equivalents — `lakeshore daemon
  list`, `lakeshore queues ls`, `lakeshore jobs list`, `lakeshore
  storage list` — instead of spewing escape codes.
- With no server configured it exits 2 telling you to run
  `lakeshore auth login --server <url>` or set `LAKESHORE_URL`.

## Flags

| Flag                   | Default   | Meaning                                                                                                        |
| ---------------------- | --------- | ---------------------------------------------------------------------------------------------------------------- |
| `--interval <seconds>` | `4`       | Live-refresh interval for the active tab. Clamped to a minimum of 1 second.                                     |
| `--tab <name>`         | `workers` | Start on a specific tab: `workers`, `queues`, `invocations`, or `storage`. An unknown value warns and falls back to `workers`. |

```bash
lakeshore tui --tab queues --interval 2
```

## The four panes

The dashboard renders a tab bar (`lakeshore · <namespace>`), the active
pane, a confirm/status line, and a footer legend. **All four tabs load
once at startup; only the active tab re-polls** on the interval.

| Tab             | Columns                                                                               |
| --------------- | ------------------------------------------------------------------------------------- |
| **Workers**     | `LABEL`, `STATE`, `RUNNERS`, `QUEUES`, `VER` (daemon version), `SEEN` (last-seen age) |
| **Queues**      | `NAME`, `KIND`, `STATE`, `MEMBERS`, `PROVIDER`, `ELASTICITY`                          |
| **Invocations** | `ID`, `STATE`, `WORKER`, `QUEUE`, `ATT` (attempt), `SUBMIT` (age)                     |
| **Storage**     | `NAME` (`<ns>/<name>`), `KIND`, `TARGET` (`s3://bucket/prefix`), `AGE`                |

Columns are fixed-width; the inline action hint for each row is
appended after the cells. `LABEL` falls back to the first 12 characters
of the `machine_id` when a worker has no label, and `VER` reads
`capabilities.versions.daemon` (shown as `—` when absent).

## Keybindings

| Key                 | Action                                                             |
| ------------------- | ------------------------------------------------------------------ |
| `Tab` / `Shift+Tab` | Next / previous tab                                                |
| `1`–`4`             | Jump to a tab by number (Workers / Queues / Invocations / Storage) |
| `↑` / `↓`           | Move the selection within the list                                 |
| `R` (capital)       | Refresh the active tab now                                         |
| _letter_            | Run the matching inline action on the **selected** row (see below) |
| `q` / `Ctrl-C`      | Quit                                                               |

Navigation deliberately uses the **arrow keys only** — the letter keys
are reserved for per-item actions, so a worker row's `[k]ill` really
does fire on `k`.

## Inline actions

Each row prints its available actions as bracketed key hints — e.g. a
live provider-backed worker shows `[k]ill [t]erminate [h]ibernate
r[e]set`, while a terminal worker shows just `[r]emove`. The hint
string and the keypress dispatch are derived from the **same** action
table, so they can't drift apart. Availability is item-aware:
`[u]narchive` only appears on an archived (or draining) queue,
`[t]erminate` only on a worker a provider backs.

**Destructive** actions open a confirm bar (`… [y/n]`) that swallows
every other key until you answer `y`, `n`, or `Esc`; non-destructive
ones run immediately. After an action the active tab re-polls and a
one-line `✓`/`✗` status reports the outcome.

### Workers

| Key | Action    | Destructive | What it does                                                                                                  |
| --- | --------- | ----------- | --------------------------------------------------------------------------------------------------------------- |
| `k` | kill      | yes         | Drops the Worker row; the cloud instance (if any) stays up.                                                    |
| `t` | terminate | yes         | Drops the row **and** terminates the cloud instance. Offered only when `providerName` is set on the worker.    |
| `h` | hibernate | no          | Hibernates the daemon for 30 minutes (`+30m`).                                                                 |
| `e` | reset     | no          | Resets poll backoff on a daemon stuck in exponential backoff.                                                  |
| `r` | remove    | yes         | Drops a **terminal** (`gone` / `terminated`) Worker row. Shown only on terminal workers.                       |

`kill` / `terminate` / `hibernate` / `reset` are offered only on
non-terminal workers; `remove` only on terminal ones.

### Queues

| Key | Action    | Destructive | What it does                                                                                                        |
| --- | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------- |
| `d` | drain     | no          | Finish in-flight work, stop admitting new. `active` queues only.                                                    |
| `a` | archive   | yes         | Soft-delete; stops accepting work. Non-archived queues only.                                                        |
| `u` | unarchive | no          | Re-activate. Archived or draining queues only.                                                                      |
| `r` | remove    | yes         | Delete the queue. Cascades — drops membership from its workers when the queue has members, so the delete can't 409. |

### Invocations

| Key | Action            | Destructive | What it does                                                                                                    |
| --- | ----------------- | ----------- | --------------------------------------------------------------------------------------------------------------- |
| `k` | kill              | yes         | Kill the selected invocation. Queued → killed; a running invocation returns 409. Non-terminal invocations only. |
| `c` | cancel-all-queued | yes         | Bulk-cancel **all** queued invocations in the namespace, regardless of which row is selected.                   |

### Storage

| Key | Action | Destructive | What it does                                                           |
| --- | ------ | ----------- | ---------------------------------------------------------------------- |
| `r` | remove | yes         | Remove the storage **record** only — the S3 bucket itself is retained. |

These actions mirror the web dashboard's per-resource menus, so the
control-plane calls are identical to clicking the equivalent button in
the browser UI.

## Read next

- [`lakeshore` CLI](/cli.md) — the full command surface.
- [`nymph tui`](/nymph/status-tui.md) — a different TUI: one daemon,
  read-only, talks to the daemon's local introspection endpoint rather
  than the control plane.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — the non-interactive
  install / list / kill / cleanup verbs.

---

Source: https://lakeshore.dreamlake.ai/get-started/tutorials/compose

# Compose a cluster

> **Example code:** [`08-queues-cpu-vs-gpu`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/08-queues-cpu-vs-gpu) and [`14-compose-cluster`](https://github.com/dreamlake-ai/lakeshore-examples/tree/main/14-compose-cluster)

Declare the cluster you want — queues, daemon groups, instance types,
counts — in a single `lakeshore.yaml`. One verb reconciles; another
tears it down. The mental model is docker-compose for daemon fleets.

## Prerequisites

| Requirement | Why |
| --- | --- |
| CLI installed (`npm i -g @dreamlake/lakeshore`) | `up` / `down` / `ps` / `status` / `logs` live here. |
| Authenticated (`lakeshore auth login --server <url>`) | The CLI needs a control plane to reconcile against. |
| A provider registered (`lakeshore providers add … --launcher EC2`) | Daemons need somewhere to launch. |

## Step 1 — Write a `lakeshore.yaml`

Create the file in the directory you will run `lakeshore up` from — the
lookup is `./lakeshore.yaml` with no upward walk.

```yaml file="lakeshore.yaml"
version: 1
project: my-first-cluster
provider: ec2-prod

queues:
  work:
    kind: fifo
    description: "General work queue"

daemons:
  worker:
    queues: [work]
    count: 2
    instance_type: t3.medium
    keep_alive_s: 600
```

Every field maps to an existing primitive:

- `project` is written to `composeProject` on every Worker row this file
  spawns, so the compose verbs scope to just this cluster.
- `queues` declares queues to create; creation is idempotent.
- `daemons` declares groups. Each group gets labels `<name>-0` through
  `<name>-(count-1)`.

`version: 1`, `project`, and a `daemons` block with at least one group
are all required. `count` is required per group and has no default.

## Step 2 — Bring it up

```bash
lakeshore up --wait
```

The CLI reads the file, creates any missing queues, then launches the
delta of daemons. `--wait` blocks until each one reaches `state=active`.

Run it again and nothing happens — desired already matches actual. `up`
is idempotent, and state is re-derived from the control plane on every
run (there is no lock file).

Preview first with `lakeshore up --plan`, which prints the plan and takes
no side effects.

## Step 3 — Inspect with `ps` and `status`

```bash
lakeshore ps          # every daemon owned by this compose
lakeshore status      # desired vs actual drift, read-only
```

Both take `--json`. Ad-hoc daemons launched with `lakeshore daemon
launch` have a null `composeProject` and never show up here.

## Step 4 — Scale

Edit `count: 2` to `count: 4` and re-run:

```bash
lakeshore up --wait
```

It sees two existing daemons, wants four, and launches the delta of two.
The originals are untouched. To reconcile only one group:

```bash
lakeshore up worker
```

## Step 5 — Tear it down

```bash
lakeshore down                 # drain — instances stay alive
lakeshore down --terminate     # also terminate the cloud instances
```

Draining is the default: `down` removes the Worker rows this project
owns without killing the machines. `--terminate` kills the EC2 / GCE /
Kube instances too. Ad-hoc daemons are untouched either way.

Optionally prune the queues:

```bash
lakeshore down --terminate --prune-queues
```

`--prune-queues` only deletes a declared queue when it has zero remaining
members, so a queue another project also joined stays alive.

## Multi-queue example

A CPU tier and a GPU tier:

```yaml file="lakeshore.yaml"
version: 1
project: training-cluster
provider: ec2-prod

queues:
  cpu-pool:
    kind: fifo
    description: "CPU preprocessing"
  gpu-t4:
    kind: priority
    description: "T4 GPUs for training"

daemons:
  cpu:
    queues: [cpu-pool]
    count: 5
    instance_type: t3.medium
    keep_alive_s: 600
  gpu:
    queues: [gpu-t4]
    count: 3
    instance_type: g4dn.xlarge
    image_id: ami-012ba162b9cd2729c
    keep_alive_s: 1800
```

```bash
lakeshore up --wait          # 2 queues + 8 daemons
lakeshore up gpu             # re-reconcile only the gpu group
lakeshore up --plan          # dry-run
lakeshore down --terminate
```

## Setup, startup, and Python version

`setup` is a **list** of bash snippets run on the host once after the
daemon registers — they arrive as setup commands and drain FIFO, one per
poll. `startup` is a Python-environment script; `python` names the target
version.

```yaml file="lakeshore.yaml"
version: 1
project: ml-training
provider: ec2-prod

queues:
  training:
    kind: fifo

daemons:
  gpu:
    queues: [training]
    count: 2
    instance_type: g4dn.xlarge
    image_id: ami-012ba162b9cd2729c
    python: "3.12"
    setup:
      - apt-get update -y && apt-get install -y python3-pip
      - pip3 install --break-system-packages dreamlake-lakeshore torch torchvision --index-url https://download.pytorch.org/whl/cu121
    startup: "pip install -q transformers datasets"
```

> **Warning:** A single YAML block scalar under `setup:` fails validation — split it
> into list entries. Any key inside a daemon group other than `queues`,
> `count`, `provider`, `instance_type`, `image_id`, `runner`,
> `keep_alive_s`, `setup`, `startup`, `python`, and `mounts` is a hard
> error naming the allowed set. In particular there is **no `mode:` key**
> and **no `tags:` key** on a daemon group.

## Mounting storage into a group

```yaml file="lakeshore.yaml"
daemons:
  train:
    queues: [training]
    count: 4
    instance_type: g5.2xlarge
    mounts:
      - storage: checkpoints
        mount_path: /mnt/checkpoints
      - storage: datasets
        mount_path: /mnt/data
        readonly: true
```

`storage` names a [Storage](/get-started/storages.md) entry (optionally
`<namespace>/<name>`); `mount_path` and `readonly` are the other two
keys.

## Declaring elasticity

Elasticity lives on the queue, not on the daemon group. In YAML the
kinds are **underscored**:

```yaml file="lakeshore.yaml"
queues:
  training:
    kind: fifo
    elasticity:
      kind: pool_with_threshold
      min: 8
      max: 20
      threshold: 0.8
```

Only `kind`, `min`, `max`, and `threshold` survive the loader —
`pool`, `match`, `steal_delay_s`, `cost_weight`, and any `admission:`
block are silently dropped. Set those by PATCHing the queue over HTTP;
see [Elasticity](/get-started/elasticity.md#setting-the-knobs). And
remember the controller is off unless the control plane runs with
`ELASTICITY_ENABLED=true`.

## Verb reference

| Command | What it does |
| --- | --- |
| `lakeshore up [group]` | Reconcile to desired state (idempotent). |
| `lakeshore up --plan` | Dry-run — print what would change. |
| `lakeshore up --wait` | Block until every spawned daemon is active. |
| `lakeshore down` | Remove this project's workers; leave instances alive. |
| `lakeshore down --terminate` | Also terminate the cloud instances. |
| `lakeshore down --prune-queues` | Delete declared queues that have no members left. |
| `lakeshore ps` | List daemons owned by this compose. |
| `lakeshore status` | Desired vs actual drift (read-only). |
| `lakeshore logs <group\|label>` | Multiplexed cloud-console output. `--follow`, `--poll-ms <n>`. |

All read `./lakeshore.yaml`; override with `-f <path>`. `up`, `down`,
`ps`, and `status` accept `--json`.

## Read next

- [Compose](/get-started/compose.md) — the concept page and full schema.
- [`/dev/compose`](/dev/compose) — the design doc.
- [Queues](/get-started/queues.md) — the primitive compose wraps.
- [Fan out](/get-started/tutorials/fan-out.md) — put work on the queue you
  just created.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/primitives

# Dataflow primitives

> **Warning:** `import lakeshore`, `import dreamlake as dl`, `@dl.pipeline`, `batch`,
>   `stack`, `to_dataset`, `requeue`, `trace`, and `@udf(kind=…)` exist only
>   in the runnable draft at `design/primitives/` in
>   [`macrodata-subtask_yanbing`](https://github.com/fortyfive-ai/macrodata-subtask_yanbing/pull/12).
>   The shipped package is `dreamlake-lakeshore`, imported as
>   `import dreamlake.lakeshore as dls`, and its `udf()` accepts exactly
>   `queue=` and `transport=`. Nothing on this page is installable from PyPI.

Three contract pipelines (`image_object_annotation`,
`camera_pose_trajectory`, `video_task_annotation`) already *import* a
primitive API that did not exist. The draft makes it real: tracing a
pipeline yields a `Graph` whose nodes are colored by one shared six-hue
palette. The noun the author types is the node the tracer graphs and the hue
the canvas paints.

This is the pipeline the draft traces, close to verbatim:

```python file="design/primitives/examples/image_object_annotation.py"
import dreamlake as dl
import lakeshore as ls
from dreamlake import batch, requeue, to_dataset
from lakeshore.types import Mask, Tensor, Tuple

@ls.udf
def detect_objects(images: Tensor["N", "H", "W", 3]) -> Tuple["boxes", "classes", "confidence"]:
    ...

@ls.udf(kind="review")                      # review folds onto the filter bucket (amber)
def review_boxes(labels) -> Mask["R", "N"]:
    ...

@dl.pipeline
def image_object_annotation(source):
    for items in batch(source, n=64):
        labels = detect_objects(items.images)
        review = review_boxes(labels)

        consensus = review.all(axis=0)      # ∩ over reviewers
        good = labels[consensus]
        needs_attn = labels[~consensus]     # complement ¬

        yield to_dataset(good)              # sink
        requeue(needs_attn)                 # rework loop = plain Python
```

## The set

The draft splits into two namespaces: `lakeshore` is compute, `dreamlake` is
dataflow built on top of it.

| Primitive | Meaning | Bucket / hue |
| --- | --- | --- |
| `@udf` / `@udf(kind=…)` | compute node; `kind` sets its color everywhere | transform `#23aaff` · model `#7c5bd9` |
| `udf.local(...)` | bypass the trace, run the body in-process | — |
| `batch(src, n=)` | source yielding fixed-size batches | source `#1f8f4a` |
| `stack([…])` | fan-in on a new leading axis | transform `#23aaff` |
| `x > t` · `&` `\|` `~` · `.all/.mean(axis=)` · `x[mask]` | result algebra: predicates, set algebra, consensus, select | filter `#c0922e` |
| `to_dataset(x)` | data leaves the graph | sink `#c8513b` |
| `requeue(x)` | rework loop-back (dashed edge to the source) | idle `#9c907a` |
| `@pipeline` / `trace()` | driver; plain Python control flow *is* the DAG | — |
| `Tuple["a","b"]` · `Tensor` · `Mask` · `String` | shape- and column-carrying annotations | — |

Six hues, no seventh — that is the palette contract in
`uikit-workspace/staging/design.md`, which forbids a seventh hue outright.
Aliases fold by *function*, not by vibe: `review` → filter, `merge` → model,
`select` → filter, `requeue` → idle. The mapping lives in one table,
`KIND_COLOR`, with `KIND_ALIAS` beside it.

## What tracing yields

Under a trace, a `@udf` call does not run its body: it appends a node to the
active `Graph` and returns a `Col`, a lazy handle. Outside a trace the same
call runs the real body. All three contract pipelines trace to non-empty
graphs:

| Pipeline | Nodes | Edges |
| --- | --- | --- |
| `image_object_annotation` | 9 | 11 |
| `camera_pose_trajectory` | 13 | 17 |
| `video_task_annotation` | 17 | 29 |

The set algebra shows up as amber filter nodes, `requeue` as a dashed
loop-back into the source, and the renderer emits both 24-bit-color ASCII
(for the REPL) and Graphviz DOT (the canvas handoff). Both read color from
the same `KIND_COLOR` table, so Python, terminal, and canvas cannot drift.

```bash
python design/primitives/run_trace.py       # trace the 3 contract pipelines
python -m pytest design/primitives/tests    # the contract, locked (6 tests)
```

The six tests lock exactly the properties the design depends on: every
pipeline traces to a non-empty graph, every node kind resolves to a palette
bucket, source and sink are present and `requeue` loops back, `kind="review"`
folds onto `filter`, named columns flow on edges, and a `@udf` outside a
trace runs its body locally.

## Relationship to the shipped SDK

The primitives are the **authoring front-end**; the shipped SDK is the
**execution back-end**. A traced node would dispatch by submitting through a
queue (`inv = f.submit(...)`, returning a durable string id), and because
the shipped `udf()` signature takes only `queue=` and `transport=`, the
`kind=` keyword is free to carry the semantic bucket without colliding with
anything.

Free, but not wired: no shipped decorator accepts `kind=` today, and no
object joins a traced `Graph` to its live invocations. That object is the
missing [run object](/python-sdk/architecture/f7-run-object.md).

## Open questions

These come from the draft's own README, surfaced by actually running the
trace:

- **Do ML UDFs default to `model` (purple) or stay `transform` (blue)?**
  Today `detect_objects` and the caption models trace as blue, because the
  contract examples do not tag them.
- **Is `merge` a model?** `merge_captions` is described as the quotient
  operation but traces blue; `kind="merge"` would fold it onto purple.
- **Is mask-indexing a node or an edge?** `labels[mask]` currently makes an
  explicit amber `select` node. Nodes are visible and colorable; folding the
  operation into a styled edge keeps the graph leaner. The same call applies
  to `>`, `&`, and `~`.
- **Namespace split.** The draft assumes `lakeshore` is compute and
  `dreamlake` is dataflow, matching the org repos. The shipped client is
  `dreamlake.lakeshore` — promote `lakeshore` to top level, or re-export?
- **`stack` axis semantics.** `stack([...]).mean(axis=0)` assumes `stack`
  adds a leading model axis; the shape types (`Mask[3, N]`) should make that
  explicit.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/queue

# Queue API

A queue is a named channel on a [dispatch plane](/python-sdk/dispatch.md).
`SyncQueue` is the blocking class and holds all the verbs; `Queue` is the
async sibling that wraps one and offloads its calls with
`asyncio.to_thread`.

> **Note:** `dls.Queue` is **async**; `dls.SyncQueue` is **blocking**. `Queue` is not
> a subclass of `SyncQueue` — it holds one and exposes it as `q.sync`.

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore import Queue, SyncQueue
```

```python
q = SyncQueue("gpu")                          # blocking surface
inv = q.submit({"x": 1}, key="scene7/x")      # -> durable string id
val = q.result(inv, timeout=60.0)

q = Queue("gpu")                              # async surface
inv = await q.submit({"x": 1})
val = await q.result(inv)
async for frame in q.stream(inv): ...
```

Payload values ride the wire codec (msgpack + ExtType), so ndarray /
tensor / image / tuple round-trip. File bytes never do — data UDFs
return [string keys](/python-sdk/udf.md) and files stay on the filesystem.

## Constructor

```python
SyncQueue(
    name: str,
    *,
    dispatch: Dispatch | None = None,   # defaults to the process plane
    kind: str = "fifo",                 # "fifo" | "priority" | "filo" | "boltzmann"
    temperature: float = 1.0,           # boltzmann sampling temperature
)
```

`Queue(name, **kw)` forwards the same arguments. `dispatch` selects the
plane: omit it for the process default (the local SQLite plane under
`~/.lakeshore`, or `HttpDispatch` when `LAKESHORE_URL` is set), pass
`Dispatch(":memory:")` for tests, or an explicit `HttpDispatch(url)` —
see [Dispatch planes](/python-sdk/dispatch.md).

### Kinds

| Kind | Claim order |
| --- | --- |
| `fifo` (default) | priority descending, then oldest first. `"priority"` is an accepted alias. |
| `filo` | priority descending, then newest first — autocomplete and live-preview workloads where only the latest request matters |
| `boltzmann` | sample proportional to `exp(priority / T)`; `T -> 0` collapses to strict priority, large `T` to uniform random |

Anything else raises
`DispatchError: unknown queue kind '...'; use fifo|filo|boltzmann`.
Ordering is what the **worker** asks for at claim time; submit only
attaches `priority`.

## Producer verbs

### `submit(payload=None, *, envelope=None, key=None, priority=0.0) -> str`

Enqueue any wire-native value, a [`Thunk`](/python-sdk/udf.md#dlsthunk--an-ad-hoc-callable),
or a pre-built envelope. Returns the **durable invocation id** — a plain
string, the whole handle.

```python
inv = q.submit({"seed": 42})
inv = q.submit({"seed": 42}, key="sweep/42")   # idempotent per queue
inv = q.submit({"seed": 42}, priority=9.0)
inv = q.submit(dls.thunk(lambda: 40 + 2))      # ad-hoc callable, by value
```

`key` makes the submit idempotent: the same key on the same queue
returns the existing invocation id, so resubmit-after-crash is safe by
construction.

> **Warning:** `SyncQueue.submit` takes `key=` / `priority=` / `envelope=`.
> `UDF.submit` takes `_key=` / `_priority=` / `_dispatch=` — underscored,
> because its other keyword arguments belong to your function.

### `result(inv_id, *, timeout=30.0) -> Any`

Block (or `await`) until the invocation is terminal, then return the
decoded result. Re-attachable from **any process** that holds the id.
Raises `DispatchError` if the invocation failed or was cancelled, and
`TimeoutError` when the window elapses.

### `call(payload, *, timeout=30.0, **kw) -> Any`

RPC in one line: `submit` then `result`, keyed by the invocation id — no
side-channel reply topic.

```python
answer = q.call({"prompt": "hello"}, timeout=10.0)
```

### `stream(inv_id, *, since=-1, timeout=30.0)`

Iterate an invocation's journal frames as decoded values, live while the
job runs. `since` is the reconnect cursor — pass the last seq you saw to
resume without loss or duplication. An `ERROR` frame raises
`DispatchError`; the `END` frame stops iteration cleanly; `timeout`
seconds of silence raises `TimeoutError`.

```python
for manifest in q.stream(inv):          # SyncQueue -> Iterator
    print(manifest)

async for manifest in q.stream(inv):    # Queue -> AsyncIterator
    print(manifest)
```

### `cancel(inv_id) -> bool` · `count(state="queued") -> int` · `len(q)`

`cancel` moves a **queued** invocation to `cancelled` and returns whether
it did; a running job finishes its attempt. `count` counts rows in one
state — `queued`, `running`, `succeeded`, `failed`, or `cancelled` — from
the same rows that get claimed, so counts cannot disagree with the
queue's contents. `len(q)` is `count()`.

> **Warning:** `HttpDispatch.cancel` and `HttpDispatch.count` raise
> `NotImplementedError` pending the control-plane routes. They work on the
> local `Dispatch` plane.

## Worker verbs

The consuming side of the same queue. Most workers just call
[`run_worker`](/python-sdk/dispatch.md#the-worker-loop) and never touch
these; they exist for custom loops.

### `claim() -> Job | None`

Claim one job, or `None` when the queue is empty. Atomic — exactly one
worker wins a given row.

```python
@dataclass
class Job:
    id: str          # the invocation id
    payload: Any     # the decoded payload / Envelope
    attempt: int
    queue: str
```

### `ack(job, result=None)` / `nack(job, *, error, retry=True)`

Terminal success / failure. Both are **fenced**: only the claiming
worker may settle the job, so a stale worker whose job was re-dispatched
cannot clobber the new attempt. Settling a job you do not hold raises
`DispatchError`.

### `working()` — the common loop, packaged

Claims one job, auto-acks on success, auto-nacks with `retry=True` on an
exception (and re-raises). Yields `None` when the queue is empty.

```python
with q.working() as job:
    if job is not None:
        handle(job.payload)
```

### `post(inv_id, value, *, final=False) -> int`

Worker-side streaming: append one VALUE frame (or, with `final=True`, the
terminal END frame) to an invocation's journal. Returns the assigned seq.
The [worker loop](/python-sdk/dispatch.md#the-worker-loop) does this for you
on every generator yield.

## `Topic` — pub/sub over the frame journal

Topics are journal channels: the same mechanism as invocation streaming,
cursor reconnect included.

```python
t = q.topic("progress")          # q.topic() auto-names it "topic-"

t.post({"step": 1})              # -> seq
t.post(None, final=True)         # END frame closes the topic

t.listen(since=-1, timeout=5.0)  # next value after `since`; None on timeout
for v in t.stream(since=-1, timeout=5.0):    # values until END
    ...
```

Note the shorter default: `Topic.listen` and `Topic.stream` time out
after 5 seconds of silence, where `q.stream` waits 30.

The journal is derived data — truth is the invocation row. Trimming a
journal degrades streaming reads to the terminal result, never to wrong
state.

## `Queue` vs `SyncQueue`

| | `SyncQueue` | `Queue` |
| --- | --- | --- |
| style | blocking | `await` / `async for` |
| producer | `submit` `result` `call` `stream` `cancel` `count` `__len__` | `submit` `result` `call` `stream` `cancel` `count` |
| worker | `claim` `ack` `nack` `working` `post` | `claim` `ack` `nack` |
| `topic()` | sync, returns `Topic` | sync, returns the same `Topic` |
| use from | scripts, workers, tests | asyncio apps, notebooks with a loop |

`Queue` offloads blocking calls so slow polls never stall the event
loop; `q.sync` exposes the blocking sibling sharing the same binding —
that is where `working()`, `post()`, and `len()` live.

Fan-out composes with stdlib asyncio; there is no bespoke gather:

```python
ids = [await q.submit({"seed": i}) for i in range(100)]
results = await asyncio.gather(*(q.result(i, timeout=300) for i in ids))
```

## Read next

- [Invocation ids](/python-sdk/invocations.md) — the no-Future model and
  the fan-out + collect patterns.
- [Dispatch planes](/python-sdk/dispatch.md) — `Dispatch`, `HttpDispatch`,
  and the worker loop.
- [`@udf` decorator](/python-sdk/udf.md) — functions bound to a queue.

---

Source: https://lakeshore.dreamlake.ai/cli/completion

# Shell completion

`lakeshore completion <shell>` prints a sourceable completion script.
`bash`, `zsh`, and `fish` are accepted (case-insensitively); anything
else exits 2.

## Install

Pick your shell and drop the eval into your rc file once:

```bash
# bash
echo 'eval "$(lakeshore completion bash)"' >> ~/.bashrc

# zsh
echo 'eval "$(lakeshore completion zsh)"'  >> ~/.zshrc

# fish
lakeshore completion fish | source                                       # this shell only
lakeshore completion fish > ~/.config/fish/completions/lakeshore.fish    # persistent
```

Restart the shell (or `source` the rc file).

## What the tree covers

The completion script is **generated from a hand-maintained table**, not
introspected from Commander at runtime. Today that table covers these
top-level groups:

```text
providers  secrets  modes  auth  config  daemon  worker
jobs  tunnels  mounts  ssh  aws  gcp  slurm  kube  docker  completion
```

> **Warning:** Groups that exist on the CLI but are **missing** from the completion
> tree: `storage`, `code`, `queues`, `exec`, `run`, `dreamlake`, `nymph`,
> `admin`, `tui`, and the compose verbs `up` / `down` / `ps` / `status` /
> `logs`. They work fine when typed in full — they just do not tab-expand.
> 
> The tree also contains two entries you should ignore: `--version`,
> which is not a real flag on this program, and `daemon ota`, which is a
> hidden back-compat alias for `daemon update`.

Subcommands that do complete, per group:

```text
lakeshore providers   → test instances add list show remove update terminate edit discover
lakeshore secrets     → add list show rotate remove
lakeshore modes       → add list show remove update
lakeshore auth        → login status logout
lakeshore config      → show refresh source
lakeshore daemon      → stop status list show launch install kill cleanup
                             hibernate ota target-version set-target-version
lakeshore worker      → start once
lakeshore jobs        → list show kill
lakeshore tunnels     → add list show remove
lakeshore mounts      → add list show remove update
lakeshore ssh         → discover upload probe status
lakeshore aws|gcp|slurm|kube|docker   → discover
lakeshore completion  → bash zsh fish
```

Long flag names complete per (group, subcommand) pair — only long
flags, since short-flag completion is noise.

## Dynamic values

Some argument and flag positions complete against real names:

```text
lakeshore providers show                        → live provider names
lakeshore secrets rotate                        → live secret names
lakeshore modes update                          → live mode names
lakeshore tunnels remove                        → live tunnel names
lakeshore mounts show                           → live mount names
lakeshore daemon show|kill|hibernate            → live worker ids
lakeshore jobs show|kill                        → live invocation ids
lakeshore daemon install                        → SSH aliases from ~/.ssh/config
lakeshore ssh upload|probe                      → SSH aliases from ~/.ssh/config
lakeshore providers add --from-aws-profile      → AWS profile names
lakeshore providers add --from-gcloud-config    → gcloud configurations
lakeshore providers update foo --tunnel         → live tunnel names
lakeshore jobs list --state                     → queued running succeeded failed killed timeout
```

The generated scripts get these by shelling out to a hidden subcommand:

```bash
lakeshore completion __suggest <kind>
```

`__suggest` is an implementation detail, not a user-facing command.
Valid kinds are `providers`, `secrets`, `modes`, `tunnels`, `mounts`,
`daemons`, `invocations`, `ssh-aliases`, `aws-profiles`,
`gcloud-configs`, `kube-contexts`, `docker-contexts`, and `job-states`.

Server-backed kinds do one `GET` against the control plane and cache
the result for **30 seconds** at
`$XDG_CACHE_HOME/lakeshore/completion-<kind>.txt`, one name per line,
so repeated tabs don't hammer the server. The invocations list is
capped at 50 entries.

Local kinds (`ssh-aliases`, `aws-profiles`, `gcloud-configs`,
`kube-contexts`, `docker-contexts`) skip the cache and read your config
files directly. `job-states` is a hardcoded static list.

If the control plane is unreachable or auth isn't set up, the
suggestion path returns nothing and tab simply does nothing — your
shell stays responsive.

## Forcing a cache refresh

The 30-second window is usually invisible. If you just registered a
provider and want it to tab-complete right now:

```bash
rm -f "${XDG_CACHE_HOME:-$HOME/.cache}/lakeshore/completion-providers.txt"
```

…or wait 30 seconds.

## Extending it

The subcommand tree, the per-command flag lists, and the
argument→suggestion mapping all live in `src/cli/completion/index.ts` in
the `lakeshore` repo. The shell scripts are generated from those
tables, so a new subcommand or flag has to be added there before it
tab-completes.

## Read next

- [`lakeshore` CLI](/cli.md) — the full command surface, including the
  groups the completion tree doesn't yet know about.
- [CLI examples](/cli/examples.md) — the runnable cookbook.

---

Source: https://lakeshore.dreamlake.ai/cli/examples/daemons-and-exec

# CLI examples — Daemons, exec, and run

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives
on [Auth + setup](/cli/examples/auth.md).

## Daemons

A **daemon** (nymph) is a worker process on a host, managed through the
control plane. Two ways to bring one up:

1. `daemon launch` — the control plane provisions a fresh cloud host
   via a registered provider.
2. `daemon install` — you already have SSH to a host; the CLI
   bootstraps nymph on it.

> **Note:** `lakeshore daemon ...` always goes through the control plane. To run a
> native Python queue worker on your own box, use `lakeshore worker
> start` — covered in its own section below. `daemon start --local` was
> removed.

### `daemon launch` — provision a host via a provider

`--name` sets both the worker label and the cloud Name tag (visible in
the EC2 / GCE console). Omit it and the label falls back to
`<provider>-<short-ulid>`.

```bash
lakeshore daemon launch --provider my-ec2 --name ge-debug-1 --wait
```

`daemon launch` reads project defaults from `.lakeshore` and
`.lakeshore.local`, walking up from cwd and stopping at the git root.
`--no-config` skips that discovery entirely.

### `daemon launch --count` — a fleet

Launches N workers in parallel. With `--count > 1`, `--name` becomes a
prefix: `fleet-00`, `fleet-01`, … `--concurrency` bounds the in-flight
launches (default 5, sized to the AWS `RunInstances` rate limit) and
retries on throttle. `--wait` waits for **all** of them.

```bash
lakeshore daemon launch --provider my-ec2 --name fleet --count 4 --wait
lakeshore daemon launch --provider my-ec2 --name fleet --count 20 --concurrency 8 --wait
```

### `daemon launch` — capacity, tags, runners

`--tag`, `--queue`, and `--runner` are each repeatable, and each one
**replaces** the corresponding list from `.lakeshore` rather than
appending to it. The first `--runner` becomes the default runner.

```bash
lakeshore daemon launch \
  --provider my-ec2 \
  --name h100-1 \
  --tag gpu:h100 \
  --tag region:us-east \
  --runner process \
  --capacity '{"cpu":8,"gpu":1}' \
  --keep-alive 600 \
  --wait
```

`--keep-alive <seconds>` is the idle policy: `0` = exit after one job,
`N` = exit after N idle seconds, `-1` = pool mode (never idle-exit).

### `daemon launch --queue` — bind to a queue

Daemons subscribe to queues by name via `Worker.queues: string[]`. Pass
`--queue` one or more times at launch; omit it entirely and the daemon
lands in the default queue (the empty array).

```bash
lakeshore daemon launch --provider my-ec2 --queue training-h100 --name train-01 --wait

lakeshore daemon launch --provider my-ec2 \
  --queue training-h100 --queue eval \
  --name multi-01 --wait
```

### `daemon launch --bootstrap-script` / `--setup-script`

`--bootstrap-script` runs as root **before** nymph starts (cloud-init).
`--setup-script` is repeatable and queues post-register setup commands,
each with `refresh_capabilities` set; it appends to the
`setup_scripts` list from `.lakeshore`. `--no-setup` clears both
sources.

```bash
cat > /tmp/bootstrap.sh <<'SH'
#!/usr/bin/env bash
set -eux
apt-get update && apt-get install -y htop
SH
lakeshore daemon launch --provider my-ec2 --name ge-debug-1 \
  --bootstrap-script /tmp/bootstrap.sh --wait
```

Both flags read the file locally and send its content inline in the
launch payload.

### `daemon install` — bootstrap on a host you can SSH to

Takes an SSH alias. `--label` defaults to the alias. `--tag` and
`--runner` are repeatable (`--runner` defaults to `process`).
`--keep-alive-s` defaults to `-1` (pool mode). The admin token defaults
to `$LAKESHORE_ADMIN_TOKEN`.

```bash
lakeshore daemon install my-server --label ge-debug-1
lakeshore daemon install my-server --tag gpu:a100 --runner docker --keep-alive-s 900
```

> **Warning:** `daemon install` waits for the Worker row by default and exposes only
> the negation `--no-wait`. There is no positive `--wait` on `install`
> (there is on `launch`).

### `daemon install --binary` — rsync a local build

Skip the download and rsync a locally-built nymph binary onto the host.
Match the host arch. `--nymph-base-url` overrides the download source
when you're not using `--binary`.

```bash
cargo build --release --target x86_64-unknown-linux-gnu --manifest-path nymph/Cargo.toml
lakeshore daemon install my-server \
  --label ge-debug-1 \
  --binary nymph/target/x86_64-unknown-linux-gnu/release/nymph
```

### `daemon install --slurm` — sbatch instead of setsid+nohup

Wrap the nymph startup in an `sbatch` heredoc so the daemon runs as a
SLURM job. Use on login nodes where a detached background process is
discouraged.

```bash
lakeshore daemon install login.cluster --slurm --slurm-partition gpu
```

### `daemon list` — what's registered

Every daemon in the namespace, with last-seen time, state, and a
`queues` column (`—` when empty, i.e. the default queue).

```bash
lakeshore daemon list
lakeshore daemon list --json | jq '.[] | {id, label, state, lastSeenAt}'
```

`--watch` takes an **optional** value; bare `--watch` refreshes every 2
seconds. Ctrl-C exits.

```bash
lakeshore daemon list --watch        # every 2s
lakeshore daemon list --watch 5      # every 5s
```

### `daemon show` — one daemon, full record

Same `--json` / `--watch [seconds]` treatment.

```bash
lakeshore daemon show "$D"
lakeshore daemon show "$D" --json | jq .
lakeshore daemon show "$D" --watch
```

### `daemon launch-log` — cloud serial console

When a launched daemon never reaches its first `/hello`, grab the
cloud serial-console output. EC2-only today.

```bash
lakeshore daemon launch-log "$D" --tail 200
lakeshore daemon launch-log "$D" --no-latest --json
```

### `daemon setup` — queue a post-register script

Upload a bash script and run it on a daemon that has already
registered. `--script` is required; the content is uploaded inline.
`--sudo` runs it as root, and `--refresh-capabilities` makes the daemon
re-detect its host capabilities afterwards and publish them on the next
poll.

```bash
cat > /tmp/post.sh <<'SH'
#!/usr/bin/env bash
echo "running on $(hostname)"
nvidia-smi || true
SH
lakeshore daemon setup "$D" --script /tmp/post.sh --name post-install --refresh-capabilities

cat > /tmp/apt-install.sh <<'SH'
#!/usr/bin/env bash
apt-get update && apt-get install -y htop
SH
lakeshore daemon setup "$D" --script /tmp/apt-install.sh --sudo --name apt-htop
```

Setup commands run **serially**, in submission order, on the daemon.

### `daemon hibernate` — pause polling

Tell a daemon to stop polling until a deadline. The process stays alive
but goes idle. `--until` is required and accepts a relative offset
(`+30s`, `+5m`, `+1h`, `+3d`) or an ISO-8601 timestamp.

```bash
lakeshore daemon hibernate "$D" --until +1h
lakeshore daemon hibernate "$D" --until 2026-06-01T00:00:00Z
```

### `daemon reset` — clear poll backoff

Queues a `reset_backoff` command so a daemon stuck in exponential
backoff drops back to `retry_min_s` on its next poll.

```bash
lakeshore daemon reset "$D"
```

### `daemon update` — OTA and queue membership

Rolls a daemon to a nymph version. Omit `--version` and it uses the
namespace's target version.

```bash
lakeshore daemon update "$D" --version 0.1.4
lakeshore daemon update --all --version 0.1.4 --yes
lakeshore daemon update --prefix fleet- --version 0.1.4 --yes
```

`--url` and `--sha256` roll a custom binary — pair with
`lakeshore nymph push`:

```bash
PUSH=$(lakeshore nymph push ./nymph --json)
# `.url` in the envelope is server-relative; daemons need the
# fully-qualified address, which the human output prints as `full:`.
lakeshore daemon update --all \
  --url "$LAKESHORE_URL$(jq -r '.url' <<<"$PUSH")" \
  --sha256 "$(jq -r '.sha256' <<<"$PUSH")" \
  --yes
```

The same verb edits `Worker.queues`. `--add-queue` / `--remove-queue`
are repeatable, need a positional `<id>`, do **not** require
`--version`, and skip the OTA path entirely.

```bash
lakeshore daemon update train-01 --add-queue eval
lakeshore daemon update train-01 \
  --add-queue training-h100 \
  --add-queue cpu-prep \
  --remove-queue scratch
```

> **Warning:** `daemon ota` still works as a hidden back-compat alias for
> `daemon update`, but it requires `--version` and carries no help text.
> Use `daemon update`.

### `daemon target-version` / `set-target-version`

The namespace-wide auto-update target. Daemons whose config sets
`runtime.auto_update = true` converge to it on their own.

```bash
lakeshore daemon target-version                   # show
lakeshore daemon set-target-version 0.1.4         # set
lakeshore daemon set-target-version --clear       # unset
lakeshore daemon target-version --json
```

### `daemon kill` — drop a daemon

Hard-deletes the Worker row. **The cloud instance is terminated by
default**; `--no-terminate` keeps the box and just forgets about it.

```bash
lakeshore daemon kill "$D"
lakeshore daemon kill "$D" --no-terminate
lakeshore daemon kill --prefix fleet- --yes        # bulk by label prefix
```

`--prefix` is mutually exclusive with the `<id>` positional.

### `daemon cleanup` — drop stale daemons

Bulk-delete daemons that haven't polled recently. `--older-than`
defaults to `24h` and accepts `30m`, `2h`, `5d`, or a bare number of
hours.

```bash
lakeshore daemon cleanup --older-than 5m --dry-run
lakeshore daemon cleanup --older-than 5m --yes
lakeshore daemon cleanup --older-than 24h --yes --json
```

### `daemon status` / `daemon stop` — the local daemon record

These two act on the **local** daemon state file only
(`~/.config/dreamlake/daemons/local.json`, or `$DREAMLAKE_STATE_DIR`),
not on the fleet.

`stop` sends SIGTERM and waits up to `--timeout` seconds (default 30).
`--force` sends SIGKILL instead.

```bash
lakeshore daemon status
lakeshore daemon stop                # SIGTERM, wait 30s
lakeshore daemon stop --timeout 5
lakeshore daemon stop --force        # SIGKILL
```

## `worker` — a native worker on this machine

`worker start` supervises a native Python queue worker in the
foreground; `worker once` drains a single job and exits. Both take the
identical flag set.

```bash
lakeshore worker start --queue default
lakeshore worker start --queue default --url http://localhost:8080 --token "$TOKEN"
lakeshore worker once --queue default
lakeshore worker start --queue default --dry-run    # print the child argv and exit
```

Under the hood the CLI spawns `python -u -m dreamlake.lakeshore.daemon`.
Flags you omit are deliberately **not** placed on the child's argv, so
the Python module's own `LAKESHORE_*` env fallbacks stay authoritative:

| Flag                | Env fallback              | Default                                            |
| ------------------- | ------------------------- | -------------------------------------------------- |
| `--queue`           | `LAKESHORE_QUEUE`         | `default`                                          |
| `--root`            | `LAKESHORE_ROOT`          | cwd                                                |
| `--url`             | `LAKESHORE_URL`           | unset → a local SQLite plane under `LAKESHORE_HOME` (`~/.lakeshore`) |
| `--namespace`       | `LAKESHORE_NAMESPACE`     | —                                                  |
| `--token`           | `LAKESHORE_CLIENT_TOKEN`  | —                                                  |
| `--runtime-policy`  | `LAKESHORE_RUNTIME_POLICY`| `strict` \| `minor` \| `off`                       |
| `--python`          | `LAKESHORE_PYTHON`        | `python3`                                          |

Plus `--module`, `--env K=V` (repeatable), `--allow-pickled`, and
`--kill-timeout <seconds>` (default 15). The token is passed to the
child through the environment, never on argv.

> **Warning:** `lakeshore worker` deliberately does **not** consult
> `~/.config/lakeshore/auth.yml`. The plane a worker drains has to come
> from `--url` / `LAKESHORE_URL` and `--token` / `LAKESHORE_CLIENT_TOKEN`.
> Running `lakeshore auth login` is not sufficient setup for a worker.

## Compose — bring up a whole group

The five compose verbs are **top-level** commands that all read a
single `lakeshore.yaml` from the current directory (`-f` to point
elsewhere; there is no upward walk in v1).

```bash
lakeshore up --plan                 # show the plan, no side effects
lakeshore up                        # reconcile
lakeshore up gpu --wait             # only the `gpu` daemon group, wait for active
lakeshore ps                        # daemons owned by this file
lakeshore status                    # current vs desired drift (read-only)
lakeshore logs gpu --follow         # multiplexed console output; --poll-ms defaults to 5000
lakeshore down                      # drain (default)
lakeshore down --terminate --prune-queues
```

`--json` is available on `up`, `down`, `ps`, and `status`. See
[Compose](/get-started/compose.md) for the file schema.

## Exec and run

Two surfaces for one-off work. `exec` runs bash; `run` ships a Python
script.

Both use Commander's `passThroughOptions()`: **CLI flags come first,
everything after the daemon argument is the remote command, and there
is no `--` separator.**

### `exec` — top-level, with a picker

```bash
lakeshore exec nvidia-smi                          # interactive daemon picker
lakeshore exec --queue training-h100 nvidia-smi    # skip the picker; the CP routes to a queue member
```

### `daemon exec` — an explicit daemon

`<daemon>` accepts a worker id, a label, or a `machine_id`.

```bash
lakeshore daemon exec "$D" nvidia-smi
lakeshore daemon exec "$D" python -c 'import torch; print(torch.cuda.is_available())'
lakeshore daemon exec --id "$D" hostname     # treat the arg as a literal worker id
```

### `daemon exec` — env, stdin, workdir

`--env` is repeatable and merges on top of the daemon's own
environment. `--stdin` takes a path, or `-` to forward this CLI's
stdin.

```bash
lakeshore daemon exec \
  --env CUDA_VISIBLE_DEVICES=0 \
  --env BATCH=64 \
  "$D" python train.py

lakeshore daemon exec --stdin input.json "$D" jq .results
echo '{"a":1}' | lakeshore daemon exec --stdin - "$D" jq .a
lakeshore daemon exec --workdir /tmp "$D" pwd
```

The remote command runs under `bash -c` — **not** a login shell, so
`/etc/profile.d` and `~/.bashrc` are not sourced.

### `daemon exec` — timeouts

`--timeout` kills the **remote** process (SIGTERM, then SIGKILL five
seconds later; the wire reports exit code `-2` for a timeout and `-1`
for a spawn failure or cancellation). `--wait` is how long the **CLI**
long-polls for the result, default 600 seconds. They are independent.

```bash
lakeshore daemon exec --timeout 30 "$D" sleep 120
lakeshore daemon exec --wait 1800 --timeout 1500 "$D" python long_train.py
```

### `daemon exec --json` — the structured envelope

The envelope carries stdout and stderr base64-encoded as `stdout_b64` /
`stderr_b64`.

```bash
lakeshore daemon exec --json "$D" uname -a \
  | jq '{exit_code, stdout: (.stdout_b64 | @base64d)}'
```

### `run <script.py>` — ship a Python script

Reads the local file and pipes its bytes as stdin to a remote
`python3 -` over the exec channel. Nothing is written to disk on the
daemon. A non-`.py` extension only warns.

```bash
cat > /tmp/hello.py <<'PY'
import platform, sys
print("hello from", platform.node(), "python", sys.version_info[:3])
PY
lakeshore run /tmp/hello.py --daemon "$D"
lakeshore run /tmp/hello.py                  # omit --daemon for the picker
```

```bash
lakeshore run /tmp/hello.py \
  --daemon "$D" \
  --env DEBUG=1 \
  --timeout 60 \
  --python /opt/conda/bin/python
```

> **Note:** `run` accepts only `--daemon`, `--env`, `--timeout`, `--python`, and
> `--json`. There is no `--workdir` and no `--stdin` — the script body
> *is* the stdin. Arguments after `<script>` are currently ignored. Use
> `daemon exec` when you need those knobs.

## Read next

- [Daemon launch](/nymph/daemons.md) — the provider-driven path in detail.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — install → list → kill → cleanup.
- [Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md)
- [Daemon protocol](/nymph/protocol.md) — the exec / poll / ack wire format.
- [Compose](/get-started/compose.md) — the `lakeshore.yaml` schema.

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths/04-smoke-exec-cascade

# 04 · Smoke-exec cascade

`scripts/smoke-exec.sh` is a cascading shell script that exercises every
primitive `lakeshore daemon exec` supports. Cheap, comprehensive, and
unforgiving — it bails on the first failure so you fix one thing at a
time.

## Exercises

In order, one check per aspect of the exec wire:

1. Exactly one live daemon is registered (or `$LAKESHORE_DAEMON` names one).
2. Minimal stdout round-trip — `echo hi`.
3. stderr stays separated from stdout.
4. Exit code propagation — `exit 7` gives the CLI rc 7.
5. `--env` passthrough.
6. `--stdin -` piping.
7. Filesystem write/read on the daemon.
8. `--timeout` enforcement — the wire's `-2`, which bash reports as 254.
9. Long-poll spanning multiple poll windows (`sleep 40`).
10. `python3` on the host.

## Requires

- A control plane reachable at `$LAKESHORE_URL` (or a saved
  `lakeshore auth login` when it's unset).
- Exactly **one** live daemon registered in the namespace. The script
  first runs `daemon cleanup --older-than 5m --yes` to reap stale rows,
  then refuses to guess on zero or multiple.

Set `LAKESHORE_DAEMON=<id-or-label>` to skip that discovery entirely
and target a specific daemon.

## Run

```bash
# Sibling checkouts live in the container repo:
#   git clone --recurse-submodules https://github.com/dreamlake-ai/lakeshore-workspace.git
export LAKESHORE_WORKSPACE=~/lakeshore-workspace

cd "$LAKESHORE_WORKSPACE"
LAKESHORE_URL=http://localhost:8080 ./scripts/smoke-exec.sh
# or press `s` on the `smoke` pane in mprocs
```

The script prefers the local build at `lakeshore/dist/cli/index.js` so
you test what's under iteration. Set `LK=lakeshore` to run the
globally-installed npm package instead.

```bash
LK=lakeshore ./scripts/smoke-exec.sh
LAKESHORE_DAEMON=ge-debug-1 ./scripts/smoke-exec.sh
```

## Expected output

A green `✓` and a label per check, ending in
`✓ all N checks passed against <url>`. Any red `✗` stops the cascade
and the script exits 1.

## If it fails

| Check | Symptom | Likely cause |
| ---- | ------- | ------------ |
| 1 | `no live daemons registered` | The nymph pane stopped — restart it, or set `LAKESHORE_DAEMON`. |
| 1 | `N live daemons — refuse to guess` | An earlier run left daemons registered. Kill the extras (`lakeshore daemon kill --prefix ... --yes`) or set `LAKESHORE_DAEMON`. |
| 1 | `daemon list reachable` failed | The control plane is down, or auth isn't configured. |
| 4 | wrong exit code | Exit-code propagation broken between the daemon, the control plane, and the CLI. |
| 8 | `expected 254 or -2, got …` | Timeout handling. The daemon either ran past the deadline or surfaced its own error code instead of the wire's `-2`. |
| 9 | long-poll check failed | The CLI's poll window or the control plane's long-poll budget regressed. |

## Status

Manual. Tracked via the script's exit code.

## Next

→ [05 · Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md)

---

Source: https://lakeshore.dreamlake.ai/get-started/jobs

# Jobs

Every dispatched function call lands as an **Invocation** row in the
control plane. `lakeshore jobs …` is the surface for seeing what is
queued, what is running, and for cancelling what you no longer want.

Every verb takes `--json`, so the output pipes into `jq` or into other
tooling; the default is a human-readable table.

## The states

```
queued ──▶ running ──▶ succeeded
                   ├─▶ failed
                   ├─▶ killed
                   └─▶ timeout
```

`succeeded`, `failed`, `killed`, and `timeout` are the terminal set.
The id is a **ULID generated by the producer** — the same string that
crosses the wire, not a server-assigned ObjectId.

## CLI

```bash
lakeshore jobs list                                  # latest 50, all states
lakeshore jobs list --state queued                   # only queued
lakeshore jobs list --state queued --state running   # union
lakeshore jobs list --worker <worker-id> --limit 10
lakeshore jobs list --json | jq '.invocations[].id'

lakeshore jobs show <id>
lakeshore jobs show <id> --json

lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue training-h100
```

Filters on `jobs list`, all optional:

| Flag | Meaning |
| --- | --- |
| `--state` | One of `queued`, `running`, `succeeded`, `failed`, `killed`, `timeout`. Repeatable; multiple values become an `in` filter. |
| `--worker` | Only invocations claimed by this worker id. |
| `--queue` | Only invocations on this queue, matched against `runConfig.queue`. |
| `--limit` | 1–200, default 50. |
| `--offset` | Rows to skip, default 0. |

Rows come back newest-submitted first.

## Cancelling

The kill verb is deliberately narrow:

- **`queued`** → the row flips to `state=killed` with `finishedAt=now`.
  Safe: no daemon has claimed it, and the dispatch claim filters on
  `state="queued"`, so future polls skip it.
- **`running`** → **409**, with the claiming daemon's id in the response
  body. There is no daemon-side cancel signal in the wire protocol yet.
  The escape hatch is to stop the daemon: `SIGTERM` fires the
  per-invocation cancellation tokens the drain loop already plumbs
  through, and the in-flight invocations get acked as `killed` once
  `shutdown.grace_s` expires.
- **Already terminal** → 409 with the row's current state.

`jobs bulk-cancel` applies the same narrow rule in bulk: it flips every
`state="queued"` row to `killed` and returns `{ "cancelled": <n> }`.
Running invocations are untouched. Scope it with `--queue <name>`.

Per-invocation cancel for running work is on the follow-up list.

## HTTP

The CLI is a thin wrapper over JSON endpoints — call them directly when
the CLI shape does not fit:

```
GET    /v1/namespaces/:ns/invocations[?state=…&worker=…&queue=…&limit=…&offset=…]
GET    /v1/namespaces/:ns/invocations/:id
DELETE /v1/namespaces/:ns/invocations/:id
POST   /v1/namespaces/:ns/invocations/bulk-cancel[?queue=…]
```

The list endpoint returns `{ invocations: [...], limit, offset, count }`.
`show` returns one invocation. `DELETE` returns the updated row on
success, or a 4xx body explaining why the kill was refused. An unknown
namespace is a 404 on all four.

Operators with an admin token also get a long-polling firehose:

```
GET /v1/admin/invocations[?cursor=…&timeout_s=…&limit=…]
GET /v1/admin/invocations/:id
```

`timeout_s` caps at 60 and `limit` defaults to 100 (max 500).

## The row shape

| Field | Meaning |
| --- | --- |
| `id` | Producer-generated ULID. |
| `state` | One of the six states above. |
| `workerId` | Claiming worker, `null` until claimed. |
| `queueName` | Read out of `runConfig.queue`. There is no `queueId` column — Phase A.5 removed the FK. |
| `functionId` | The `Function` row this invocation calls. |
| `priority`, `attempt` | Dispatch priority and retry count. |
| `runConfig` | The full submitted run config, including `queue`. |
| `progress`, `error` | Whatever the daemon last reported. |
| `submittedAt` / `startedAt` / `finishedAt` | ISO timestamps; the last two are nullable. |
| `parentInvocationId`, `rootInvocationId`, `correlationId` | Pipeline lineage. `rootInvocationId` equals the row's own id when it is the root. |

`jobs list` renders a subset — id, state, worker, queue, attempt, and
relative ages for submitted/started. `jobs show` prints the whole record,
and `--json` gives it verbatim on both.

## Streaming a result

Results that arrive as a journal rather than one blob are read from the
producer surface, not from `jobs`:

```
GET /v1/producer/invocations/:id/result/stream?since=<seq>
```

It replays the durable `ResultChunk` rows past `since`, then follows the
live Redis journal. The response is raw chunked binary with content-type
`application/x-msgpack-stream` — self-delimiting msgpack frames with no
SSE wrapper and no delimiter between them. From Python, use
`SyncQueue.stream(inv)` / `Queue.stream(inv)` instead of parsing it by
hand.

## Tab-completion

Once shell completion is installed (see [Completion](/cli/completion.md)):

```text
lakeshore jobs                   → list show kill
lakeshore jobs list --state      → queued running succeeded failed killed timeout
lakeshore jobs show              → live invocation ids (30 s cache, 50 max)
```

`bulk-cancel` is a real subcommand but is missing from the hand-maintained
completion tree, so it does not tab-expand — type it in full.

## Read next

- [Queues](/get-started/queues.md) — where these invocations are waiting.
- [Invocation ids](/python-sdk/invocations.md) — the Python side of the same
  handle.
- [Architecture](/get-started/architecture.md) — the Invocation model in
  context.

---

Source: https://lakeshore.dreamlake.ai/get-started/tutorials/end-to-end

# End-to-end smoke

A walkthrough of every operator-facing surface. Each section is
self-contained — run them in any order. Good as a release smoke test
(mark each section ✓ / ✗ in your release notes) and as a guided tour.

## Nomenclature

| Term | What it means |
| --- | --- |
| Provider | A recipe for reaching compute — an SSH config, an AWS region + credentials, a Slurm cluster. |
| Instance | One live cloud resource (EC2 VM, GCE VM, Kube pod) launched from a Provider. |
| Daemon | A `nymph` process registered with the control plane. One Worker row. |
| Mode | A named RunConfig — runner, image, env, resources. |
| Queue | A named pipe of pending Invocations that daemons subscribe to. |
| Invocation | One unit of work dispatched to a daemon. |
| ExecJob | An ad-hoc shell command run on a daemon, outside the Invocation path. |

## 1. Provider lifecycle

```bash
lakeshore providers add my-test --launcher SSH \
  --kwarg host=my-laptop.local --kwarg user=$USER
lakeshore providers list                       # appears
lakeshore providers show my-test               # full config
lakeshore providers update my-test --kwarg port=2222
lakeshore providers show my-test               # port=2222 reflected
lakeshore providers remove my-test             # soft delete
```

**Expect:** every step exits 0; `show` reflects the latest state.

The name is a **positional**, and the flag is `--launcher` — one of
`SSH`, `SLURM`, `EC2`, `GCE`, `Kube`, capitalized exactly like that.
There is no `--name` and no `--type`. `--launcher` is immutable:
supplying it to `providers update` is an error.

## 2. Soft delete, restore, and hide

```bash
lakeshore providers remove my-test         # soft delete
lakeshore providers list                   # not shown
lakeshore providers list --deleted         # shown with [D <ts>]
lakeshore providers list --all             # hidden + deleted

# Restore over HTTP
curl -X POST -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test/restore"

# Hide / unhide over HTTP
curl -X PATCH -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' -d '{"hidden":true}' \
  "$LAKESHORE_URL/v1/namespaces/$NS/providers/my-test"
lakeshore providers list --hidden

# Or do all of it interactively
lakeshore providers edit
```

**Expect:** a soft delete renames the row with a ` [deleted <iso>]`
suffix so the `(namespace, name)` unique index stays intact; `restore`
matches both the plain and the tombstoned name. Hidden rows survive but
stay out of default listings. `providers edit` refetches between actions
so the visible state always reflects what just happened.

## 3. Cloud-config auto-detect

Pre: `~/.aws/credentials` and/or a gcloud configuration.

```bash
lakeshore aws discover             # profiles from ~/.aws
lakeshore aws discover --json
lakeshore gcp discover             # gcloud configurations
lakeshore slurm discover           # /etc/slurm/slurm.conf + sinfo
lakeshore kube discover            # kubectl config view
lakeshore docker discover          # docker context ls

lakeshore providers add aws-dev --from-aws-profile sandbox
lakeshore providers add gcp-dev --from-gcloud-config research
lakeshore providers show aws-dev
```

**Expect:** every `discover` is read-only and makes no cloud API calls —
`--json` is their only flag. `--from-aws-profile` implies
`--launcher EC2` and `--from-gcloud-config` implies `--launcher GCE`, so
you do not pass `--launcher` alongside them.

## 4. SSH discover and the interactive picker

Pre: at least two entries in `~/.ssh/config`.

```bash
lakeshore ssh discover                         # table + footer legend
lakeshore ssh discover --filter new            # only entries not on the server
lakeshore ssh discover --show-stripped         # per-entry stripped directives
lakeshore ssh upload                           # interactive multi-select
lakeshore ssh upload my-dev-box --interactive  # picker even with an alias
lakeshore ssh upload --all                     # non-interactive, everything
lakeshore ssh probe my-dev-box --timeout 5
lakeshore ssh status --history 3
```

**Expect:** `--filter` takes `all` | `new` | `existing` and exits 2 on
anything else. Local-only directives are stripped silently, and the
picker groups rows into "new" and "existing".

## 5. Daemon install over SSH

Pre: an SSH alias you can reach, and `LAKESHORE_ADMIN_TOKEN` exported.

```bash
lakeshore daemon install <ssh-alias> --label <name>
lakeshore daemon list                    # <name> appears, state=active
lakeshore daemon show <id>               # version, runners, capabilities
lakeshore daemon list --watch            # live view, 2 s interval
```

**Expect:** the CLI SSHes in, drops the binary and config, and waits for
the Worker row to register. `--no-wait` skips the wait — there is no
positive `--wait` on `install`. `--binary <path>` rsyncs a local nymph
instead of downloading; `--slurm` wraps the launch in an `sbatch`
heredoc.

Note `--keep-alive-s <n>` here (`-1` = pool, `0` = single job, `N` =
seconds), which is spelled `--keep-alive <seconds>` on `daemon launch`
and `keep_alive_s` in YAML.

## 6. Cloud launch

```bash
lakeshore daemon launch --provider my-ec2 --name gpu --count 2 \
  --queue training --runner docker --wait
lakeshore daemon launch-log <id> --tail 50
lakeshore providers instances my-ec2
lakeshore providers instances --all
lakeshore providers terminate my-ec2 i-0abc...
```

**Expect:** `--name gpu --count 2` produces labels `gpu-00`, `gpu-01` and
sets the cloud Name tag to match. `launch-log` fetches the cloud serial
console (EC2 only today). `daemon launch` reads project defaults from
`.lakeshore` / `.lakeshore.local` — pass `--no-config` to ignore them.

## 7. Kill, cleanup, hibernate, reset

```bash
lakeshore daemon kill <id>                        # also terminates the instance
lakeshore daemon kill <id> --no-terminate         # keep the instance
lakeshore daemon kill --prefix gpu- --yes         # bulk by label prefix

lakeshore daemon cleanup --older-than 2h --dry-run
lakeshore daemon cleanup --older-than 2h
lakeshore daemon cleanup --older-than 5d --yes

lakeshore daemon hibernate <id> --until +30m
lakeshore daemon reset <id>
```

**Expect:** `kill` **terminates the cloud instance by default** —
`--no-terminate` is the opt-out. `cleanup` defaults to `24h` and caps at
`168h` (one week); values outside that range are a 400. `--until`
accepts a relative duration (`+30s`, `+5m`, `+1h`, `+3d`) or an ISO-8601
timestamp; a past deadline is a 400. `reset` queues a `reset_backoff`
command so a daemon stuck in exponential backoff retries promptly.

## 8. Queues

```bash
lakeshore queues add smoke --kind fifo --elasticity fixed
lakeshore queues ls
lakeshore queues show smoke
lakeshore queues patch smoke --description "release smoke"
lakeshore queues stats smoke
lakeshore queues events smoke --verbose
lakeshore queues drain smoke
lakeshore queues unarchive smoke
lakeshore queues rm smoke --force
```

**Expect:** `ls` and `rm` are the irregular verbs — `list` and `remove`
do not exist in this group. The `default` queue is auto-created on first
list and refuses both `archive` and `rm` with a 409.

## 9. Jobs

```bash
lakeshore jobs list
lakeshore jobs list --state queued --state running
lakeshore jobs show <id>
lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue smoke
```

**Expect:** a queued invocation flips to `killed`; a **running** one
returns 409 with the claiming worker id, because there is no daemon-side
cancel signal in the wire protocol yet. `bulk-cancel` touches only
`state=queued` rows.

## 10. Exec on a daemon

```bash
# Pinned to one daemon. Flags come BEFORE the daemon arg; no `--`.
lakeshore daemon exec <daemon> nvidia-smi
lakeshore daemon exec --timeout 30 --workdir /tmp <daemon> ls -la

# Queue-routed — the control plane picks an eligible member.
lakeshore exec --queue training 'python -c "print(1+1)"'

# Pipe a local Python script to a remote python3 -
lakeshore run ./train.py --daemon <id> --env SEED=1
```

**Expect:** both exec commands use pass-through options, so CLI flags
must precede the command and no `--` separator is used. `lakeshore exec`
with no `--queue` pops an interactive picker. `lakeshore run` is
Python-only: it pipes the file bytes to a remote `python3 -` and writes
nothing to disk on the daemon.

## 11. Secrets, storage, and code

```bash
cat ~/.aws/keypair.json | lakeshore secrets add aws-lab --kind aws_keypair
lakeshore secrets list
lakeshore secrets show aws-lab                 # metadata only, never plaintext
lakeshore secrets rotate aws-lab --from-file ./new.json

lakeshore storage add smoke-bucket --kind s3 \
  --bucket my-smoke-bucket --region us-east-1 --creds aws-lab
lakeshore storage presign smoke-bucket hello.txt --put
lakeshore storage credentials smoke-bucket --env
lakeshore storage remove smoke-bucket

lakeshore code push --storage code-staging
lakeshore code list
```

**Expect:** `secrets add --kind` is required and accepts `ssh_key`,
`aws_keypair`, `gcp_sa_json`, `opaque`; with no `--from-file`, the
plaintext is read from stdin. Plaintext is AES-256-GCM encrypted and is
never returned by any GET. `code push` defaults to the storage named
`code-staging`.

## 12. Tunnels and mounts (metadata only)

```bash
lakeshore tunnels add ny-lab --kind wireguard --config-file ~/wg/ny-lab.conf
lakeshore tunnels list
lakeshore providers update bos14 --tunnel ny-lab
lakeshore providers update bos14 --tunnel ""
lakeshore tunnels remove ny-lab

lakeshore mounts add data --kind nfs --kwarg server=10.0.0.1 --kwarg path=/data
lakeshore mounts list
lakeshore mounts update data --kwarg options=ro
lakeshore mounts remove data
```

**Expect:** both round-trip cleanly with no side effects on running
daemons. Neither is activated by the runner yet — see
[Tunnels](/get-started/tunnels.md) and [Mounts](/get-started/mounts.md).

## 13. Admin tokens and namespaces

Requires `LAKESHORE_ADMIN_TOKEN`; the group exits 1 without it.

```bash
lakeshore admin tokens create ci-runner
lakeshore admin tokens ls
lakeshore admin tokens revoke <id>

lakeshore admin ns create scratch
lakeshore admin ns ls
lakeshore admin ns rm scratch
```

**Expect:** the minted plaintext (`dlk_…`) is shown exactly once.
`tokens list` ≡ `ls`; `namespace` ≡ `ns`, with `ls` and `rm` aliases.
Revocation is a tombstone — the row stays listable and the auth hook
returns 403.

## 14. Shell completion

```bash
echo 'eval "$(lakeshore completion bash)"' >> ~/.bashrc
exec bash
lakeshore                                   # top-level verbs
lakeshore providers                         # subcommands
lakeshore providers add --from-aws-profile  # AWS profiles from ~/.aws
lakeshore daemon kill                       # live worker ids
lakeshore providers show                    # live provider names
```

**Expect:** `completion` accepts `bash`, `zsh`, or `fish` and exits 2 on
anything else. Server-backed values are cached for 30 s under
`$XDG_CACHE_HOME/lakeshore/`; local lookups (SSH aliases, AWS profiles,
gcloud configs, kube and docker contexts) are instant and uncached.

## 15. Dashboard

```bash
lakeshore tui                       # aliases: dashboard, top
lakeshore tui --tab queues --interval 2
```

**Expect:** `--tab` takes `workers`, `queues`, `invocations`, or
`storage`; an unknown value warns and falls back to `workers`. It exits 2
with a friendly message when stdout is not a TTY or no control plane is
configured.

> **Note:** The CLI defines no `.version()`, so `--version` is rejected as an unknown
> option. Check the installed version with `npm ls -g @dreamlake/lakeshore`.

## Read next

- [CLI](/cli.md) — the full command reference.
- [CLI examples](/cli/examples.md) — runnable snippets per verb group.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — what the daemon does
  between install and kill.
- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — per-launcher configuration.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f1-sync

# F1 — sync blocking

F1 is the first row of the [execution matrix](/python-sdk/architecture/execution-matrix.md):
invoke, then stand still until the value comes back. It is the least clever
form and the one to exhaust before reaching for ids, generators, or async.
Most code is control flow around values, and a blocking call is the shortest
path from a call site to a value.

| Level | Spelling | Blocks on |
| --- | --- | --- |
| L0 one call | `f(x)` · `f.local(x)` | the body itself |
| L1 queue fan-out | `q.result(inv, timeout=)` per id | one invocation |
| L2 static pipeline | list-resolve between stages | the slowest job in a stage |
| L3 dynamic graph | `while` / `if` around blocking calls | each step, by design |

## L0 — one call

A bare `@dls.udf` runs in-process on a plain call, inside a fresh child of
the ambient run context. This is the whole local tier: no plane, no worker,
no wire, and no import of `httpx` or `msgpack`.

```python
import dreamlake.lakeshore as dls

@dls.udf
def stats(a: list[float]) -> dict:
    return {"mean": sum(a) / len(a)}

stats([1.0, 2.0, 3.0])   # {'mean': 2.0}
```

On a queue-bound UDF the plain call keeps F1 semantics — it routes to
`.remote()`, which is submit plus wait. Same call site, same shape of code,
different tier:

```python
import threading
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")
q = dls.SyncQueue("f1-demo", dispatch=plane)

@dls.udf(queue="f1-demo")
def embed(x: str) -> list[float]:
    return [float(len(x))]

done = threading.Event()
worker = threading.Thread(target=dls.run_worker, args=(q,), kwargs={"stop": done.is_set})
worker.start()

embed("hello", _dispatch=plane)   # [5.0] — blocks until the worker answers
done.set()
worker.join()
```

> **Warning:** `.remote()` resolves through `Dispatch.result(inv_id)` with no timeout
>   argument, so it inherits that method's 30-second default and raises
>   `TimeoutError` past it. A plain call is therefore only F1-shaped for jobs
>   that finish inside 30 seconds. For anything longer, submit and resolve
>   explicitly: `q.result(f.submit(x), timeout=3600)`.

`f.local(x)` forces the local tier even when a queue is bound, so debugging
and unit tests stay plain function calls:

```python
embed.local("hello")   # [5.0] — no queue, no network
```

`.local()` builds a fresh unbound copy of the function, which means it does
not carry the original `transport=` setting across. That is irrelevant for
local execution — nothing is serialized — but do not read `.local()` as
"the same UDF, minus the queue".

Data UDFs are F1 at L0 too: under `with dls.scope(prefix):` a plain call
reads and writes through the `dls.run` seam and returns plain string keys.
See [F6 — scopes](/python-sdk/architecture/f6-scopes.md) for the key contract, and
[`@udf`](/python-sdk/udf.md) for the decorator reference.

## L1 — fan-out, blocking resolve

At L1 the F1 spelling is `q.result(inv, timeout=)` — the blocking resolve
for one durable invocation id. Fan out with `submit`, then block per id, in
submission order:

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")
q = dls.SyncQueue("f1-demo", dispatch=plane)

@dls.udf(queue="f1-demo")
def square(x: int) -> int:
    return x * x

invs = [square.submit(i, _dispatch=plane) for i in range(4)]
for _ in invs:
    dls.run_worker(q, once=True)     # zero-infra drain: one call per job

results = [q.result(inv, timeout=5) for inv in invs]   # blocks, in order
assert results == [0, 1, 4, 9]
```

There is no `gather`. A list comprehension of `submit` calls plus one
`result` per id is the fan-out, and it is deliberate: every element of the
fan-out is a string that outlives the process holding it.

The cost is head-of-line waiting. Jobs still run concurrently on the
workers, so total wall time is the slowest job rather than the sum — but you
observe completions in submission order, and a fast finisher at the end of
the list sits resolved and unread while you wait on an earlier slow one.
Reacting to whichever finishes first is the [F2](/python-sdk/architecture/f2-ids.md)
row's problem, not a reason to make this code async.

## L2 — static pipeline, the barrier

Chain stages with ordinary calls; the list-resolve between stages is a
barrier. Stage 2 sees plain values, never ids:

```python
@dls.udf(queue="f1-demo")
def total(xs: list[int]) -> int:
    return sum(xs)

inv2 = total.submit(results, _dispatch=plane)   # stage-1 values feed stage 2
dls.run_worker(q, once=True)
q.result(inv2, timeout=5)                       # 14
```

Barrier semantics are the point, not a limitation: every stage boundary is a
checkpoint where all upstream values exist and the whole DAG so far is
debuggable with `print`. The price is that no stage-2 work starts until the
slowest stage-1 job lands. For wide, uneven stages that idle time is real —
pipeline the ids instead, as `examples/02_pipeline.py` does.

## L3 — dynamic graph, blocking on purpose

When the next step depends on the last value, blocking is not a compromise —
it is the semantics. A rework loop cannot proceed without the result it is
looping on:

```python
x = 3
while x < 100:
    inv = square.submit(x, _dispatch=plane)
    dls.run_worker(q, once=True)
    x = q.result(inv, timeout=5)     # each step waits — by design

x   # 6561
```

The graph this loop traces was not knowable at author time; its shape *is*
the sequence of resolved values. Conditionals, retries, and data-dependent
fan-out all read this way: ordinary Python control flow, with a blocking
resolve wherever an edge needs a value. Width comes back only when a single
dynamic wave is itself wide — then the wave fans out via F2 and F1 blocks
once at the wave boundary.

## When to reach for F1

Default to it. F1 is correct whenever the caller has nothing better to do
than wait: scripts, tests, `.local()` debugging, stage barriers you actually
want, and any dynamic step whose successor needs the value. Leave it only
for a named reason — overlapping uneven fan-out (F2), streaming partials
(F3/F5), or a caller that must stay responsive (F4).

Runnable end to end in the dreamlake-lakeshore repo:
`examples/02_pipeline.py` (the two-stage barrier) and
`examples/04_ids_not_futures.py` (the same levels, spelled with ids).

---

Source: https://lakeshore.dreamlake.ai/python-sdk/invocations

# Invocation ids

There is **no Future type** in the SDK. A pending job is its
**invocation id** — a plain string (a ULID), returned by every submit:

```python
inv = q.submit({"seed": 42})       # 'inv' is a str
inv = train.submit(seed=42)        # same for a queue-bound @udf
```

The id is the whole handle. It is durable state on the dispatch plane,
not an object tied to your process — so everything a Future does, a
string does better:

| With a Future you'd… | With an id you… |
| --- | --- |
| hold the object until it resolves | `q.result(inv, timeout=...)` |
| lose the handle when the process dies | write the id to a file or DB; any process re-attaches |
| await it | `await q.result(inv)` on the async `Queue` |
| iterate streamed results | `q.stream(inv, since=cursor)` |
| cancel it | `q.cancel(inv)` |

## Reconnect-by-id

Because the id is just a string, crash recovery is trivial — persist ids
at submit time, re-attach later from anywhere:

```python
# process A — submit and record
ids = [train.submit(seed=i, _key=f"sweep/{i}") for i in range(100)]
Path("ids.txt").write_text("\n".join(ids))

# process B — hours later, on another machine
q = dls.SyncQueue("gpu")
for inv in Path("ids.txt").read_text().splitlines():
    print(inv, q.result(inv, timeout=600))
```

`_key` (the idempotency key) closes the last gap: if process A crashes
*between* submitting and recording, rerunning it resubmits the same keys
and gets the **same ids back** — no duplicate jobs.

## Fan-out + collect

The Python for-loop is the spawn; the result loop is the barrier. There
is no scheduling DSL — the call graph is the program.

```python
@dls.udf(queue="compute")
def square(x: int) -> int:
    return x * x

q = dls.SyncQueue("compute")

ids = [square.submit(i) for i in range(100)]            # fan out
results = [q.result(i, timeout=60.0) for i in ids]      # collect, in order
```

Async fan-out composes with **stdlib asyncio**:

```python
q = dls.Queue("compute")
ids = [await q.submit({"x": i}) for i in range(100)]
results = await asyncio.gather(*(q.result(i, timeout=60) for i in ids))
```

As-they-finish is `asyncio.as_completed` over the same coroutines:

```python
for coro in asyncio.as_completed([q.result(i) for i in ids]):
    handle(await coro)
```

> **Warning:** `dls.gather` and `dls.as_completed` do not exist. They belonged to the
> pre-rewrite surface and were deleted. Use the stdlib versions over
> `q.result` coroutines, or a plain loop on `SyncQueue`.

### Pipelines

Chain stages by feeding one collect into the next comprehension — that
is the whole DAG mechanism (`examples/02_pipeline.py`):

```python
raw = [q.result(i) for i in [download.submit(u) for u in urls]]
out = [q.result(i) for i in [process.submit(r) for r in raw]]
```

> **Warning:** The SDK has no `PipelineRun`, no run handle, and no DAG object — a
> pipeline is the ids you keep. This is an acknowledged gap (the F7 row in
> the SDK repo's `docs/execution-patterns.md`); see
> [F7 · The run object](/python-sdk/architecture/f7-run-object.md).

## Errors

`q.result` raises `DispatchError` when the invocation failed or was
cancelled (the message carries the worker's error) and `TimeoutError`
when the wait window elapses. A timeout is **not** a failure — the id is
still live; call `result` again with a longer window.

```python
try:
    val = q.result(inv, timeout=10.0)
except TimeoutError:
    ...            # still running; the id remains valid
except DispatchError as e:
    ...            # the body raised, or the invocation was cancelled
```

`DispatchError` lives in `dreamlake.lakeshore.dispatch`:

```python
from dreamlake.lakeshore.dispatch import DispatchError
```

The default `timeout` on `result` is 30 seconds — including for the
implicit fetch behind a plain call on a queue-bound UDF. Long jobs want
an explicit `submit` + `result(timeout=...)` pair.

## Streaming

A generator or async-generator body appends one **frame** to the
invocation's journal per yield; consumers iterate live while the job
runs:

```python
@dls.udf(queue="gpu")
async def traj_to_ego_views(traj: str):
    for i, frame in enumerate(render(dls.run.read(traj))):
        frame.save(dls.run.write(f"images/{i:04d}.png"))
        yield [f"images/{i:04d}.png"]          # a partial manifest, live

inv = traj_to_ego_views.submit("outputs/0007/traj.npz")

q = dls.Queue("gpu")
async for manifest in q.stream(inv):           # frames as they land
    print(manifest)                            # ['images/0000.png'], ...
```

The journal has an explicit terminal frame, so *producer finished* and
*connection dropped* are distinguishable: `END` stops iteration cleanly,
`ERROR` raises `DispatchError` carrying the producer's exception type
and message, and silence past `timeout` raises `TimeoutError`.

### Cursor reconnect

Every frame carries a monotonic `seq`, zero-based per invocation. Pass
the last seq you saw as `since` to resume without loss or duplication:

```python
frames = list(q.stream(inv, timeout=5))          # all four frames
tail   = list(q.stream(inv, since=1, timeout=5)) # frames 2 and 3
```

The journal is derived data — the terminal result always lives on the
invocation row, so even a trimmed journal degrades to `q.result(inv)`,
never to wrong state.

## Read next

- [Queue API](/python-sdk/queue.md) — the verbs these ids flow through.
- [`@udf` decorator](/python-sdk/udf.md) — `submit` on a queue-bound
  function, and streaming bodies.
- [Dispatch planes](/python-sdk/dispatch.md) — where invocation rows and
  journals actually live.

---

Source: https://lakeshore.dreamlake.ai/cli/examples/storage-code-mounts

# Storage, code, and mounts

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives on
[Auth + setup](/cli/examples/auth.md).

## Storage

`lakeshore storage` registers S3-compatible object stores. Note the
verb is **singular** — `storage`, not `storages` — even though every
sibling group is plural.

Two kinds:

| Kind        | Meaning                    | Required config      |
| ----------- | -------------------------- | -------------------- |
| `s3`        | A full bucket              | `bucket`             |
| `s3-prefix` | A scoped directory source  | `bucket`, `prefix`   |

Both also accept `region`, `endpoint` (for MinIO / R2 / Ceph /
Backblaze), and `creds.$secret` pointing at a registered
`aws_keypair` secret.

### `storage add` — register a backend

The common fields have direct flags. `--creds <secret>` is shorthand
for `--kwarg creds.$secret=<name>`.

```bash
lakeshore storage add my-s3 \
  --kind s3 \
  --bucket my-bucket \
  --region us-east-1 \
  --creds aws-prod-keys
```

```bash
lakeshore storage add training-data \
  --kind s3-prefix \
  --bucket my-bucket \
  --prefix datasets/imagenet \
  --creds aws-prod-keys \
  --description "read-only imagenet source"
```

Non-AWS S3 needs `--endpoint`:

```bash
lakeshore storage add r2-cache \
  --kind s3 \
  --bucket cache \
  --endpoint https://<account>.r2.cloudflarestorage.com \
  --creds r2-keys
```

Longer configs can come from a file, and `--kwarg` sets arbitrary
config fields with dotted keys nesting:

```bash
lakeshore storage add my-s3 --kind s3 --config-file /tmp/storage.yaml
lakeshore storage add my-s3 --kind s3 --kwarg bucket=my-bucket --kwarg 'creds.$secret=aws-prod-keys'
```

> **Note:** `storage add` calls Commander's `allowUnknownOption()`. Any other
> `--<key> <value>` pair you pass is folded into the config object as
> `key=value`. That means a typo'd flag silently becomes a config field
> instead of erroring — check `storage show` after adding.

### `storage add --provision` — create the bucket too

Requires `creds` to resolve to an `aws_keypair` secret. The
controlplane calls `CreateBucket` on your behalf; it's idempotent if
the bucket already exists under the same account. `--s3-option` is
repeatable and is forwarded verbatim as a `CreateBucket` parameter.

```bash
lakeshore storage add new-bucket \
  --kind s3 \
  --bucket lakeshore-scratch-42 \
  --region us-east-1 \
  --creds aws-prod-keys \
  --provision \
  --s3-option ACL=private
```

### `storage list` / `show`

```bash
lakeshore storage list
lakeshore storage list --json
lakeshore storage show my-s3
lakeshore storage show my-s3 --json
```

### `storage update` — patch the config

`--kwarg` deep-merges (pass `null` to drop a key); `--config-file`
replaces the config wholesale. `--description` updates the description.

```bash
lakeshore storage update my-s3 --kwarg region=us-west-2
lakeshore storage update my-s3 --description "prod artifacts"
```

### `storage remove` — delete the entry

Deletes the record only. `--purge` also deletes the S3 bucket, which
needs credentials in the config.

```bash
lakeshore storage remove my-s3
lakeshore storage remove my-s3 --purge
```

### `storage presign` — a presigned URL

Two positionals: the storage name and the object key (relative to the
storage prefix). Defaults to a GET; `--put` presigns an upload.
`--expires-in` defaults to 3600 seconds and is capped at 86400.

```bash
lakeshore storage presign my-s3 path/to/artifact.tar.gz
lakeshore storage presign my-s3 path/to/artifact.tar.gz --expires-in 600
lakeshore storage presign my-s3 uploads/new.bin --put
```

Because the result is a plain URL, `curl` is the object-verb surface:

```bash
URL=$(lakeshore storage presign my-s3 hello.txt --put --expires-in 600)
curl -X PUT --upload-file ./hello.txt "$URL"
curl -s "$(lakeshore storage presign my-s3 hello.txt)"
```

### `storage credentials` — temporary STS credentials

Short-lived credentials a local script or SDK can use against the
bucket directly. `--duration` defaults to 3600 seconds and accepts
900–129600.

```bash
lakeshore storage credentials my-s3
lakeshore storage credentials my-s3 --duration 7200 --json
eval "$(lakeshore storage credentials my-s3 --env)"   # export into this shell
```

## Code

`lakeshore code` archives the current **git tree** and uploads it to a
storage entry, so a remote daemon can pull the exact snapshot.

### `code push` — upload a snapshot

Deduped by `(git remote, commit)` — re-running on the same clean commit
is a no-op that reports `already archived`. The default storage entry
name is the literal `code-staging`.

```bash
lakeshore code push
lakeshore code push --storage my-s3
```

An uncommitted working tree is refused (exit 2) unless you pass
`--dirty`, which stashes, archives, and pops.

```bash
lakeshore code push --dirty
```

If the storage entry doesn't exist yet, the failure message tells you
how to create it:

```bash
lakeshore storage add code-staging --kind s3 \
  --kwarg bucket=<your-bucket> --kwarg 'creds.$secret=<aws-creds>'
```

### `code list` — show pushed snapshots

```bash
lakeshore code list
lakeshore code list --remote https://github.com/you/your-repo.git
lakeshore code list --json
```

## Mounts

A **mount** is a declaration the runner attaches into the job workdir.
The CLI writes metadata; the daemon does the actual mounting.

`--kind` is required and must be one of:

```text
nfs  samba  ftp  sftp  s3  s3fs  google_drive  dropbox  bind  configmap
```

> **Warning:** The `mounts add --kind` validator in CLI v0.2.0 accepts exactly the ten
> kinds above. If you see a `git` kind documented elsewhere, it is ahead
> of the shipped CLI.

Each kind has its own required config fields, validated server-side:

| Kind           | Required            | Also accepted                            |
| -------------- | ------------------- | ---------------------------------------- |
| `nfs`          | `server`, `path`    | —                                        |
| `samba`        | `server`, `share`   | path within the share, credentials       |
| `ftp` / `sftp` | `server`            | path, credentials                        |
| `s3`           | `bucket`            | `endpoint`, `region`, `prefix`, `creds`  |
| `s3fs`         | `bucket`            | `endpoint`, `region`, `prefix`, `creds`, `options` |
| `google_drive` | `folderId`          | `creds`                                  |
| `dropbox`      | `path`              | `creds`                                  |
| `bind`         | `hostPath`          | `readOnly`                               |
| `configmap`    | `kubeNamespace`, `name` | `items`                              |

`s3` is the userspace-client form (copy on read); `s3fs` is the
FUSE driver that gives job code a real filesystem path.

### `mounts add`

```bash
# NFS export
lakeshore mounts add datasets \
  --kind nfs \
  --kwarg server=10.0.0.10 \
  --kwarg path=/srv/datasets

# S3 via s3fs; creds.$secret references a registered secret by name
lakeshore mounts add s3-cache \
  --kind s3fs \
  --kwarg bucket=my-bucket \
  --kwarg prefix=cache/ \
  --kwarg 'creds.$secret=s3-creds'

# Host bind
lakeshore mounts add hf-cache \
  --kind bind \
  --kwarg hostPath=/home/ge/.cache/huggingface \
  --kwarg readOnly=true

# Kubernetes ConfigMap
lakeshore mounts add my-cm \
  --kind configmap \
  --kwarg kubeNamespace=default \
  --kwarg name=app-cm
```

`--config-file` supplies the same fields from YAML or JSON. `--kind`
stays on the command line either way:

```bash
cat > /tmp/mount.yaml <<'YAML'
server: 10.0.0.10
path: /srv/shared
YAML
lakeshore mounts add shared --kind nfs --config-file /tmp/mount.yaml
```

Any `$secret` marker inside the config must name a secret that exists
in the same namespace, or the write is rejected with a 422.

### `mounts list` / `show` / `update` / `remove`

```bash
lakeshore mounts list
lakeshore mounts list --json
lakeshore mounts show datasets --json

lakeshore mounts update datasets --kwarg path=/srv/datasets-v2
lakeshore mounts remove datasets
```

## Read next

- [Storages](/get-started/storages.md) · [Mounts](/get-started/mounts.md)
- [Payloads](/get-started/payloads.md) — how job inputs and outputs move.
- [Secrets, modes, tunnels](/cli/examples/secrets-modes-tunnels-config-nymph.md) — registering the credentials these reference.

---

Source: https://lakeshore.dreamlake.ai/get-started/functions

# Functions

A **Function** is a Python callable decorated with `@dls.udf`. The
decorator has exactly one job: decide whether calling the function runs
it here or submits it somewhere else.

```python
import dreamlake.lakeshore as dls

@dls.udf                          # local tier — a plain in-process call
def stats(xs: list[float]) -> dict:
    return {"mean": sum(xs) / len(xs)}

@dls.udf(queue="gpu")             # remote tier — submit and wait
def embed(text: str) -> list[float]:
    ...
```

The full signature is short, and there is nothing else on it:

```python
udf(fn=None, *, queue: str | None = None, transport: str = "auto")
```

No `mode=`, no `retries=`, no `timeout=`, no `image=`. Runtime shape
belongs to the queue and the daemon, not to the decorator.

## The two tiers

| Form | `fn(...)` does |
| --- | --- |
| `@udf` (bare) | Runs the body in-process and returns the value. Imports nothing distributed. |
| `@udf(queue="…")` | Submits, then blocks for the result — equivalent to `fn.remote(...)`. |

Three entry points sit on the decorated object:

| Method | Returns | Behaviour |
| --- | --- | --- |
| `fn.local(*a, **kw)` | the value | Always in-process, even on a queue-bound UDF. |
| `fn.submit(*a, **kw)` | `str` | Enqueue and return the invocation id immediately. |
| `fn.remote(*a, **kw)` | the value | Submit, then fetch the result. An `async def` body returns an awaitable instead of blocking. |

`submit()` accepts three private keyword arguments alongside your own:
`_dispatch=` to target a specific plane, `_key=` for an idempotency key,
and `_priority=`.

## There is no Future type

A pending job **is** its invocation id — a plain string (a ULID). Persist
it, log it, hand it to another process; anything that can reach the same
dispatch plane can re-attach with it.

```python
inv = embed.submit("hello")           # -> "01JZ…" immediately
...                                   # crash, restart, different machine
q = dls.SyncQueue("gpu")
value = q.result(inv, timeout=60.0)   # re-attach by id
```

Fan-out is therefore a list comprehension and the join is a loop (or
`asyncio.gather` over the async `Queue.result`) — there is no `gather()`
helper and no `as_completed()` in the SDK.

## The four body kinds

The function's own shape picks the execution shape — sync returns the
value, async returns a coroutine, a generator body returns an iterator,
and an async-generator body returns an async iterator. `inspect` detects
which one once at decoration, and the `UDF` object exposes the result as
`.is_async`, `.is_gen` and `.is_asyncgen`.

See [the four body kinds](/python-sdk/udf.md#the-four-body-kinds) on the
`@udf` reference for the worked examples of all four.

## How a callable crosses the wire

`Envelope.fn` is a tagged union with two arms, chosen by `transport`:

| Transport | Wire shape | Meaning |
| --- | --- | --- |
| `ref` | `{kind, module, qualname, src_hash}` | The worker imports the module and walks the qualname, so it runs its own checked-out code. `src_hash` is a version stamp — a mismatch fails loud rather than running stale code. |
| `pickle` | `{kind, blob, hash, source, qualname, runtime}` | A by-value cloudpickle snapshot, for lambdas, closures, and `__main__` defs. |

`transport="auto"` — the default — picks `pickle` exactly when the
callable is not importable by reference, and `ref` otherwise.
`transport="ref"` insists on importability and fails loud if it cannot.

Only the **callable** is ever pickled. Arguments, keyword arguments,
results, and journal frames all ride the msgpack + ExtType codec.
Workers refuse pickled functions unless started with `allow_pickled` —
unpickling is code execution, so accepting it is a deliberate act of
trust. The inline pickle ceiling is 32 KiB.

```bash
lakeshore worker start --queue gpu --allow-pickled
```

## Thunks — ad-hoc callables

A `Thunk` bundles a non-importable callable with concrete arguments so
it can be submitted by value without a decorator:

```python
t = dls.thunk(lambda: score(model, val_ds))
inv = q.submit(t)          # -> durable id
val = q.result(inv)
```

The thunk's callable ships under the same `auto` rule (so in practice
by pickle); its args and kwargs ride the normal codec.

## Data UDFs return keys, not bytes

A function that touches files never sees a bucket, a credential, or an
absolute path. It asks the ambient run context for local paths and
returns plain string **keys**:

```python
@dls.udf
def splats_to_mesh(splats: str) -> str:
    mesh = extract(dls.run.read(splats))            # key -> local Path
    mesh.save(dls.run.write("process/mesh.ply"))    # key -> local Path
    return "process/mesh.ply"                       # a KEY, not a path

with dls.scope("scenes/0007"):
    key = splats_to_mesh("source/splats.ply")
```

The **return-your-keys contract** is enforced after every call: every key
written through `dls.run.write` must appear in the returned manifest. The
reverse is deliberately not enforced — a returned string that was never
written may be plain data or an upstream key passed through.

File bytes never ride the msgpack wire; only keys do. See
[`@udf` decorator](/python-sdk/udf.md) for the full `dls.run` surface and
the current limits on storage-backed remote I/O.

## Where the call goes

`_default_dispatch()` resolves the process-wide plane once:

| Condition | Plane |
| --- | --- |
| `LAKESHORE_URL` is set | `HttpDispatch` against that control plane. Token from `LAKESHORE_CLIENT_TOKEN`, namespace from `LAKESHORE_NAMESPACE` (default `"default"`). |
| otherwise | The local SQLite plane at `$LAKESHORE_HOME/dispatch.sqlite` (default `~/.lakeshore`). |

Pass `_dispatch=` on `submit()` — or `dispatch=` on a `Queue` /
`SyncQueue` — to override it per call.

> **Warning:** `nymph`, the Rust daemon, does not execute queue-bound `@udf` bodies.
> Run `lakeshore worker start --queue <name>` (which spawns
> `python -m dreamlake.lakeshore.daemon`) on a host that can import your
> code. The worker deliberately ignores saved `lakeshore auth login`
> credentials — pass `--url` or set `LAKESHORE_URL`.

## What the decorator does not capture

The decorator does not capture closures under the `ref` transport. If
your function depends on outer state, pass it as an argument:

```python
@dls.udf(queue="gpu")
def bad(x):
    return weights @ x        # `weights` comes from the enclosing scope

@dls.udf(queue="gpu")
def good(x, weights):
    return weights @ x
```

Under `pickle` a closure *is* snapshotted, which is why `auto` falls back
to it — but that snapshot freezes at submit time, which is rarely what
you want for production code.

## Read next

- [`@udf` decorator](/python-sdk/udf.md) — every parameter, the four body
  kinds, and the `dls.run` seam in full.
- [Queue API](/python-sdk/queue.md) — `Queue` / `SyncQueue`, streaming, and
  the worker verbs.
- [Invocation ids](/python-sdk/invocations.md) — the no-Future model and
  reconnect-by-id.
- [Queues](/get-started/queues.md) — the operator surface for the queue the
  decorator binds to.
- [Function protocol](/api/function-protocol.md) — the wire-level shape.

---

Source: https://lakeshore.dreamlake.ai/nymph/protocol

# Daemon protocol

The nymph daemon talks to the control plane over **outbound HTTPS only**.
It opens a long-poll, receives commands in the response, runs them, and
posts results back. It never listens on a port and it never accepts an
inbound connection, so a host behind NAT, a VPN, or a corporate firewall
works with no extra plumbing.

This page is the wire-level reference. For the operator-facing commands
that *produce* this traffic, see
[Daemon lifecycle](/nymph/daemon-lifecycle.md).

## Transport and framing

Every daemon → control-plane body is msgpack, encoded as a **map with
string keys** (`rmp_serde::to_vec_named`) — not as positional arrays.
Responses are decoded the same way.

| Property        | Value                                                        |
| --------------- | ------------------------------------------------------------ |
| Method          | `POST` on every route                                        |
| `Content-Type`  | `application/x-msgpack`                                       |
| `Accept`        | `application/x-msgpack`                                       |
| `Authorization` | `Bearer <token>` when a token is configured                   |
| Request timeout | 120 s (HTTP client), idle pool timeout 90 s                   |

Two exceptions:

- **Log chunks** are posted as raw bytes with
  `Content-Type: application/octet-stream` — no base64, no msgpack
  envelope. The Bearer is still attached.
- The **exec-result callback** is accepted by the control plane as
  either JSON or msgpack; nymph sends msgpack.

There is no envelope. There is no `v` / `id` / `ts` / `kind` wrapper
around the body — the body *is* the request struct. Protocol versioning
rides on `HelloRequest.versions.protocol`, whose current value is the
string `"1"`.

## Endpoints

Everything the daemon initiates:

| Path                                    | Purpose                                                                 |
| --------------------------------------- | ----------------------------------------------------------------------- |
| `/v1/daemon/hello`                      | First contact. Identity, capabilities, versions, optional public key.   |
| `/v1/daemon/poll`                       | Long-poll. Carries in-flight status; receives commands.                 |
| `/v1/daemon/ack`                        | Terminal outcome of one `run` invocation.                               |
| `/v1/daemon/keys`                       | [Key rotation](/nymph/key-rotation.md) — signed by the outgoing key.       |
| `/v1/daemon/exec/{exec_id}/result`      | Exec completion callback (exit code, stdout, stderr, duration).         |
| *(server-supplied `LogSink.stdout_url`)* | Streaming exec log chunks. In practice `/v1/daemon/exec/{exec_id}/chunk`. |
| `/v1/namespaces/{ns}/payloads/presign`  | Resolve a large arg payload into a download URL.                        |

There are no server → daemon endpoints. Everything the server wants
done arrives in a poll response.

> **Note:** The control plane serves a `/v1/daemon/event` route (it is what writes
> `invocation.progress` / `invocation.log` / `invocation.event` rows), and
> a `/v1/daemon/result-chunk` route for streamed results. The Rust daemon
> posts to neither today — a stale module docstring in `src/protocol.rs`
> claims otherwise.

## Hello

The first call after boot. It carries everything the control plane needs
to admit the daemon and materialise its Worker row.

### Request

| Field          | Type                    | Notes                                                                  |
| -------------- | ----------------------- | ---------------------------------------------------------------------- |
| `machine_id`   | string                  | Stable per host. `"auto"` in the config resolves to the hostname.      |
| `label`        | string                  | Human label. Empty falls back to the worker id.                        |
| `tags`         | string[]                | `key=value` strings, config tags merged with detected host tags.        |
| `lanes`        | string[]                | Queue memberships. Omitted from the wire entirely when empty.           |
| `capabilities` | `{ runners, max_invocations, workdir }` | What this host can actually run.                        |
| `versions`     | object                  | Self-reported binary + tooling versions (below).                        |
| `public_key`   | string \| absent        | Raw 32-byte Ed25519 key, base64url no padding. Present only when an identity is configured. |

`lanes` is the historical name and remains the wire field; the control
plane maps it onto `Worker.queues`. An empty/absent value means
default-queue membership.

`versions` is probed once at boot, best-effort — anything that fails to
resolve is `null`:

| Key        | Probe                                                       |
| ---------- | ----------------------------------------------------------- |
| `daemon`   | `CARGO_PKG_VERSION` of the running binary                    |
| `protocol` | the constant `"1"`                                           |
| `runsc`    | `runsc --version`                                            |
| `docker`   | `docker version --format {{.Server.Version}}`                |
| `python`   | `python3 --version`, else `python --version`                 |
| `kernel`   | `uname -sr`                                                  |
| `distro`   | `PRETTY_NAME=` from `/etc/os-release`                        |
| `arch`     | `uname -m`                                                   |
| `cpu`      | `/proc/cpuinfo` → `{ model, cores }`                         |
| `gpu`      | `nvidia-smi --query-gpu=name,driver_version` + the `CUDA Version:` header → `[{ name, driver, cuda }]` |

### Response

| Field                    | Type            | Notes                                                                  |
| ------------------------ | --------------- | ---------------------------------------------------------------------- |
| `worker_id`              | string \| null  | Canonical Mongo ObjectId. When null, the daemon keeps its provisional `{machine_id}-{ULID}`. |
| `session_token`          | string \| null  | A ULID. **Not persisted server-side** — vestigial for signed daemons.  |
| `session_ttl_s`          | number          | 86400.                                                                 |
| `config`                 | object          | Live config map. Empty (`{}`) today.                                   |
| `cursor`                 | string \| null  | Always null today.                                                     |
| `reject`                 | object \| null  | `{ code, message }`. Non-null aborts daemon startup.                   |
| `key_id`                 | string \| null  | Echoed when a keypair was enrolled on this hello.                      |
| `key_rotation_max_age_s` | number          | Advisory only — the daemon stores nothing and waits for the `rotate_key` directive. |
| `server_time_epoch_s`    | number \| null  | For clock-skew calibration.                                            |

### Rejection

A refusal comes back as **HTTP 200** with a `reject` object, not as a
4xx status. The exact codes:

`protocol_mismatch` · `enroll_token_required` · `invalid_enroll_token` ·
`enroll_token_consumed` · `enroll_token_expired` ·
`enroll_signature_required` · `key_id_mismatch` ·
`enroll_signature_invalid` · `enrollment_signature_required`

A malformed request (missing `machine_id`, `label`, `tags`,
`capabilities.runners`, `versions.daemon`, or `versions.protocol`) is a
`422` carrying `{ error, field }`.

## Poll

### Request

| Field                  | Type                    | Notes                                                       |
| ---------------------- | ----------------------- | ----------------------------------------------------------- |
| `worker_id`            | string                  | Canonical id when hello returned one.                       |
| `queue`                | string                  | From `--queue` / `DREAMLAKE_QUEUE`. Trailing `*` = prefix.  |
| `timeout_s`            | number                  | `[poll] wait_max_s`, default **20.0**. The server caps it at 20. |
| `cursor`               | null                    | Always sent as `null` — cursor tracking is not implemented. |
| `invocations`          | array                   | One `InvocationStatus` per locally-tracked run.             |
| `updated_capabilities` | object \| absent        | Sent on the first poll after a `setup` with `refresh_capabilities`, then cleared. |

`InvocationStatus` is
`{ invocation_id, state, started_at, exit_code, stdout_tail }`. In
practice the daemon only ever publishes `state: "running"` with
`exit_code: null` and an empty `stdout_tail` — stdout tailing is not
implemented.

### Response

```text
{ commands: [ { id, kind, body }, … ], target_nymph_version: string | null }
```

`body` is decoded as a raw msgpack value and then dispatched on `kind`.
A known `kind` whose body does not match, or an entirely unknown `kind`,
decodes to `Unknown` and is logged and skipped — it never wedges the
loop.

### Long-poll mechanics

Each iteration races three things: the cancellation token (SIGTERM /
SIGINT), an internal wake signal fired the instant any exec finishes, and
the poll itself. That wake signal is why back-to-back execs flow at
completion speed instead of waiting out `wait_max_s`.

On a poll error the daemon backs off with full jitter —
`backoff + rand(0,1) * min(backoff, 5.0)` — then doubles up to
`[poll] retry_max_s`. A successful poll resets it to `retry_min_s`.

Concurrency is bounded by a semaphore sized to `[runtime] max_invocations`
(`0` means unbounded). When no permit is free the daemon stops
long-polling entirely and waits for one.

## Command kinds

Exactly seven, evaluated in this order:

| Kind            | Body                                                                | What the daemon does                                                                                        |
| --------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `hibernate`     | `{ until_epoch_s }`                                                  | Returns from the poll loop immediately, sleeps until the deadline on its own clock, then re-enters from scratch. |
| `setup`         | `{ script, sudo?, refresh_capabilities?, name? }`                    | Pushes onto a serial FIFO drained by a dedicated task. Never awaited inline.                                 |
| `exec`          | `{ exec_id, command, env, stdin, workdir?, timeout_s?, log?, payload_inline?, payload_ref? }` | Spawns `bash -c <command>`, posts the result callback.                     |
| `ota_update`    | `{ version, url, sha256 }`                                           | Downloads, verifies, rename-installs over itself, and (by default) `exec()`s the new binary.                 |
| `reset_backoff` | *(none)*                                                             | Resets the poll backoff to `retry_min_s`.                                                                    |
| `rotate_key`    | *(dispatched by kind)*                                               | Generates a new keypair and calls `/v1/daemon/keys`. See [Key rotation](/nymph/key-rotation.md).                |
| `run`           | `{ invocation_id, function, run_config, args_blob, parent_invocation_id, root_invocation_id, script?, as_user? }` | Dispatches to a runner; acks the outcome.                |

> **Warning:** `ota_update`, `reset_backoff`, and `rotate_key` are snake_case. The
> other four are bare words. `reset_backoff` and `rotate_key` carry no
> typed body — they are dispatched purely on the `kind` string.

### `hibernate`

Handling a hibernate **returns from the inner poll loop**, so any
commands later in the same response are discarded. The daemon sleeps
`until_epoch_s − now` seconds (its own clock, cancellable by shutdown),
sets its run-state to `hibernating`, clears the idle tracker, and starts
a fresh poll loop.

### `setup`

Each setup runs through the `process` runner as a synthetic invocation:
`function.module = "_setup"`, `function.qualname` = the supplied `name`
or `"setup"`, and `script` = the body's script. `sudo: true` sets
`as_user = "root"`, which wraps the spawn in `runuser -u root -- …`.

On exit 0 with `refresh_capabilities: true`, host detection re-runs and
the new `Capabilities` snapshot ships on the next poll as
`updated_capabilities`. The control plane merges it into the Worker row
and flips `setting_up` → `active` once the queue has drained.

Setup scripts **must be idempotent** — probe before installing, use
`apt-get install -y`, prefer `systemctl enable --now`, and
read-modify-write configs rather than blind-append.

### `exec`

The command runs under `bash -c` — deliberately **not** `bash -lc`, so
there is no login shell and no `/etc/profile.d`. The supplied `env` map
is merged *on top of* the daemon's own environment; `stdin` is piped then
closed.

Exit-code sentinels on the result callback:

| Value      | Meaning                                                   |
| ---------- | --------------------------------------------------------- |
| *(actual)* | The child exited normally with this code.                 |
| `-1`       | Spawn failure, wait error, or cancellation at shutdown.   |
| `-2`       | Timeout. The child is killed with a 5 s settle grace.     |

Captured stdout and stderr are each capped at **1 MiB**; overflow sets
`error` to `"stdout truncated"` / `"stderr truncated"` /
`"stdout truncated; stderr truncated"`.

When `log` is present the daemon streams instead: reader tasks forward
8 KiB reads into a channel, and a shipper posts the accumulated buffer to
`log.stdout_url` whenever it crosses `flush_bytes` or `flush_ms` elapses.
A failed chunk post keeps the bytes buffered for the next flush and never
fails the exec. In streaming mode the final result callback ships
**empty** `stdout` / `stderr`.

`LogSink` is `{ stdout_url, stdin_url?, control_url?, flush_ms, flush_bytes }`;
`stdin_url` and `control_url` are declared but unused, reserved for
interactive sessions.

The result body is
`{ exit_code, stdout, stderr, duration_ms, error?, result_inline?, result_ref? }`.
`result_inline` and `result_ref` are always null today — the shell exec
path has no typed return value.

### `ota_update`

`sha256` must be exactly 64 ASCII hex characters. The daemon streams the
download to `.nymph-ota-{pid}-{ulid}.tmp` in the *same directory* as the
running executable so the install is an atomic `rename(2)`, hashes on the
fly, refuses on mismatch, `chmod`s `cur_perms | 0o111`, renames over
itself, and then `exec()`s the new binary with the original arguments.
That last step is gated on the `ota-self-replace` cargo feature, which is
**enabled by default**.

> **Warning:** With `[runtime] auto_update = true`, a `target_nymph_version` on the poll
> response that differs from the running version synthesises an OTA no more
> often than `auto_update_check_interval_s`. The URL it builds hard-codes
> **musl** triples (`https://lakeshore.dreamlake.ai/bin/nymph-{triple}-{version}`)
> while CI actually publishes **gnu** triples to the R2 release bucket. The
> source labels this path as scaffolded; prefer the explicit
> `lakeshore daemon update` path, where the control plane derives the URL.

### `run`

`run_config.runner` picks the runner (falling back to
`[runtime] default_runner`). See
[Daemons → Runners](/nymph/daemons.md#runners) for the six accepted kinds
and their contracts. An unknown kind does **not** fall back to the
default — it fails the invocation with `unknown runner kind: {other}`.

## Ack

A terminal `run` outcome is posted immediately rather than waiting for
the next poll:

| Field           | Type                                    |
| --------------- | --------------------------------------- |
| `invocation_id` | string                                  |
| `state`         | `"succeeded"` \| `"failed"` \| `"killed"` |
| `result_blob`   | bytes (empty unless succeeded)          |
| `error`         | `{ type, message, traceback? }` \| null |
| `worker_id`     | string (omitted when empty)             |

`error` is a structured object, never a bare string. On the wire the
inner field is `type` (the Rust struct spells it `type_name` and renames
via serde, because `type` is a keyword). A runner that fails with only a
message gets `type = "DaemonError"`; a cancelled invocation gets
`{ type: "Cancelled", message: "daemon shutdown grace expired" }`.

Ack failures are logged and dropped — they never propagate into the poll
loop.

## Payload store

When a `run` or `exec` body carries a `payload_ref` instead of inline
bytes, the daemon resolves it by posting to
`/v1/namespaces/{ns}/payloads/presign` with
`{ op: "get" | "put", key?, size?, jobId?, kind? }` (camelCase on the
wire) and downloading from the returned URL.

| Constant                  | Value              |
| ------------------------- | ------------------ |
| Inline threshold          | 256 KiB            |
| Max payload               | 100 MiB            |
| Presign fetch timeout     | 60 s               |

`PayloadRef` is `{ bucket, key, size, content_type, sha256? }` — the
daemon accepts both `contentType` and `content_type` on the wire. A
sha256 mismatch short-circuits the exec with `exit_code = -1` and no
child is spawned.

## Auth

Without an identity configured, requests carry the bootstrap bearer from
`[server] token_file` and **no** `X-LS-*` headers — byte-identical to the
legacy path.

With `[identity] enabled = true`, the daemon holds a per-host Ed25519
keypair, enrolls it with a one-shot `dle_` token on `/hello`, and signs
every subsequent request over a canonical string. The control plane's
signature hook guards exactly `/v1/daemon/poll`, `/ack`, and `/event`;
`/hello` and `/keys` verify their own signatures in-route; the exec
result and chunk callbacks are authenticated by the unguessable
`exec_id` in the URL.

The full contract — canonical string, headers, enrollment, rotation,
revocation, and the `LAKESHORE_REQUIRE_DAEMON_SIGNATURES` migration flag
— lives on [Daemon identity and key rotation](/nymph/key-rotation.md).

## Known gaps

- **Cursor tracking is not implemented.** `PollRequest.cursor` is
  hard-coded to `null`; there is no at-most-once replay guard on the
  daemon side. Setup and exec commands are expected to be idempotent for
  this reason.
- **No stdout tailing on poll.** `InvocationStatus.stdout_tail` is always
  empty; live output for a `run` is not on the wire.
- **No cancel signal.** Killing a *running* invocation from the control
  plane returns 409 — the wire protocol has no per-invocation cancel
  command. `SIGTERM` to the daemon is the escape hatch.

## Read next

- [Daemons](/nymph/daemons.md) — runners, workdir contracts, host detection.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — the CLI verbs that emit
  these commands.
- [Daemon identity and key rotation](/nymph/key-rotation.md) — signing.
- [Function protocol](/api/function-protocol.md) — the invocation envelope a
  `run` command carries.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f2-ids

# F2 — deferred ids

Row F2 of the [execution matrix](/python-sdk/architecture/execution-matrix.md): deferred
calls. `submit()` returns a **durable string invocation id** (a ULID) — not
a Future, not a handle object. The id is the whole contract: persist it,
poll it, resolve it, from this process or any other.

| Level | Shape | Spelling |
| --- | --- | --- |
| L0 | one call | `f.submit(x)` → id |
| L1 | fan-out | `[f.submit(x) for x in xs]` |
| L2 | static pipeline | resolved values feed stage 2 |
| L3 | dynamic graph | wave 2 shaped by wave 1's results |

All snippets below share one setup — an in-memory plane, one queue, two
UDFs. Draining is `dls.run_worker(q, once=True)`, one call per job.

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")
q = dls.SyncQueue("ids", dispatch=plane)

@dls.udf(queue="ids")
def square(x: int) -> int:
    return x * x

@dls.udf(queue="ids")
def add(a: int, b: int) -> int:
    return a + b
```

## L0 — one deferred call

Submit, drain, resolve. The id is a plain string from the moment `submit`
returns.

```python
inv = square.submit(7, _dispatch=plane)   # "01KX…" — a plain string
assert isinstance(inv, str)

dls.run_worker(q, once=True)              # drain one job
assert q.result(inv, timeout=5) == 49
```

> **Warning:** `UDF.submit` forwards everything else straight to your function, so its own
>   knobs are namespaced out of the way: `_dispatch=`, `_key=`, `_priority=`.
>   The queue-level `SyncQueue.submit` has no user function to collide with and
>   uses the plain names `envelope=`, `key=`, `priority=`. Mixing them up does
>   not fail at the call site: `f.submit(x, key="…")` packs `key` into the
>   envelope as one of your function's keyword arguments, and the job blows up
>   inside the worker instead.

## L1 — fan-out

A list comprehension over inputs is a list of ids. There is no `gather` and
no `as_completed`; resolve in submission order and the results line up with
the inputs by position.

```python
invs = [square.submit(x, _dispatch=plane) for x in range(5)]
for _ in invs:
    dls.run_worker(q, once=True)

results = [q.result(i, timeout=5) for i in invs]
assert results == [0, 1, 4, 9, 16]        # submission order
```

## L2 — static pipeline

Stage boundaries are resolution points. Resolve stage 1, feed the values to
stage 2 — the DAG is ordinary Python control flow, not a graph API.

```python
stage2 = add.submit(results[3], results[4], _dispatch=plane)
dls.run_worker(q, once=True)
assert q.result(stage2, timeout=5) == 25  # add(9, 16)
```

## L3 — dynamic graph

The second wave's width depends on the first wave's results — structure
discovered at run time. Ids are the edges; a hand-kept edge list **is** the
run record, and it serializes for free because every node is a string.

```python
edges = []                                # (parent_id, child_id) pairs
wave2 = []
for parent, r in zip(invs, results):
    if r > 5:                             # width decided by the data
        child = square.submit(r, _dispatch=plane)
        edges.append((parent, child))
        wave2.append(child)

for _ in wave2:
    dls.run_worker(q, once=True)
assert [q.result(c, timeout=5) for c in wave2] == [81, 256]
```

Two of five first-wave results cleared the bar, so the second wave has two
nodes. Dump `edges` to a journal line and the whole run replays. Keeping
that list by hand is exactly the work the missing
[run object](/python-sdk/architecture/f7-run-object.md) would take over.

## Idempotent submits

The same key on the same queue returns the **same** id — resubmit after a
crash and you get the invocation you already had, not a duplicate job.
Idempotency is scoped per namespace, queue, and key.

```python
a = square.submit(7, _key="scene7/x", _dispatch=plane)
b = square.submit(7, _key="scene7/x", _dispatch=plane)
assert a == b                             # same key → same id
```

The queue-level spelling drops the underscore. It enqueues a raw wire
payload rather than a UDF envelope, so it is claimed by a hand-rolled worker
(`q.claim()` / `q.working()`), not by `dls.run_worker` — that loop decodes
every job as an `Envelope`:

```python
c = q.submit({"x": 1}, key="scene7/y")
d = q.submit({"x": 1}, key="scene7/y")
assert c == d
```

## An id is a fact, not state

A Future is process-bound state: it dies with the event loop that made it,
cannot be written to disk, and means nothing to another process. An id is a
fact about the world — "invocation `01KX…` exists on queue `ids`" — true
regardless of who holds it. That is why the whole Future family was deleted:
every capability it had is a string plus `q.result`, and the string survives
restarts.

Persist the id, resolve it later — the reader needs nothing but the id and a
queue on the same plane.

```python
with open("/tmp/run-id.txt", "w") as f:
    f.write(inv)                          # the checkpoint is one line

later = open("/tmp/run-id.txt").read()    # any process, any time
assert q.result(later, timeout=5) == 49
```

Note that `q.result` defaults to `timeout=30.0` seconds and raises
`TimeoutError` past the deadline; pass an explicit `timeout=` for anything
long-running.

## Full example

`examples/04_ids_not_futures.py` in the dreamlake-lakeshore repo walks
exactly this row, L0 through L3 plus idempotency keys, runnable with zero
infrastructure:

```bash
python examples/04_ids_not_futures.py
```

---

Source: https://lakeshore.dreamlake.ai/python-sdk/dispatch

# Dispatch planes

A dispatch plane is where invocation rows and frame journals live.
`Queue` / `SyncQueue` / `run_worker` take any object with the dispatch
verbs; two implementations ship:

| Plane | Backing | Use for |
| --- | --- | --- |
| `Dispatch` | stdlib sqlite3 (WAL), single file | local runs, dev loops, tests — no server, no network |
| `HttpDispatch` | the Node control plane over httpx | deployed queues, remote workers |

```python
from dreamlake.lakeshore import Dispatch, HttpDispatch, SyncQueue

q = SyncQueue("gpu")                                   # process-default plane
q = SyncQueue("gpu", dispatch=Dispatch(":memory:"))    # ephemeral, tests
q = SyncQueue("gpu", dispatch=HttpDispatch(
    "https://cp.example.com", token=TOKEN,
))
```

Omitting `dispatch` uses the process-default plane, resolved once from
the environment: `HttpDispatch` when `LAKESHORE_URL` is set, otherwise
the local SQLite file at `$LAKESHORE_HOME/dispatch.sqlite` (default
`~/.lakeshore`). Two processes on one machine pointing at the same file
share a working queue: submit in one shell, run a worker in another,
zero infrastructure.

> **Note:** `dls.Dispatch(...)` constructs a plane. There is no module-level
> `dispatch(...)` function.

## `Dispatch` — the minimal plane

The executable specification of the dispatch contract. The Node control
plane implements the same semantics at deployment scale; the conformance
suite runs against both, so the contract cannot fork.

```python
Dispatch(path: str = ":memory:", *, ns: str = "default")
```

The design rules (each one is a named regression test):

1. **One row per invocation** — queue entry, metadata, and state machine
   in one place. No pending list, no secondary index, nothing to drift.
   States are `queued`, `running`, `succeeded`, `failed`, `cancelled`.
2. **Every transition is one guarded atomic UPDATE.** A lost race is a
   clean miss, never a double-delivery.
3. **Fencing** — ack/nack require the claiming worker's id; a stale
   worker whose job was re-dispatched cannot clobber the new attempt.
4. **Idempotent submit** — a caller-supplied `key` is unique per
   (namespace, queue); resubmitting returns the existing invocation.
5. **Counts come from the same rows that get claimed** — they cannot
   disagree with the queue contents.
6. **The journal is derived, not truth** — frames feed streaming
   consumers; the terminal result lives on the invocation row.
   Trimming degrades reads to the row, never to wrong state.

### Verbs

```python
dp.submit(queue, envelope, *, id, key=None, priority=0.0, parent=None) -> str
dp.claim(queue, worker_id, *, kind="fifo", temperature=1.0) -> Claimed | None
dp.ack(inv_id, worker_id, *, result=None)
dp.nack(inv_id, worker_id, *, error, retry=True)
dp.cancel(inv_id) -> bool
dp.get(inv_id) -> dict | None          # the invocation row
dp.result(inv_id, *, timeout=30.0, poll=0.02) -> bytes
dp.count(queue, state="queued") -> int
dp.close()
```

`Claimed` is a dataclass of `id`, `envelope`, `attempt`, `queue`.
Claim ordering by `kind`: `fifo` (priority desc, oldest first; `priority`
is an alias), `filo` (priority desc, newest first), `boltzmann` (sample
proportional to `exp(priority / T)`) — all queries over the same rows
that hold the state.

### The frame journal

```python
dp.journal_append(inv_id, seq, frame_bytes, *, final=False) -> int
dp.journal_read(inv_id, *, since=-1, timeout=30.0, poll=0.02) -> Iterator[(seq, bytes)]
dp.journal_trim(inv_id, *, keep_last=0) -> int
```

An explicit `seq` makes appends idempotent on `(inv_id, seq)` — a
retried frame is absorbed, not duplicated; `seq=None` assigns the next
one. `journal_read` blocks for new frames until a final frame or a
terminal invocation state, and activity resets the timeout; the `since`
cursor makes reconnect lossless. You normally consume this through
`q.stream` and `Topic` on the [Queue API](/python-sdk/queue.md), which
decode frames for you.

Frames are `[kind, seq, body, meta]` on the wire. `FrameKind` is
`VALUE=0`, `CHUNK=1`, `BEGIN=2`, `END=3`, `ERROR=4`; `END` and `ERROR`
are terminal.

## `HttpDispatch` — the Node control plane

The same verbs, spoken over HTTP. The native `Envelope` rides
**opaquely** inside the existing submit body, so the control plane never
decodes user payloads:

```python
HttpDispatch(
    base_url: str,
    *,
    token: str | None = None,
    namespace: str = "default",
    timeout: float = 30.0,
)
```

| Verb | Control-plane route |
| --- | --- |
| `submit` | `POST /v1/producer/submit` |
| `result` / `get` | `POST /v1/producer/await` (poll loop client-side) |
| `claim` | `POST /v1/daemon/poll` |
| `ack` / `nack` | `POST /v1/daemon/ack` |
| `journal_append` | `POST /v1/daemon/result-chunk` |
| `journal_read` | `GET /v1/producer/invocations/:id/result/stream?since=` |

> **Warning:** `HttpDispatch.cancel` and `HttpDispatch.count` raise
> `NotImplementedError` — those control-plane routes have not landed. Both
> work on the local `Dispatch` plane.

`HttpDispatch` also carries `push_code()`, which archives and uploads the
current git tree (deduped per commit) and returns the code-mount stamp.
`UDF.submit` calls it for reference-transport functions so a fresh remote
worker can import your modules — see
[code mount](/python-sdk/happy-paths/10-code-mount.md).

## The worker loop

```python
from dreamlake.lakeshore import SyncQueue, run_worker

q = SyncQueue("gpu")
run_worker(q, root="/data")        # loop until stopped
run_worker(q, once=True)           # drain one job (tests, cron)
```

```python
run_worker(
    q: SyncQueue,
    *,
    root: Path | str | None = None,     # filesystem root for dls.run (default cwd)
    registry: dict[str, Callable] | None = None,
    once: bool = False,
    poll: float = 0.05,
    retries: int = 1,
    stop: Callable[[], bool] | None = None,
    allow_pickled: bool | None = None,
    runtime_policy: str = "minor",      # "strict" | "minor" | "off"
) -> int                                # jobs executed
```

For each claimed job the worker decodes the `Envelope`, resolves the
function (a `registry` hit first, else the `module:qualname` import or
the pickled blob), installs a `RunContext` from the envelope's prefix and
the worker's `root`, runs the body by its kind, and acks the packed
result — or appends an ERROR frame and nacks on failure, retrying while
`attempt <= retries`. Generator yields stream live: one VALUE frame per
yield followed by an END frame, so consumers iterate `q.stream(inv)`
while the job runs. The return-your-keys contract is enforced by the
`UDF` wrapper itself, so a body fails remotely with the same message it
fails with locally.

Two safety knobs:

- **`allow_pickled`** gates by-value (cloudpickle) callables — executing
  one is arbitrary code execution. `None` (the default) resolves by trust
  domain: `True` on the local `Dispatch` plane, `False` on any HTTP
  plane.
- **`runtime_policy`** governs the pre-unpickle interpreter-stamp check
  and the reference-transport `src_hash` staleness check. `"strict"`
  demands exact Python and cloudpickle versions, `"minor"` (default)
  matches `major.minor`, `"off"` skips both — same-machine dev only.

`stop` is polled between jobs for graceful shutdown. `run_worker` returns
the number of jobs it executed.

## The worker entrypoint

The module path `python -m dreamlake.lakeshore.daemon` is the stable
spelling. The TypeScript CLI's `lakeshore worker` (npm
`@dreamlake/lakeshore`) spawns and supervises exactly that module:

```bash
lakeshore worker start --queue gpu --root /data
lakeshore worker start --queue gpu --url https://cp.example.com --token "$TOK"
lakeshore worker once  --queue gpu      # drain at most one job, then exit

python -m dreamlake.lakeshore.daemon --queue gpu --root /data
python -m dreamlake.lakeshore.daemon --queue gpu --once
```

Flags win; env fills the gaps:

| Flag | Env | Does |
| --- | --- | --- |
| `--queue` | `LAKESHORE_QUEUE` | queue name (default `"default"`) |
| `--root` | `LAKESHORE_ROOT` | filesystem root for `dls.run.read/write` (default cwd) |
| `--url` | `LAKESHORE_URL` | control-plane base URL → `HttpDispatch`; unset → the local SQLite plane |
| `--namespace` | `LAKESHORE_NAMESPACE` | control-plane namespace (default `"default"`) |
| `--token` | `LAKESHORE_CLIENT_TOKEN` | bearer token for `--url` |
| `--once` | — | drain at most one job, then exit |
| `--allow-pickled` | — | execute by-value callables |
| `--runtime-policy` | `LAKESHORE_RUNTIME_POLICY` | `strict` \| `minor` \| `off` (default `minor`) |

With no `--url` the entrypoint uses the local SQLite plane and forces
`allow_pickled=True` — the same machine and user is one trust domain.
With `--url` it builds an `HttpDispatch` and accepts pickled callables
only when `--allow-pickled` is passed.

> **Warning:** The `lakeshore-daemon` console script no longer exists — the SDK
> installs no console scripts at all.
> `dreamlake.lakeshore.daemon.entry` still exists, but only as a
> compatibility shim for nymph's hard-coded per-job exec contract; it is
> not a way to run a worker.

## Read next

- [Queue API](/python-sdk/queue.md) — the surface you normally use; these
  planes sit behind it.
- [Invocation ids](/python-sdk/invocations.md) — what `submit` hands back.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths/05-hello-queue-udf

# 05 · Hello, queue-bound UDF

The first happy path where the SDK actually talks to a dispatch plane.
Decorate a function with `@udf(queue=...)`, submit it, and wait for the
result by invocation id.

## Exercises

- `SyncQueue` on the Python SDK side.
- `@udf(queue=...)` submit semantics — `submit()` returns a durable
  invocation id (a string; there is no Future).
- The queue as the function's **scaling unit** — the pool of workers
  consuming `hello` is what you scale to speed this function up.
- `q.result(inv, timeout=…)` — poll-wait for completion, re-attachable
  from any process holding the id.
- The native worker loop — `python -m dreamlake.lakeshore.daemon`, either
  directly or supervised by `lakeshore worker start`.

## Requires

- **Local plane:** nothing. The producer and the worker both default to
  the shared SQLite plane under `~/.lakeshore` — no control plane, no
  queue creation.
- **Remote plane:** a control plane up (path 03 or hosted); pass `--url`
  to the worker and set `LAKESHORE_URL` for the producer (or construct
  the queue with `HttpDispatch`).

> **Warning:** The Rust daemon executes exec-style jobs. A `@udf` on a queue needs the
> Python worker:
> 
> ```bash
> lakeshore worker start --queue hello
> # or, against a control plane:
> lakeshore worker start --queue hello --url http://localhost:8080 --token <token>
> ```

## Verifies

The submit → claim → execute → return round-trip works. `q.result`
returns the function's return value, unmodified.

## Run

Terminal 1 — the worker:

```bash
lakeshore worker start --queue hello
# equivalently:
python -m dreamlake.lakeshore.daemon --queue hello
```

Terminal 2 — the producer:

```python
# hello_udf.py
import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

inv = add.submit(2, 3)
print("invocation:", inv)

q = dls.SyncQueue("hello")
print("result:", q.result(inv, timeout=30.0))
```

For a single-process version with no worker terminal at all — the
producer drains its own queue — see `examples/00_hello_udf.py` in the
SDK repo, which runs on `Dispatch(":memory:")`.

## Expected output

```text
invocation: 01JZ...
result: 5
```

`2 + 3 = 5`. If the second line prints, the round-trip works.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `TimeoutError: invocation ... still 'queued' after 30.0s` | No worker is consuming the queue. Nymph alone does not count — run `lakeshore worker start --queue hello`. |
| `TimeoutError` with the worker running | The producer and the worker are on different planes. Both must share `~/.lakeshore` (the same `LAKESHORE_HOME`) or the same `--url`. |
| `DispatchError: invocation ... failed: ...` | The body raised on the worker — the message carries the error. |
| `WorkerError: cannot import module ...` | The worker cannot import `module:qualname`. The local plane assumes a shared environment — start the worker from the same project, or pass `registry={qualname: fn}` to `run_worker`. |
| `TransportError: this worker refuses pickled functions` | The function is a lambda, a closure, or a `__main__` def, so it shipped by value. Move it to an importable module, or start the worker with `--allow-pickled` if you trust the producers. |

## Status

Manual.

## Next

→ [06 · Fan out and collect](/python-sdk/happy-paths/06-fan-out-gather.md)

---

Source: https://lakeshore.dreamlake.ai/api/function-protocol

# Function protocol

This is the wire-level reference for a remote `@udf` invocation. It
covers the bytes, not the Python API — for the producer surface read
[`@udf` decorator](/python-sdk/udf.md) and [Queue API](/python-sdk/queue.md).

Every body on every hop is **msgpack**. Three hops carry one
invocation:

| Hop | Route | Payload |
| --- | --- | --- |
| Producer → control plane | `POST /v1/producer/submit` | the [Envelope](#the-envelope), opaque inside `args_blob` |
| Control plane → daemon | `POST /v1/daemon/poll` → a `run` command | the same bytes, echoed as `args_blob` |
| Daemon → control plane | `POST /v1/daemon/ack` | the wire-packed return value |

The control plane never decodes user values. It routes on the outer
JSON-ish fields (`queue`, `priority`, `state`) and treats the Envelope
as bytes.

## Value codec

`dreamlake.lakeshore.wire` is the codec. It is msgpack with **integer
`ExtType` codes**, not `{"ztype": ...}` marker dicts — a tagged leaf can
never collide with a user dict, and msgpack does the container recursion.

| Code | Name | Ext body |
| --- | --- | --- |
| `1` | `numpy.ndarray` | `msgpack([dtype_str, shape_list, raw_bytes])` |
| `2` | `torch.Tensor` | same as ndarray (detached, moved to CPU) |
| `3` | `PIL.Image` | `msgpack([format_str, encoded_bytes])` |
| `5` | `tuple` | `wire.pack(list(t))` — recursive |

Everything else must be msgpack-native (`str`, `bytes`, `int`, `float`,
`bool`, `None`, `list`, `dict`). Register more with
`wire.register(wire.Codec(...))`.

```python
from dreamlake.lakeshore import wire

blob = wire.pack({"x": np.zeros(4), "pair": (1, 2)})
wire.unpack(blob)          # {"x": array([0., 0., 0., 0.]), "pair": (1, 2)}
```

Three properties worth knowing:

- **`strict_types=True`** — a `tuple` round-trips as a tuple (code 5),
  not silently coerced to a list.
- **Unknown type at pack time raises `TypeError`** with the fix in the
  message, instead of failing deep inside `msgpack`.
- **Unknown ext code at unpack time is preserved verbatim** as
  `msgpack.ExtType(code, data)`, so a newer peer's values pass through an
  older reader unharmed.

`numpy`, `torch`, and `PIL` imports are all deferred inside the codecs —
importing `wire` drags in none of them.

> **Note:** There is deliberately no file/blob codec. A data UDF returns **string
> keys**; the bytes move worker ↔ storage on a side channel. See
> [`@udf` decorator](/python-sdk/udf.md) for the return-your-keys contract.

## The Envelope

One remote invocation. Built by `UDF.submit()` (or `Thunk.to_envelope()`),
serialized with `Envelope.to_bytes()`, and handed to the dispatch plane as
opaque bytes.

```python file="wire.Envelope.to_bytes()"
{
    "v": 1,                     # WIRE_VERSION
    "id": "01HX2K3M4N5P…",      # ULID — the durable handle
    "fn": {...},                # tagged union, see below
    "args": b"…",               # wire.pack(list(args)) — pre-packed
    "kwargs": b"…",             # wire.pack(dict(kwargs)) — pre-packed
    "prefix": "scenes/0007",    # dls.scope prefix for dls.run key resolution
    "key": None,                # optional idempotency key
    "cfg": {},                  # {"code": {...}} when a code archive shipped
    "parent": None,             # parent invocation id (pipeline lineage)
}
```

| Field | Type | Notes |
| --- | --- | --- |
| `v` | int | `WIRE_VERSION`, currently `1`. `from_bytes` rejects `v < 1` and any `v` newer than the reader. |
| `id` | string | ULID generated by the producer. This *is* the handle — there is no Future type. |
| `fn` | dict | How the callable crosses the wire. See [Function transport](#function-transport). |
| `args` | bytes | Pre-packed, so routers never decode user values. |
| `kwargs` | bytes | Same. |
| `prefix` | string | Item root the worker installs into its `RunContext`. `""` when no `dls.scope`. |
| `key` | string \| null | Idempotency key. See the caveat under [Submit](#submit). |
| `cfg` | dict | Only `code` is written today — the code-archive stamp `{remote, commit, archiveKey, storageName}`. |
| `parent` | string \| null | Set on chained submits. |

The header (`id` / `fn` / `prefix` / `cfg`) decodes without touching
`args` / `kwargs`. Local calls never build an Envelope at all — this is
strictly the remote hop.

### Function transport

`Envelope.fn` is a tagged union with two arms.

```python file="fn (kind=ref)"
{
    "kind": "ref",
    "module": "user_pkg.search",
    "qualname": "topk",
    "src_hash": "sha256:5a4f0c…",   # 16 hex chars, or None if source unavailable
}
```

```python file="fn (kind=pickle)"
{
    "kind": "pickle",
    "blob": b"…",                   # cloudpickle, ≤ 32 KiB inline
    "hash": "sha256:…",             # full hex digest of the blob
    "source": "def f(): …",         # best-effort, for audit/display — never executed
    "qualname": "<lambda>",
    "where": "<lambda> (/path/x.py:12)",
    "runtime": {"python": "3.12.4", "impl": "cpython",
                "cloudpickle": "3.0.0", "pickle_protocol": 5,
                "platform": "linux-x86_64"},
}
```

`transport="auto"` (the `@udf` default) picks `pickle` exactly when the
callable is **not** importable by reference — a lambda, a `<locals>`
closure, a `__main__` def, or a callable instance with no `__qualname__`
of its own. Everything else ships by `ref`. `transport="ref"` insists on
importability and raises `TransportError` otherwise; `transport="pickle"`
always snapshots.

Only the **callable** is ever pickled. Args, kwargs, results, and journal
frames stay on the msgpack codec.

Worker-side gates, in order:

1. `allow_pickled` — a pickled `fn` is arbitrary code execution, so
   workers refuse by default. `run_worker` resolves `None` by trust
   domain: `True` on the local SQLite plane, `False` on any HTTP plane.
   `lakeshore worker start --allow-pickled` opts in.
2. `check_runtime()` runs **before** any unpickle. Policy
   `strict` (exact python + cloudpickle) / `minor` (major.minor python,
   major cloudpickle — the default) / `off`.
3. `src_hash` staleness check on the `ref` arm: if the worker's resolved
   source hashes differently from the submitter's, the invocation fails
   loud rather than running stale code. Skipped when the envelope carries
   no hash, the target's source is unavailable, or the policy is `off`.

`INLINE_PICKLE_LIMIT` is 32 KiB. A larger blob is a captured-state
mistake, and the error says so — pass the data as arguments instead.

## Submit

```python file="POST /v1/producer/submit"
{
    "invocation_id": "01HX…",
    "queue": "gpu",
    "function": {"module": "_native_wire",
                 "qualname": "_native_wire.envelope",
                 "version": "v1"},
    "args_blob": b"<the Envelope>",
    "run_config": {"queue": "gpu", "priority": 0.0, "key": None},
    "code_context": {},
    "parent_invocation_id": None,       # omitted when absent
}
```

`function` is a **stub** on the native path — the real callable identity
lives inside the Envelope, and the control plane only needs a `Function`
row to hang the invocation off. `args_blob` is the Envelope, verbatim.

Server side:

- The named queue is ensure-created (idempotent), defaulting to
  `"default"` when `queue` is absent.
- `run_config` is stored with `queue` stamped onto it. Dispatch matches
  on `runConfig.queue`; there is no queue foreign key on the invocation
  row.
- `priority` is denormalized off `run_config.priority` onto the row so
  the claim sort actually sees it. Non-numeric / non-finite values fall
  back to `0`; fractional values are truncated.
- `parent_invocation_id` resolves the row's `rootInvocationId` — the
  parent's root, or the parent itself.

Response: `{"invocation_id": "…", "queue": "…"}`.

> **Warning:** The idempotency key rides `Envelope.key` and `run_config.key`, but no
> control-plane route writes `Invocation.idempotencyKey`, and the sparse
> unique index it would need has to be created by hand in MongoDB. Two
> submits with the same key produce two invocations today.

## Dispatch

A daemon long-polls `POST /v1/daemon/poll`. When the claim succeeds it
gets back one `run` command:

```python file="poll response"
{
    "commands": [{
        "id": "01HX…",                 # command id (ULID)
        "kind": "run",
        "body": {
            "invocation_id": "01HX…",
            "function": {"module": …, "qualname": …, "version": …},
            "args_blob": b"<the Envelope>",
            "run_config": {...},
            "prefix": "scenes/0007",   # only when run_config.prefix is set
        },
    }],
    "target_nymph_version": "0.1.4",   # only when the namespace pins one
}
```

The claim is a single MongoDB `findAndModify` on the `Invocation`
collection: `query: {state: "queued", "runConfig.queue": …}`,
`sort: {priority: -1, submittedAt: 1}`, flipping the row to `running`.

nymph's `RunBody` additionally accepts `parent_invocation_id`,
`root_invocation_id`, `script` (run `bash -c` instead of the Python
entry), and `as_user` (wrap the spawn in `runuser -u`). See
[Daemon protocol](/nymph/protocol.md).

### The workdir contract

nymph's `process` / `docker` / `gvisor` / `slurm` / `kube` runners do not
speak the Envelope. They write `args_blob` verbatim to
`{workdir}/{invocation_id}/args.msgpack`, spawn
`python3 -m dreamlake.lakeshore.daemon.entry` (or `python -m …` inside a
container) with that directory as cwd, and read back:

- exit `0` → `result.msgpack`, the wire-packed return value;
- exit non-zero → `error.json`, `{"type", "message", "traceback"}`,
  reported as `"<type>: <message>"`.

The workdir is deleted on success and left on disk on failure. Full
contract: [Daemons → Shared workdir contract](/nymph/daemons.md#shared-workdir-contract).

> **Note:** `entry.py` drains generators and async-generators into the concatenated
> manifest and returns it as one value. Per-yield streaming exists only on
> the queue-native worker (`lakeshore worker start`), which appends
> [frames](#the-frame-journal) as the body yields.

## Ack

```python file="POST /v1/daemon/ack"
{
    "invocation_id": "01HX…",
    "state": "succeeded",        # or "failed" | "killed" | "timeout"
    "result_blob": b"…",         # wire-packed return value; empty otherwise
    "error": {                   # ErrorInfo, on failure
        "type": "ValueError",
        "message": "…",
        "traceback": "…",        # optional
    },
    "worker_id": "…",            # nymph sends this; omitted when empty
}
```

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `invocation_id` | string | yes | Echoed from the `run` command body. |
| `state` | string | yes | The handler writes it straight to the row. Terminal states the rest of the system recognizes: `succeeded`, `failed`, `killed`, `timeout`. |
| `result_blob` | bytes \| null | no | Stored as-is; `null` when absent. |
| `error` | object \| null | no | `{type, message, traceback?}`. On the Rust side the field is `type` on the wire. |

The handler sets `finishedAt = now()` unconditionally. `startedAt` is set
server-side by the claim, not reported by the daemon.

Outcomes a nymph runner can produce: `succeeded`, `failed` (a bare-string
failure gets `type: "DaemonError"`), and `killed` (cancellation, with
`{type: "Cancelled", message: "daemon shutdown grace expired"}`).

`POST /v1/producer/await` is the producer-side counterpart: a long poll
(default `timeout_s` 60, 100 ms spin) returning
`{state, result_blob, error}` once terminal, or `{"state": "pending"}` on
timeout.

> **Warning:** The ack envelope carries no `started_at` / `finished_at`, no
> `stdout_uri` / `stderr_uri`, and no resource metrics. Progress and log
> lines flow on the separate `POST /v1/daemon/event` channel, which the
> control plane stores as `invocation.progress` / `invocation.log` /
> `invocation.event`. nymph itself does not post to `/v1/daemon/event`
> today.

## The frame journal

A streaming invocation writes **frames**. One frame is a 4-element
msgpack array:

```python
[kind, seq, body, meta]
```

| `kind` | Name | Meaning |
| --- | --- | --- |
| `0` | `VALUE` | One yielded/returned value; `body` is the value. |
| `1` | `CHUNK` | A slice of one large value; `body` is raw bytes. |
| `2` | `BEGIN` | Opens a CHUNK run; `meta` may carry `{total?, content_type?}`. |
| `3` | `END` | Terminal — the producer finished. |
| `4` | `ERROR` | Terminal — the producer raised; `meta` is `{type, message}`. |

`seq` is a monotone per-invocation cursor starting at `0` — the reconnect
handle. An explicit `END`/`ERROR` frame closes the stream, so "producer
finished" stays distinguishable from "connection dropped".

Frames are msgpack-self-delimiting: N frames concatenated, or split at
arbitrary byte boundaries by HTTP chunking or Redis, reassemble through
one `msgpack.Unpacker`. That is what `wire.FrameReader` does.

**Append** — `POST /v1/daemon/result-chunk`:

```python
{"invocation_id": "01HX…", "seq": 3, "bytes": b"<frame>", "final": False}
```

The row is written first (Mongo is truth), then the frame is `XADD`ed to
the Redis stream for live readers. A retried POST of an already-journaled
`(invocation_id, seq)` returns `{"ok": true, "seq": 3, "duplicate": true}`
and deliberately does *not* re-`XADD`. A Redis failure is logged and
swallowed — live readers catch up from Mongo.

**Read** — `GET /v1/producer/invocations/:id/result/stream?since=<seq>`:

A chunked `application/x-msgpack-stream` body. Frames are written
**verbatim, with no SSE wrapper and no delimiter**. The route replays
Mongo rows with `seq > since`, then follows the Redis stream, ending on a
final frame or once the invocation reaches a terminal state. The `seq`
you resume from comes from inside the decoded frame, not from the
transport.

## Versioning

`Envelope.v` is the single version integer. `Envelope.from_bytes` rejects
a version newer than the reading client with a message naming the fix
(upgrade `dreamlake-lakeshore`). Unknown ext codes and unknown `cfg` keys
pass through untouched, so additive changes do not need a version bump.

> **Warning:** `HttpDispatch.cancel()` and `HttpDispatch.count()` raise
> `NotImplementedError`. Killing an invocation goes through
> `DELETE /v1/namespaces/:ns/invocations/:id`, which only works on a
> **queued** row — a running invocation returns 409, because the wire
> protocol has no daemon-side cancel signal yet.

## See also

- [Daemon protocol](/nymph/protocol.md) — the full `/v1/daemon/*` surface.
- [Dispatch planes](/python-sdk/dispatch.md) — `Dispatch`, `HttpDispatch`, `run_worker`.
- [Invocation ids](/python-sdk/invocations.md) — reconnect-by-id and streaming from Python.
- [Payload store](/get-started/payloads.md) — the out-of-band path for large blobs.

---

Source: https://lakeshore.dreamlake.ai/cli/examples/secrets-modes-tunnels-config-nymph

# Secrets, modes, tunnels, config, nymph

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives on
[Auth + setup](/cli/examples/auth.md).

## Secrets

Server-stored credentials referenced by providers, mounts, storage
entries, and tunnels. Plaintext **only flows on the way in** — `list`
and `show` return metadata.

`--kind` is **required** and must be one of `ssh_key`, `aws_keypair`,
`gcp_sa_json`, or `opaque`.

### `secrets add` — from a file

```bash
lakeshore secrets add my-pem \
  --kind ssh_key \
  --from-file ~/.ssh/id_ed25519 \
  --description "primary dev key"
```

### `secrets add` — from stdin

When `--from-file` is absent the CLI reads piped stdin, which keeps the
plaintext off disk.

```bash
pbpaste | lakeshore secrets add gcp-sa --kind gcp_sa_json          # macOS
cat ~/Downloads/sa.json | lakeshore secrets add gcp-sa --kind gcp_sa_json
```

### `secrets list` / `show` — metadata only

```bash
lakeshore secrets list
lakeshore secrets show my-pem
```

### `secrets rotate` — replace the plaintext

Same input rules as `add`: `--from-file`, or piped stdin.

```bash
lakeshore secrets rotate my-pem --from-file ~/.ssh/id_ed25519.new
```

### `secrets remove`

```bash
lakeshore secrets remove my-pem
```

Secrets are referenced elsewhere by the `$secret` marker — e.g.
`--kwarg 'creds.$secret=aws-prod-keys'` on `storage add` and
`mounts add`, or `--creds aws-prod-keys` as shorthand on `storage add`.

## Modes

A **mode** is a server-stored RunConfig: a named compute target
(provider, resources, tags). The Python API dispatches against modes;
the CLI manages their definitions.

> **Warning:** Every other group's inline override flag is `--kwarg`. `modes add` and
> `modes update` use `--field`. (The generated shell completion still
> advertises `--kwarg` here — that table is stale.)

### `modes add` — from a file

```bash
cat > /tmp/h100.yaml <<'YAML'
provider: my-ec2
resources:
  gpus: 1
  cpus: 8
  memory_gb: 64
tags:
  - gpu:h100
YAML
lakeshore modes add h100 --config-file /tmp/h100.yaml
```

### `modes add` — inline field overrides

Dotted keys nest.

```bash
lakeshore modes add cpu-small \
  --field provider=my-ec2 \
  --field resources.cpus=2 \
  --field resources.memory_gb=8
```

### `modes list` / `show` / `update` / `remove`

`update` merges; `=null` drops a key.

```bash
lakeshore modes list
lakeshore modes show h100

lakeshore modes update h100 --field resources.gpus=2
lakeshore modes update h100 --field tags=null
lakeshore modes update h100 --config-file /tmp/h100.yaml

lakeshore modes remove h100
```

## Tunnels

WireGuard configs the control plane uses to reach providers behind a
VPN. `--kind` defaults to `wireguard`, and `wireguard` is the only
supported value.

### `tunnels add` — from a config file

`--config-file` accepts `.conf` (wg-quick INI), `.yaml`, or `.json`.

```bash
lakeshore tunnels add wg-vpc --config-file /tmp/wg-vpc.conf
```

### `tunnels add --conf` — inline INI

```bash
lakeshore tunnels add wg-vpc --conf "$(cat <<'INI'
[Interface]
PrivateKey = abc123=
Address    = 10.0.0.2/24

[Peer]
PublicKey  = peer-key=
Endpoint   = 1.2.3.4:51820
AllowedIPs = 10.0.0.0/24
INI
)"
```

### `tunnels list` / `show` / `remove`

`remove` refuses while any provider still references the tunnel.

```bash
lakeshore tunnels list
lakeshore tunnels list --json
lakeshore tunnels show wg-vpc --json
lakeshore tunnels remove wg-vpc
```

Attach a tunnel to a provider with `providers update <name> --tunnel
<tunnel>` — see
[Providers and discover](/cli/examples/providers-and-discover.md).

## Nymph binaries

For rolling debug builds at a fleet without cutting an R2 release.
Pair with `daemon update --url`. Both subcommands need an admin token —
`--admin-token`, defaulting to `$LAKESHORE_ADMIN_TOKEN`.

### `nymph push` — upload a binary

`--target <triple>` defaults to a value inferred from the ELF header's
`e_machine`; `--name <label>` labels the upload.

```bash
LAKESHORE_ADMIN_TOKEN=changeme \
  lakeshore nymph push ./target/x86_64-unknown-linux-gnu/release/nymph
```

```text
✓ uploaded nymph (4.1 MB)
  sha256: 3f2a…
  label:  nymph
  target: x86_64-unknown-linux-gnu
  url:    /v1/admin/nymph-releases/3f2a…
  full:   http://localhost:8080/v1/admin/nymph-releases/3f2a…
```

> **Warning:** The `url` field in the `--json` envelope is a path, not an absolute
> URL. Daemons OTA from the fully-qualified address — the `full:` line in
> the human output. Prefix the server origin yourself when scripting.

### `nymph list` — what's stored

```bash
LAKESHORE_ADMIN_TOKEN=changeme lakeshore nymph list --json
```

### Push and roll the fleet

```bash
PUSH=$(LAKESHORE_ADMIN_TOKEN=changeme lakeshore nymph push ./nymph --json)
lakeshore daemon update --all \
  --url "$LAKESHORE_URL$(jq -r '.url' <<<"$PUSH")" \
  --sha256 "$(jq -r '.sha256' <<<"$PUSH")" \
  --yes
```

> **Note:** `nymph push` deliberately has no `--update <daemon...>` flag. Chain it
> to `daemon update --url` as above.

## Config

Diagnostics for the resolved `.dreamrc`. None of the three subcommands
take flags.

```bash
lakeshore config show      # print the resolved DreamRc as YAML
lakeshore config source    # which source it came from
lakeshore config refresh   # force a server pull and update the on-disk cache
```

`config source` prints one of `env`, `workspace`, `home`, `server`,
`cache`, or `none`, matching the resolution order (first hit wins):

1. `--dreamrc <path>` — the CLI's one global option (reported as
   `explicit`).
2. `DREAMRC` env var → `env`.
3. `./.dreamrc`, walking up from cwd → `workspace`.
4. `$HOME/.dreamrc` → `home`.
5. A server pull (needs auth or `LAKESHORE_URL`) → `server`.
6. The on-disk cache from a previous pull → `cache`.
7. Nothing — an empty DreamRc → `none`.

The server pull composes a DreamRc from the namespace's `providers` and
`modes`, and caches it at
`$XDG_CACHE_HOME/lakeshore/<server-host>/<namespace>/dreamrc.yml`
(dir 0700, file 0600). If the pull fails, the CLI falls back to that
cache and prints `⚠ server unreachable, using cached config from <ts>`.

> **Warning:** `.dreamrc` (providers + modes, YAML with `!providers.` tags) is
> **not** the same as `.lakeshore` / `.lakeshore.local` (project defaults
> for `daemon launch`, a top-level `daemon:` block, walked up from cwd
> and stopped at the git root), and neither is `lakeshore.yaml` (the
> compose file for `up` / `down` / `ps` / `status` / `logs`, read from
> cwd only). See [Configuration](/api/configuration.md).

## Completion

```bash
echo 'eval "$(lakeshore completion bash)"' >> ~/.bashrc
echo 'eval "$(lakeshore completion zsh)"'  >> ~/.zshrc
lakeshore completion fish | source
```

Full details — the static tree, the dynamic `__suggest` callback, and
the 30-second cache — at [`/cli/completion`](/cli/completion.md).

## Read next

- [Auth and secrets](/api/auth-and-secrets.md) — the server-side model.
- [Tunnels](/get-started/tunnels.md) · [Configuration](/api/configuration.md)
- [Daemons + exec/run](/cli/examples/daemons-and-exec.md) — where `daemon update --url` lands.

---

Source: https://lakeshore.dreamlake.ai/cli/release-notes

# Release notes

Per-version changelog for the `@dreamlake/lakeshore` CLI. Pre-1.0, so
minor bumps may include breaking changes — each release flags those
under **Changed**.

> **Note:** Entries below 0.1.16 carry the dates from the original changelog. The
> 0.1.16 and 0.2.0 entries are reconstructed from the release commits and
> are listed without a date rather than with a guessed one.

## 0.2.0

### Added

- **`lakeshore tui`** — an Ink terminal dashboard over Workers /
  Queues / Invocations / Storage, with per-item action hints rendered
  inline and dispatched from the same table. Aliases `dashboard` and
  `top`. See [TUI dashboard](/cli/tui.md).
- **`lakeshore worker start` / `worker once`** — supervise a native
  Python queue worker on the local machine (`python -u -m
  dreamlake.lakeshore.daemon`). Omitted flags are deliberately kept off
  the child argv so the module's own `LAKESHORE_*` env fallbacks stay
  authoritative. Added Bun runtime support to the `bin` shim.
- **`lakeshore dreamlake`** — the DreamLake data-plane group ported
  from the Python CLI: `login` / `logout` / `profile`, `upload` /
  `download` / `list`, `create` / `update` / `delete` for bindrs and
  datasets, `video`, `artifact`, and `vectorize`. It uses its own
  servers and its own credential store under `~/.dreamlake/`.
- **`lakeshore storage`** — S3-compatible storage entries: `add`
  (with `--provision` to create the bucket), `list`, `show`, `update`,
  `remove --purge`, `presign`, and `credentials` (STS). Direct flags
  for the common config fields (`--bucket`, `--region`, `--prefix`,
  `--endpoint`, `--creds`) plus pass-through for anything else.
  `<namespace>/<name>` syntax works in every storage subcommand.
- **`lakeshore code push` / `code list`** — archive the current git
  tree and upload it to a storage entry, deduped by
  `(remote, commit)`. Default storage name is `code-staging`.
- **`lakeshore admin`** — `tokens create` / `list` / `revoke` and
  `namespace create` / `list` / `delete`. Every subcommand requires
  `LAKESHORE_ADMIN_TOKEN`.
- **Compose daemon-group fields** — `setup` (bash snippets),
  `startup` (Python-env script), `python`, and `mounts` in
  `lakeshore.yaml`.

### Changed

- **`host_setup` → `setup`** in the compose daemon-group schema.
- **Mounts are per daemon group** in the compose runConfig, not
  project-level.

### Fixed

- **S3 PUT uploads are buffered**, not streamed — S3 rejects
  `Transfer-Encoding: chunked`.
- **`mounts add --kind` help** now lists all ten kinds the server
  accepts: `nfs`, `samba`, `ftp`, `sftp`, `s3`, `s3fs`,
  `google_drive`, `dropbox`, `bind`, `configmap`.
- Server-side `hint` fields on errors are surfaced in CLI output, and
  the `token create` login hint includes `--namespace`.

## 0.1.16

### Added

- **`lakeshore queues`** — the queue CRUD surface: `add`, `ls`,
  `show`, `patch`, `archive`, `unarchive`, `drain`, `rm`, plus
  `events` (the elasticity controller's audit trail) and `stats`
  (current daemon / utilization / pending signals). `daemon launch
  --queue` and `daemon update --add-queue` / `--remove-queue` plumb
  membership through `Worker.queues`.
- **Compose** — `lakeshore up` / `down` / `ps` / `status` / `logs`,
  reading a `lakeshore.yaml` (`-f` to override). Stateless: desired
  state comes from the file, current state from the control plane on
  every run. See [Compose](/get-started/compose.md).
- **`lakeshore exec --queue <name>`** — skip the daemon picker and let
  the control plane route the command to an eligible queue member.

### Changed

- **Lanes were replaced by queues.** The `lanes` CLI surface is gone;
  `Worker.queues` is the membership field.

### Fixed

- **`daemon update`** resolves a label to a worker id before PATCHing
  queue membership.

## 0.1.15 — 2026-05-22

### Added

- **`lakeshore exec <cmd>`** — top-level shorthand. Pops an
  interactive picker over the daemon list (freshest-first) and runs the
  command on the chosen daemon.
- **Name resolution on `lakeshore daemon exec`** — the `<daemon>` arg
  accepts a worker id, a label (set by `daemon launch --name`), or a
  `machine_id`. ObjectId-shaped strings skip the lookup. Duplicates
  report the candidates and ask for `--id <id>`; `--id` forces
  literal-id mode.
- **`lakeshore daemon install --binary <path>`** — rsyncs a local nymph
  binary to the SSH host instead of downloading a release. Mutually
  exclusive with `--nymph-base-url`. ELF-validated to reject Mach-O.
- **`lakeshore run <script.py>`** — pipes a local Python file to a
  remote `python3 -` over the exec channel. Picker if no `--daemon`.
- **`lakeshore nymph push <file>` / `nymph list`** — uploads a hashed
  nymph binary to the control plane (admin-authed) so debug-fleet
  daemons can OTA without cutting a release.
- **No `--` separator on `exec` / `daemon exec`.**
  `passThroughOptions()` means CLI-side flags go before the daemon
  argument and everything after it is the remote command.

### Changed

- **`POLL_WINDOW_S` 60 → 25** — fits under Heroku's hard 30-second
  router cap (H12). Long-poll cycles now land under the limit.

### Fixed

- **Exec stdin bytes weren't reaching the daemon.** Storage-vs-wire
  mismatch in the control plane (`stdin_b64: string` on the ExecJob row
  vs `stdin: Vec<u8>` on the daemon's `ExecBody`). The transform now
  decodes base64 → bytes at the poll-pop boundary.
- **`daemon exec <label>` 500'd during name lookup.** The control
  plane's `prisma.worker.findUnique({where:{id}})` rejected
  non-ObjectId strings as malformed; CLI-side name resolution
  pre-resolves to the canonical id.

## 0.1.14 — 2026-05-21

### Added

- **`daemon exec <id> <command>`** — run an arbitrary command on a
  daemon with stdin / env / timeout / long-poll. Returns stdout,
  stderr, and the exit code.
- **`daemon launch --count N`** — parallel fleet launches with a
  progress bar; bounded concurrency plus retry on AWS
  `RequestLimitExceeded`.
- **`daemon launch --name <name>`** and `daemon.name` in `.lakeshore`
  for human-friendly labels.
- **`daemon.capacity` in `.lakeshore`** — per-project instance-type
  override that flows through `daemon launch`.
- **`daemon launch-log <id>`** — fetch cloud console output for
  boot-time debugging (EC2 only).
- **`.lakeshore` project config** — a local file declaring
  project-level daemon launch / setup defaults, consumed by
  `daemon launch`. Walks up from cwd and stops at the git root;
  `.lakeshore.local` layers on top.
- **`daemon install`** improvements — runs as the installing user,
  adds the user to the `docker` group, suppresses TTY warnings,
  4-step progress bar.
- **Picker polish** — hashed tag colors for stable visual identity,
  expand-on-`→`, SLURM / Kube / Docker entries visible alongside cloud
  providers.

### Changed

- **`daemon ota` → `daemon update`** — renamed for clarity, with new
  `--all` and `--prefix <pattern>` flags for bulk updates. `ota`
  survives as a **hidden, deprecated alias** that still requires
  `--version`; new scripts should use `daemon update`.
- **`daemon kill`** — terminating the cloud instance is now the
  default (opt out with `--no-terminate`); the `--provider` flag was
  dropped; bulk kill via `--prefix <pattern>` + `--yes`.

## 0.1.13 — 2026-04-18

### Added

- **`--watch` on `daemon list` / `daemon show`** — live refresh in the
  terminal, `Ctrl-C` to exit. The value is optional; bare `--watch`
  means every 2 seconds.

## 0.1.12 — 2026-04-15

### Added

- **`daemon hibernate` / `daemon ota` / `daemon target-version`** —
  remote-control subcommands wired end to end through the control
  plane and the nymph daemon.
- **`--watch`** on daemon and jobs `list` / `show` (initial cut).

## 0.1.10 — 2026-04-01

### Added

- **Mounts CRUD** — `lakeshore mounts add / list / show / update /
  remove` for `nfs`, `samba`, `ftp`, `sftp`, `s3`, `s3fs`,
  `google_drive`, `dropbox`, `bind`, and `configmap`. Server-side
  source of truth; the CLI writes metadata only. See
  [Mounts](/get-started/mounts.md).

### Fixed

- **`daemon install`** re-anchored as a subcommand of `daemon` (it had
  been accidentally chained off `show.action()`).

## 0.1.9 — 2026-03-15

### Added

- **One-command daemon lifecycle** — `lakeshore daemon install
  <ssh-alias>` SSHes to the host, drops the `nymph` binary, registers
  it, and waits for the Worker row. Pair with `daemon kill <id>` and
  `daemon cleanup --older-than <duration>` to prune stale rows. See
  [Daemon lifecycle](/nymph/daemon-lifecycle.md).
- **Interactive provider editor** — `lakeshore providers edit` opens a
  two-step picker (provider → action) for hide / unhide / soft-delete /
  restore. `providers list --hidden --deleted --all` widens the view.
  See [Providers](https://docs.dreamlake.ai/lakeshore/providers#interactive-editor).
- **Cloud-config auto-detect** — `lakeshore aws discover` and
  `lakeshore gcp discover` enumerate profiles already on your laptop;
  `providers add --from-aws-profile <name>` /
  `--from-gcloud-config <name>` seed credentials and region in one
  step. See [EC2 Provider](https://docs.dreamlake.ai/lakeshore/providers/ec2#seeding-from-an-aws-profile) and
  [GCE Provider](https://docs.dreamlake.ai/lakeshore/providers/gce#seeding-from-a-gcloud-configuration).
- **SSH picker** — `lakeshore ssh discover` marks entries with
  local-only directives stripped and aliases already on the server.
  `lakeshore ssh upload` with no alias drops into a multi-select
  grouped into "new" / "existing", dedup'd by SSH fingerprint. See
  [SSH Provider](https://docs.dreamlake.ai/lakeshore/providers/ssh#importing-from-sshconfig).
- **Shell completion** covering the top-level groups, their
  subcommands, flag names, and dynamic values. Install with
  `lakeshore completion bash|zsh|fish`. See
  [Completion](/cli/completion.md).
- **Tunnels** (metadata only) — `lakeshore tunnels add / list / show /
  remove` stores wg-quick configs; `providers update <name> --tunnel
  <tunnel>` attaches one. See [Tunnels](/get-started/tunnels.md).

---

Source: https://lakeshore.dreamlake.ai/nymph/key-rotation

# Daemon identity and key rotation

Each daemon can hold a per-host **Ed25519** keypair. The private key
never leaves the host; the daemon signs every control-plane request with
it, and the control plane verifies against the public key it recorded at
enrollment. This replaces the bearer-only handshake with a real host
identity, per-request signatures with replay protection, scheduled and
on-demand rotation, and immediate revocation.

The signing string is byte-identical on both sides, so it is documented
here exactly.

> **Note:** The control plane accepts signatures today and the daemon can generate,
> sign, and rotate. Neither is enforced: nymph's `[identity] enabled`
> defaults to `false`, and the control plane defaults to
> signatures-optional. Enabling is a two-sided, per-fleet decision — see
> [Enforcing signatures](#enforcing-signatures).

## Enroll tokens

Enrollment is gated by a one-shot **enroll token**, distinct from the
namespace API tokens in [Auth and secrets](/api/auth-and-secrets.md).

- **Format** `dle_<base64url(32 bytes)>`. The `dle_` prefix is greppable
  for accidental-paste audits — note it differs from the CLI/SDK's
  `dlk_`.
- **One-shot.** Consumed atomically on first successful enrollment; two
  concurrent enrollments collapse to exactly one winner.
- **TTL.** Default **3600 s** (1 hour); override per-mint with `ttl_s`,
  capped at 30 days.
- **Namespace-scoped.** A token only enrolls a daemon into the namespace
  it was minted for.
- **Hash at rest.** Only `sha256(plaintext)` (hex) and an 8-character
  display prefix are stored. The plaintext is returned **once**, at mint.

| Verb     | Path                                   | Notes                                                                                                    |
| -------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `POST`   | `/v1/namespaces/:ns/enroll-tokens`     | Admin only. Body `{ label, ttl_s? }`. Returns the plaintext once.                                        |
| `GET`    | `/v1/namespaces/:ns/enroll-tokens`     | Lists `{id, label, prefix, createdAt, expiresAt, consumedAt, consumedByWorkerId, spent}` — never plaintext. |
| `DELETE` | `/v1/namespaces/:ns/enroll-tokens/:id` | Admin only. Burns the token (marks it consumed) while preserving the audit row.                           |

Labels match `^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$`. The mint returns `201`:

```json
{
  "id": "…",
  "label": "bos14-rig",
  "prefix": "dle_AbCd",
  "plaintext": "dle_AbCd…",
  "expiresAt": "2026-06-26T13:34:56.000Z",
  "createdAt": "2026-06-26T12:34:56.000Z"
}
```

## Daemon configuration

```toml file="udf-daemon.toml"
[identity]
enabled = true
key_file = "/etc/dreamlake/daemon.key"
enroll_token_file = "/etc/dreamlake/enroll-token"   # or: enroll_token = "dle_…"
```

| Key                 | Default                      | Meaning                                                        |
| ------------------- | ---------------------------- | -------------------------------------------------------------- |
| `enabled`           | `false`                      | Master switch. When off, requests carry **no** `X-LS-*` headers. |
| `key_file`          | `/etc/dreamlake/daemon.key`  | Private key. Created `0600` on first boot, reused thereafter.   |
| `enroll_token`      | *(unset)*                    | Inline `dle_…` plaintext.                                       |
| `enroll_token_file` | *(unset)*                    | File containing the token, trimmed. Keeps it out of the TOML.   |

The inline value wins when both are set. If `enabled = true` but the key
cannot be loaded, the daemon **fails closed** and exits rather than
silently degrading to bearer.

The key is persisted as the raw 32 secret bytes, base64url-nopad, one
line, mode `0600` — written to a `<name>.tmp` sibling that is `chmod`'d
before an atomic rename into place. `load()` accepts base64url or
standard base64 and rejects anything that is not exactly 32 bytes. The
`Debug` impl prints only the key id, never the secret.

## Enrollment

On first boot the daemon generates its keypair, writes the private key,
and derives its **key id**:

```text
keyId = base64url_nopad(sha256(raw_32_byte_public_key))[:16]
```

It is a 16-character truncation of the base64url encoding — not of the
hex digest, and not the full 43-character encoding.

The daemon then calls `POST /v1/daemon/hello` carrying the raw public key
(base64url, no padding) in `public_key`, the enroll token as
`Authorization: Bearer dle_…`, and the
[signature headers](#the-signing-scheme) self-signed with the new key.
Because the daemon does not yet know its control-plane worker id, the
canonical string uses `machine_id` in the worker-id slot.

The control plane then:

1. Decodes the public key (must be exactly 32 bytes).
2. Validates the enroll token — exists, not consumed, not expired — but
   does **not** consume it yet.
3. Confirms `X-LS-Key-Id` matches the key id it derives from the supplied
   key, then verifies the self-signature.
4. Upserts the Worker, stores the key as `active` with
   `expiresAt = now + key_rotation_max_age_s`, and **atomically** consumes
   the enroll token.

Because the consume happens last, a bad signature never burns a token. If
the consume loses a race, no key is stored and the response rejects with
`enroll_token_consumed`.

Namespace precedence on hello: the enroll token's namespace wins, then
any pre-existing Worker row matched by `machineId`, then `default`.

> **Warning:** A refused enrollment comes back as `200` with a
> `{ "reject": { "code", "message" } }` msgpack body — not a 401 or 403.
> Codes: `enroll_token_required`, `invalid_enroll_token`,
> `enroll_token_consumed`, `enroll_token_expired`,
> `enroll_signature_required`, `key_id_mismatch`,
> `enroll_signature_invalid`, `enrollment_signature_required`, and
> `protocol_mismatch`.

The successful hello response carries the canonical `worker_id` (which
the daemon stamps in for all subsequent signing), the confirmed `key_id`,
`key_rotation_max_age_s`, and `server_time_epoch_s`.

## The signing scheme

Every signed request carries four headers and a detached Ed25519
signature over a **canonical string** — eight lines joined by a single
`\n`:

```text file="canonical-signing-string"
lsd-v1

<PATH, no query string>

<NONCE, verbatim from the header>
<lowercase hex sha256(BODY bytes)>
```

`lsd-v1` is the literal domain separator — it versions the scheme.
Hashing the body into the string means one mechanism covers msgpack and
JSON routes alike, and it is why the control plane re-registers its body
parsers to stash the exact on-wire bytes.

| Header           | Encoding                                                  |
| ---------------- | --------------------------------------------------------- |
| `X-LS-Key-Id`    | The 16-character key id.                                  |
| `X-LS-Timestamp` | Integer epoch **seconds**, not milliseconds.              |
| `X-LS-Nonce`     | base64url (no padding) of 16 CSPRNG bytes.                |
| `X-LS-Signature` | base64url (no padding) of the 64-byte signature.          |

The prefix is `X-LS-`, not `X-Lakeshore-`. Headers are looked up
lowercase in the header bag.

### Which routes are signed

| Route                                    | How it is authenticated                                  |
| ---------------------------------------- | -------------------------------------------------------- |
| `/v1/daemon/poll`, `/ack`, `/event`      | The signature `preHandler` hook.                          |
| `/v1/daemon/hello`                       | Verifies its own self-signature in-route.                 |
| `/v1/daemon/keys`                        | Verifies in-route, **always** — no bearer fallback ever.  |
| `/v1/daemon/exec/:id/{result,chunk}`, `/v1/daemon/result-chunk` | Not signature-guarded — authenticated by the unguessable id in the URL. |

### Verification order

Each step is a distinct rejection, in this order:

1. Key lookup by `(worker_id, key_id)` — missing → `401` "unknown daemon key".
2. Status: `revoked` → `403`; anything other than `active` or `retiring`
   → `403`.
3. Past the hard `expiresAt` → `401` "daemon key expired".
4. `|now − timestamp| > skew` → `401` "timestamp outside skew window".
5. Ed25519 verify over the recomputed canonical string → `401`
   "signature verification failed".
6. Nonce check-and-record → `401` "replayed nonce".

The nonce step runs **last on purpose**, so an unauthenticated caller
cannot burn cache entries.

The replay cache keys on `(worker_id, nonce)` with a TTL equal to the
skew window. It uses Redis (`SET NX PX` on `nonce:{worker}:{nonce}`) when
`REDIS_URL` is configured, and an in-memory map otherwise. The in-memory
store is correct for a single instance; **configure Redis before running
more than one control-plane dyno**, since only the Redis path's atomic
`SET NX` prevents two dynos accepting the same nonce.

### Header-presence policy

| Headers present    | Behaviour                                                                    |
| ------------------ | ---------------------------------------------------------------------------- |
| All four           | Verify as above (`authMode = "daemon-sig"`).                                 |
| **None**           | Fall back to legacy bearer (`authMode = "daemon-bearer"`) — unless enforcement is on, in which case `401`. |
| Some but not all   | Hard `401` "incomplete signature headers". Never a silent bearer fallback.    |

## Rotation

Keys rotate on two triggers: **scheduled** (the active key reaches
`key_rotation_max_age_s`, default 30 days, or passes `expiresAt`) and
**on-demand**. The flow is zero-downtime via an **overlap window**
(default 600 s).

1. On a poll, the control plane attaches a `rotate_key` command:

   ```json
   {
     "id": "cmd-rotate_key-1750000000",
     "kind": "rotate_key",
     "body": {
       "reason": "max_age_exceeded",
       "deadline_epoch_s": 1750000600,
       "key_id": "<current key id>"
     }
   }
   ```

   `reason` is `max_age_exceeded` or `key_expired`; `deadline_epoch_s` is
   `now + overlap`.

2. The daemon generates a new keypair and calls `POST /v1/daemon/keys`
   with `{ worker_id, public_key }`, **signed by the old key** using the
   canonical worker id. Rotating to the same key id is rejected `422`
   ("new key must differ from the current key").

3. On success the new key is stored `active`; the old key flips to
   `retiring` with `retiringAt = now` and `revokedAt = now + overlap` — a
   *future* timestamp acting as the overlap deadline. Both keys verify
   during the window, so in-flight old-key requests still succeed.

4. The response is
   `{ key_id, status, expires_at, overlap_s, server_time_epoch_s }`. The
   daemon swaps its live signing key only after that confirmation, via an
   atomic swap behind a lock — safe while polls are in flight.

```text
active ──rotate──▶ retiring ──overlap elapses──▶ revoked
   ▲                  │
   └── both verify during the overlap ──┘
```

An opportunistic reaper (run on each poll, idempotent) flips the
`retiring` key to `revoked` once the deadline passes.

A failed rotation is best-effort: the daemon logs a warning, keeps its
current key, and never wedges the poll loop. With no identity configured
the directive is a logged no-op.

`HelloResponse.key_rotation_max_age_s` is purely advisory — the daemon
stores nothing and acts only on the server-issued directive.

## Inspection and revocation

```bash
# List a worker's keys (keyId, status, createdAt, retiringAt, revokedAt,
# expiresAt, publicKey)
curl $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"

# Immediate kill
curl -X POST \
  $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys/$KEY_ID/revoke \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"
```

Revocation sets `status = "revoked"` and `revokedAt = now`; every
subsequent request signed with that key is rejected `403`. The call is
idempotent — a second revoke returns `alreadyRevoked: true`. A revoked
daemon must **re-enroll** with a fresh enroll token.

Note that `keyId` is unique per `(workerId, keyId)`, not globally.

## Enforcing signatures

`LAKESHORE_REQUIRE_DAEMON_SIGNATURES` is the migration switch. It must be
the literal string `"true"`; anything else, including unset, is off.

| State             | `/hello`                                                         | `/poll`, `/ack`, `/event`                     | `/keys`                     |
| ----------------- | ---------------------------------------------------------------- | ---------------------------------------------- | --------------------------- |
| **Off** (default) | `public_key` optional; legacy bearer enrollment still accepted   | Signatures optional; bearer fallback accepted  | Always requires a signature |
| **On**            | Must provide `public_key` + enroll token; legacy path rejected   | Signatures **required**; bearer fallback `401` | Always requires a signature |

Roll out in order: (1) run a control plane that accepts both; (2) enable
`[identity]` on the fleet; (3) confirm no `daemon-bearer` traffic in the
logs; (4) flip the flag and restart.

### Bearer fallback (migration only)

With the flag off and no signature headers present, the request is
accepted in `daemon-bearer` mode and the bearer token itself is **not
validated** — its presence is informational, a bridge for daemons that
have not started signing. Likewise the `session_token` in the hello
response is vestigial: minted, never persisted, on the way out. Treat
both as legacy.

### Tuning knobs

| Env var                               | Default   | Meaning                                            |
| ------------------------------------- | --------- | -------------------------------------------------- |
| `LAKESHORE_DAEMON_SKEW_S`             | `300`     | Timestamp skew window, seconds.                    |
| `LAKESHORE_DAEMON_KEY_OVERLAP_S`      | `600`     | Retiring-key overlap window, seconds.              |
| `LAKESHORE_DAEMON_KEY_MAX_AGE_S`      | `2592000` | Max key age before a rotation directive (30 days). |
| `LAKESHORE_REQUIRE_DAEMON_SIGNATURES` | *(unset)* | `"true"` enforces signatures.                       |

## Operator runbook

**Bring up a signed daemon.** Mint a token, capture the plaintext (shown
once), and hand it to the daemon:

```bash
curl -sX POST $CP/v1/namespaces/$NS/enroll-tokens \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"label": "bos14-rig", "ttl_s": 3600}'
# → { "plaintext": "dle_…", … }   save it now — never shown again
```

Write it to the host's `enroll_token_file`, set `[identity] enabled =
true`, and start the daemon. It generates its keypair on first boot,
enrolls, and the token is spent the moment it registers.

**Force a rotation.** Scheduled rotation is automatic. To pull it
forward, lower `LAKESHORE_DAEMON_KEY_MAX_AGE_S` on the control plane —
the next poll for an over-age daemon then carries the directive. No
restart required.

**Revoke a compromised host.** Find the worker and key id
(`lakeshore daemon show <id>`, or the keys endpoint above), then POST the
revoke route. Re-enroll with a fresh token if the host should rejoin.

## Read next

- [Daemon protocol](/nymph/protocol.md) — the wire protocol these signed
  requests ride on.
- [Auth and secrets](/api/auth-and-secrets.md) — namespace API tokens
  (`dlk_`), distinct from daemon enroll tokens (`dle_`).
- [Daemons](/nymph/daemons.md) — what the daemon does once it is trusted.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f3-generators

# F3 — generators

The generator row of the [execution matrix](/python-sdk/architecture/execution-matrix.md):
results come back one at a time, and the **consumer drives**. Locally that
is Python's own laziness. Remotely, each `yield` becomes a VALUE frame on
the invocation's journal, and `q.stream(inv)` is the pull. Backpressure is
the consumer's pace; durability is the journal; the `since` cursor makes
every stream reconnectable — the property no Future ever had.

| Level | Spelling |
| --- | --- |
| L0 one call | `for y in f(x)` — lazy pull, body runs on demand |
| L1 systems | `q.stream(inv, timeout=)` over journal frames; `Topic` pub/sub |
| L2 static pipeline | stages as generators; the consumer pulls through the chain |
| L3 dynamic graph | a `.send()` trampoline — local plain Python only, **GAP** at system level |

## L0 — a generator body is a generator

`@dls.udf` wraps generator bodies natively — no mode flag, no second
decorator; `inspect.isgeneratorfunction` decides once at decoration. A plain
call returns an iterator, and nothing runs until you pull.

```python
import dreamlake.lakeshore as dls

@dls.udf
def frames(n: int):
    for i in range(n):
        print("rendering", i)
        yield {"frame": i, "px": i * i}

it = frames(3)          # nothing has run yet
first = next(it)        # runs the body up to the first yield
rest = list(it)         # pulls the remainder
```

`frames(3)` prints nothing. The first `print` fires inside `next(it)` —
laziness is the whole point of the row, and the local tier preserves it
exactly. The wrapper installs a per-call run context only around each
`next()`, never while control is back with the consumer, so a `yield` cannot
leak the callee's scope into consumer code.

## L1 — yields become journal frames

Bind the same body to a queue and each `yield` becomes a VALUE frame on the
invocation's journal, followed by an END frame when the body returns. The
consumer is `q.stream(inv)` — live while the job runs, from any process
holding the id.

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")
q = dls.SyncQueue("render", dispatch=plane)

@dls.udf(queue="render")
def render(n: int):
    for i in range(n):
        yield {"frame": i, "px": i * i}

inv = render.submit(4, _dispatch=plane)     # durable string id
dls.run_worker(q, once=True)                # yields -> VALUE frames -> END

for frame in q.stream(inv, timeout=5.0):
    print(frame["frame"], frame["px"])
```

`run_worker(q, once=True)` executes one job; a generator job streams all its
frames and ENDs during that call. In production the worker runs elsewhere
and `stream` follows the journal while the body is still yielding — the
frames are rows, not socket traffic, so producer and consumer never need to
overlap in time.

If the body raises, the worker appends an ERROR frame instead of END, and
`q.stream` re-raises it on the consumer side as a `DispatchError` naming the
producer's exception type and message. A stream ends for exactly one of
three reasons: an END frame, an ERROR frame, or a terminal invocation state
with no further frames.

### Cursor reconnect

Frame seqs are 0-based. `since` is the last seq you saw; `since=-1` replays
from the start. A **new** consumer — a different process, after a crash, an
hour later — resumes without loss or duplication:

```python
cursor = 1                                       # last seq this consumer saw
tail = list(q.stream(inv, since=cursor, timeout=5.0))
assert [f["frame"] for f in tail] == [2, 3]      # frames 0, 1 skipped

replay = list(q.stream(inv, since=-1, timeout=5.0))
assert len(replay) == 4                          # full history, any time
```

This is what F3 buys over a Future. A Future is process-bound state: once
the process that held it dies, the stream is gone. The journal is a fact
about the world — the invocation id plus a cursor reconstructs the consumer.

`timeout` here is patience for *silence*, not a deadline for the whole
stream: every frame that arrives resets it. `q.stream` defaults to 30
seconds of silence before raising `TimeoutError`.

## `Topic` — the same journal, spelled pub/sub

A topic is a named journal channel not tied to any invocation. Same frames,
same cursor semantics. `q.topic(name)` returns one; an omitted name gets a
generated `topic-`.

```python
t = q.topic("progress")
t.post({"done": 10, "of": 30})
t.post({"done": 20, "of": 30})
t.post({"done": 30, "of": 30})
t.post(None, final=True)         # END frame closes the stream

for msg in t.stream(since=-1):   # replay from the start
    print(msg)

late = t.listen(since=0, timeout=5.0)      # next value after seq 0
assert late == {"done": 20, "of": 30}
```

A late subscriber passes the last seq it saw and picks up mid-stream.
`listen` returns one value, or `None` on timeout; `stream` iterates to END.
Both default to `timeout=5.0` — shorter than the invocation stream's 30, so
do not assume one number covers both. One mechanism covers invocation
streaming and pub/sub, so the cursor is learned once.

## L2 — stages as generators

A static pipeline in this row is just composed generators. The consumer's
pull propagates back through the chain; no stage runs ahead of demand.

```python
import dreamlake.lakeshore as dls

@dls.udf
def load(paths):
    for p in paths:
        yield {"path": p}

@dls.udf
def score(rows):
    for r in rows:
        yield {**r, "score": len(r["path"])}

pipeline = score(load(["a/b.png", "c.png", "d/e/f.png"]))
best = max(pipeline, key=lambda r: r["score"])

first = next(score(load(["x.png", "never/loaded.png"])))
```

The second chain pulls exactly one item: `"never/loaded.png"` is never
loaded. This composition is local-tier only — an iterator is in-process
state and cannot ride the wire as an argument. Across queues, each stage's
frames land on its own invocation journal, so every stage boundary is also a
reconnect point.

## L3 — the `.send()` trampoline (GAP)

Python generators are two-way: `.send(value)` delivers a value **into** the
paused body. With plain generator functions that gives a driver/worker
trampoline in a few lines:

```python
def tuner():                     # a plain generator, NOT a @dls.udf
    lr = 0.1
    while True:
        loss = yield lr          # value OUT, measurement IN
        if loss is None:
            return
        lr = lr * 0.5 if loss > 1.0 else lr * 1.1

g = tuner()
lr = next(g)                     # prime: first suggestion
for loss in [2.0, 0.8, 0.5]:
    lr = g.send(loss)            # send the measurement, get the next lr
g.close()
```

> **Warning:** `@dls.udf` returns a wrapper generator that advances the body with plain
>   `next()` calls. A value sent into the wrapper is delivered to the
>   wrapper's own `yield` and discarded — the body never sees it, and no error
>   is raised. Two-way generators must stay undecorated today.

The system-level version — a remote generator whose journal carries frames
in **both** directions, so a consumer can send a value into a paused remote
body — **does not exist**. The journal is one-way: workers append, consumers
pull. This cell is a named gap, not a hidden one. Today you spell two-way
flow as two one-way channels — an invocation stream out, a `Topic` back in —
with the loop in ordinary Python.

## Read next

- [Queue API](/python-sdk/queue.md) — `stream`, `Topic`, and the worker verbs
  in full.
- [F5 — async generators](/python-sdk/architecture/f5-async-generators.md) — the same
  row with an event loop underneath.
- `examples/05_streams_topics.py` in the dreamlake-lakeshore repo — this
  page's L1 material as one runnable, zero-infra script.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths/06-fan-out-gather

# 06 · Fan out and collect

The first happy path that proves parallelism. N invocations submitted in
a list comprehension; the ids are collected with `q.result` in
submission order. There is no `gather` — the id list is the handle.

## Exercises

- Concurrent submission from one client.
- The dispatch plane distributing work across multiple workers.
- Invocation ids as the fan-out handles — the id list **is** the
  barrier; iterate it to collect in submission order.

## Requires

- Everything in [05 · Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md).
- **At least two** workers on the same queue and the same plane
  (otherwise it parallelizes to 1 and only throughput differs from
  path 05):
  ```bash
  lakeshore worker start --queue compute &
  lakeshore worker start --queue compute &
  ```

## Verifies

The plane hands different invocations to different workers (atomic
claim — exactly one worker wins a given row), and collecting by id
preserves submission order regardless of completion order.

## Run

```python
# fan_out.py
import time
import dreamlake.lakeshore as dls

@dls.udf(queue="compute")
def square(x: int) -> int:
    time.sleep(1.0)          # make parallelism visible
    return x * x

q = dls.SyncQueue("compute")

t0 = time.monotonic()
ids = [square.submit(i) for i in range(8)]                 # fan out
results = [q.result(i, timeout=60.0) for i in ids]         # collect
print(results, f"{time.monotonic() - t0:.1f}s")
```

`examples/01_fan_out.py` in the SDK repo is the same shape on an
in-memory plane, with no worker terminals to start.

## Expected output

`[0, 1, 4, 9, 16, 25, 36, 49]` — in input order. Total wall clock should
be roughly `work_time / n_workers`, modulo overhead.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| Results out of order | Impossible by construction — you collect by id in the order you submitted. If the *values* are wrong, dig into the worker. |
| Wall clock ≈ `N × work_time` (no parallelism) | Only one worker is consuming. You need ≥ 2 workers on the same queue and the same plane. |
| Some invocations sit `queued` forever | Workers died mid-run — check their stderr. `q.count("queued")` shows the backlog on the local plane. |
| `DispatchError` partway through | A failing invocation raised at its `q.result` call — the message carries the worker error. |

## Status

Manual.

## Next

→ [07 · SSH provider smoke](/cli/happy-paths/07-ssh-provider-smoke.md)

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths/07-ssh-provider-smoke

# 07 · SSH provider smoke

Tests the provider abstraction directly: run a script on a remote host
through Lakeshore's SSH launcher, without involving a daemon.

## Exercises

- `lakeshore ssh upload` (or `providers add --launcher SSH`) — provider
  config write.
- `lakeshore providers test` — the provider invocation path, SSH
  transport, and remote command execution. SSH providers dispatch
  `direct` by default, so the CLI drives the connection from your
  machine.
- Secret resolution at launch time, when the key was uploaded with
  `--with-key`.

## Requires

- An SSH-reachable host: `ssh <alias>` already works from your laptop,
  and the alias is a `Host` entry in `~/.ssh/config`.
- Python 3 on that host — the built-in smoke script is a Python file.

## Run

Register the host as a provider. The ergonomic path reads
`~/.ssh/config` for you:

```bash
lakeshore ssh discover                       # confirm the alias is parsed
lakeshore ssh upload my-box --with-key       # provider name defaults to the alias
```

`--with-key` uploads the parsed `IdentityFile` as an `ssh_key` secret
(named `ssh-<alias>` unless you pass `--key-secret`) and references it
from the provider. Without it, the provider records the local pem path
instead and only works from this machine.

By hand, if the host isn't in `~/.ssh/config`:

```bash
lakeshore secrets add my-key --kind ssh_key --from-file ~/.ssh/id_ed25519

lakeshore providers add my-box \
  --launcher SSH \
  --kwarg ip=<hostname-or-ip> \
  --kwarg username=<username> \
  --kwarg port=22 \
  --kwarg 'ssh_key.$secret=my-key'
```

Confirm it landed, then smoke-test it:

```bash
lakeshore providers show my-box
lakeshore providers test my-box
```

## Expected output

`providers test` with no `--script` runs the bundled stdlib-only
`hello.py`, so the remote stdout looks like:

```text
hostname: my-box
python: 3.12.3
platform: Linux-6.8.0-x86_64-with-glibc2.39
gpu: (torch not installed)
ok
```

The command exits non-zero on failure. Swap in your own script with
`--script`, and raise `--timeout` (default 600 seconds) for anything
slow.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `ssh: connect to host … port 22: Connection refused` | Wrong host or port. Run `ssh -v <alias>` first to verify. |
| `Permission denied (publickey)` | Wrong key secret, or the key isn't authorized on the host. |
| `no host '<alias>' in ~/.ssh/config` | `ssh upload` / `ssh probe` only know aliases from that file — use `providers add` by hand instead. |
| `python3: command not found` | The smoke payload is Python. Install Python 3 on the host or pass a `--script` that isn't Python-dependent. |
| Hangs with no output | The host accepted the SSH but the command stalled — often a prompt (sudo, host-key confirmation). `lakeshore ssh probe <alias>` isolates the connection step. |

## Status

Manual.

## Next

→ [08 · Bootstrap a daemon on a remote host](/admin/happy-paths/08-bootstrap-daemon-remote.md)

---

Source: https://lakeshore.dreamlake.ai/nymph/telemetry

# Telemetry and introspection

The nymph daemon emits structured logs and — when you enable it — exposes
a **loopback-only** HTTP server that surfaces its current status and a
buffer of recent log events. [`nymph tui`](/nymph/status-tui.md) is the
primary consumer; `curl` works just as well.

Nothing here leaves the host. The server is unauthenticated **because**
it refuses to bind anywhere but loopback.

## Structured logging

Configured under `[log]` in `udf-daemon.toml`:

```toml file="udf-daemon.toml"
[log]
level = "info"          # error | warn | info | debug | trace
format = "human"        # human | json
event_buffer = 1024     # recent-events ring capacity; 0 disables it
```

| Key            | Type    | Default   | Meaning                                                                                     |
| -------------- | ------- | --------- | --------------------------------------------------------------------------------------------- |
| `level`        | string  | `"info"`  | Minimum level to emit.                                                                       |
| `format`       | string  | `"human"` | `human` (pretty) or `json` (one object per line).                                            |
| `event_buffer` | integer | `1024`    | Capacity of the ring buffer `/events` serves. `0` disables it — the layer still installs but drops everything. |

### Environment overrides

| Env var                | Overrides    | Notes                                                                                                  |
| ---------------------- | ------------ | -------------------------------------------------------------------------------------------------------- |
| `RUST_LOG`             | `log.level`  | Full `tracing` `EnvFilter` syntax, e.g. `RUST_LOG=nymph::drain=debug,hyper=warn`. Highest precedence.  |
| `LAKESHORE_LOG_FORMAT` | `log.format` | `human` or `json`, case-insensitive. Highest precedence.                                                |

An unrecognised `format` value falls back to human output and prints
`nymph: unknown log.format <value>; falling back to "human"` to stderr.

With `format = "json"` the daemon installs `tracing_subscriber`'s JSON
layer, so each line is one object carrying `timestamp`, `level`,
`target`, and a `fields` object (the log message itself lands inside
`fields` as `message`). Point a log shipper at stdout.

The same events are mirrored into the ring buffer the introspection
server reads, using the slightly different `RecordedEvent` shape
described [below](#get-events).

## The introspection server

Off by default. Enable it under `[introspect]`:

```toml file="udf-daemon.toml"
[introspect]
enabled = true          # default false
bind = "127.0.0.1"      # loopback only
port = 9876             # default; 0 = OS-assigned ephemeral
```

| Key       | Type    | Default       | Meaning                                                                     |
| --------- | ------- | ------------- | ----------------------------------------------------------------------------- |
| `enabled` | boolean | `false`       | Master switch. TOML is the **only** way to turn this on.                      |
| `bind`    | string  | `"127.0.0.1"` | Interface to bind. A non-loopback address is refused.                         |
| `port`    | integer | `9876`        | TCP port. `0` lets the OS pick — useful in tests.                             |

The default endpoint is `http://127.0.0.1:9876`.

> **Warning:** A comment in `nymph/src/config.rs` advertises a `LAKESHORE_INTROSPECT`
> environment variable. Nothing in the codebase reads it. Set
> `[introspect] enabled = true` in the TOML — there is no env-var shortcut.

Two behaviours worth knowing:

- **Startup order.** The server binds *before* `/hello`, so a daemon that
  cannot reach the control plane is still inspectable.
- **Bind failure is not fatal.** A refused or unavailable address logs a
  warning and the daemon continues without introspection. A non-loopback
  `bind` (e.g. `0.0.0.0`) is an error at that point — introspection is
  skipped, the daemon runs on.

> **Warning:** The server has no authentication. That is safe only because it will not
> bind a routable interface. Do not put a reverse proxy in front of it.

The implementation is a hand-rolled HTTP/1.1 server on a
`tokio::net::TcpListener` — no keep-alive, explicit `Content-Length`,
`Connection: close`. Every response is JSON.

| Request                | Response                                            |
| ---------------------- | ---------------------------------------------------- |
| `GET /status`          | `200` — the `StatusSnapshot` object.                 |
| `GET /events`          | `200` — `{ count, capacity, events[] }`.             |
| `GET /` or `/healthz`  | `200` — `{"ok":true}`.                               |
| Any other path         | `404` — `{"error":"not_found","message":"no such route"}`. |
| Any non-`GET` method   | `405` — `{"error":"method_not_allowed","message":"only GET is supported"}`. |

### `GET /status`

```json file="status.json"
{
  "version": "0.1.4",
  "worker_id": "65f…",
  "queue": "default",
  "run_state": "polling",
  "backoff_s": 0.0,
  "last_poll_ok_ts": 1750000000.5,
  "uptime_s": 42.7,
  "inflight": { "count": 1, "last_exec_id": "01J…" }
}
```

| Field             | Type           | Meaning                                                                                       |
| ----------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| `version`         | string         | Daemon binary version (`CARGO_PKG_VERSION`).                                                   |
| `worker_id`       | string \| null | Worker id in use — server-issued when available, else the local `{machine_id}-{ulid}`. Null before hello. |
| `queue`           | string \| null | Queue this daemon subscribes to.                                                               |
| `run_state`       | string         | `starting` \| `polling` \| `backoff` \| `hibernating` \| `draining`.                           |
| `backoff_s`       | number         | Current poll backoff. `0.0` in steady state; grows on repeated failures.                       |
| `last_poll_ok_ts` | number \| null | Unix seconds of the last **successful** poll.                                                  |
| `uptime_s`        | number         | Seconds since daemon boot.                                                                     |
| `inflight`        | object         | `{ count, last_exec_id }`. `count` covers invocations **and** execs; `last_exec_id` is a best-effort hint, not a list. |

These field names are the `nymph tui` contract — treat them as stable.

### `GET /events`

The ring buffer, oldest → newest:

```json file="events.json"
{
  "count": 2,
  "capacity": 1024,
  "events": [
    { "ts": 1750000000.1, "level": "INFO", "target": "nymph::drain",
      "message": "first event", "fields": {} },
    { "ts": 1750000000.2, "level": "WARN", "target": "nymph::drain",
      "message": "poll failed", "fields": { "reason": "connection timeout" } }
  ]
}
```

| Query param | Default       | Meaning                                                                                                   |
| ----------- | ------------- | ----------------------------------------------------------------------------------------------------------- |
| `limit=N`   | `200`         | Return at most the N most-recent events after filtering.                                                   |
| `level=L`   | *(no filter)* | Only events at or above severity L: `error`, `warn`/`warning`, `info`, `debug`, `trace` (case-insensitive). An unrecognised value skips the filter rather than erroring. |

Level ranks are `ERROR=1`, `WARN=2`, `INFO=3`, `DEBUG=4`, `TRACE=5`, and
the filter keeps everything with a rank at or below the requested one.

Each entry is a `RecordedEvent`: `ts` (unix seconds, float, sub-second
resolution), `level` (uppercased), `target` (the tracing target / module
path), `message` (empty when the event carried only structured fields),
and `fields` (a string → string map — every value is stringified, ordered
for stable output). `count` is the number returned *after* filtering;
`capacity` is `log.event_buffer`.

The buffer is a `VecDeque` behind a mutex; the oldest entry is evicted
once it is full.

## Read next

- [`nymph tui`](/nymph/status-tui.md) — the live monitor that renders
  `/status` and `/events`.
- [Daemon configuration](/api/configuration.md) — the full
  `udf-daemon.toml` schema.
- [Daemon protocol](/nymph/protocol.md) — the control-plane channel, which
  is entirely separate from this local server.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f4-async

# F4 — async

The async row of the [execution matrix](/python-sdk/architecture/execution-matrix.md).
Write the body with `async def` and `@dls.udf` does the rest — the decorator
inspects the function once at decoration, so a plain call returns a
coroutine and `await f(x)` is the whole calling convention. No mode flag, no
second decorator.

On the queue side, `dls.Queue` is the async surface. Note the naming is
inverted from most SDKs: the **unprefixed** name is the async one, and
`dls.SyncQueue` is the blocking one. `Queue` is not a subclass of
`SyncQueue` — it wraps one and exposes it through the `.sync` property.

## L0 — an async body

A plain call returns a coroutine; nothing runs until you await it.

```python
import asyncio

import dreamlake.lakeshore as dls

@dls.udf
async def embed(x: float) -> float:
    await asyncio.sleep(0.01)   # real async I/O goes here
    return x * 2.0

async def main() -> None:
    coro = embed(21.0)          # plain call -> coroutine; nothing ran yet
    print(await coro)           # 42.0

asyncio.run(main())
```

This is the local tier — same process, same event loop, with the run-context
fork and the return-your-keys check installed around the `await` exactly as
they are around a sync call.

## L1 — concurrent fan-out with `asyncio.gather`

Local async UDF calls are ordinary awaitables, so the stdlib's
`asyncio.gather` is the fan-out primitive. Five 50 ms calls take 50 ms, not
250.

```python
import asyncio
import time

import dreamlake.lakeshore as dls

@dls.udf
async def score(x: int) -> int:
    await asyncio.sleep(0.05)
    return x * x

async def main() -> None:
    t0 = time.perf_counter()
    vals = await asyncio.gather(*(score(x) for x in range(5)))
    print(vals, f"{time.perf_counter() - t0:.2f}s")   # [0, 1, 4, 9, 16] 0.05s

asyncio.run(main())
```

That is `asyncio.gather`, from the standard library. There is no
`dls.gather` — the SDK has no gather, no `as_completed`, and no Future type
at any level.

## L1 — the async queue surface

`dls.Queue` mirrors `SyncQueue`'s producer verbs — `submit`, `result`,
`call`, `stream`, `cancel`, `count` — plus the low-level worker verbs
`claim`, `ack`, and `nack`. Each is `await`-able and internally offloaded
with `asyncio.to_thread`, so a slow `result` poll never blocks the loop.
`topic()` stays synchronous and returns a `Topic`. The blocking-only
helpers — `working()`, `post()`, and `len(q)` — live on `SyncQueue` and are
reachable through `q.sync`.

The handle is still a durable string id, never a Future.

```python
import asyncio

import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")

@dls.udf(queue="gpu")
async def embed(x: float) -> float:
    await asyncio.sleep(0)
    return x * 2.0

async def main() -> None:
    q = dls.Queue("gpu", dispatch=plane)          # async surface, same verbs
    inv = embed.submit(21.0, _dispatch=plane)     # durable string id
    await asyncio.to_thread(dls.run_worker, q.sync, once=True)
    print(await q.result(inv))                    # 42.0

    val = await embed(21.0, _dispatch=plane)      # submit + wait, loop stays live
    print(val)

asyncio.run(main())
```

A queue-bound async UDF can be awaited directly, as the last two lines show:
the plain call routes to `.remote()`, which submits and then awaits the
result on a worker thread. Note that this path resolves with the 30-second
default and raises `TimeoutError` past it — for longer jobs, submit and
`await q.result(inv, timeout=…)` explicitly.

`async for frame in q.stream(inv)` works on this surface too, but a plain
async body posts no journal frames — the stream ends cleanly with zero
values once the invocation reaches a terminal state. Frames come from
generator and async-generator bodies; streaming is the
[F5 row](/python-sdk/architecture/f5-async-generators.md).

## L2 — an await chain

A fixed pipeline is just sequential awaits; each stage's resolved value
feeds the next.

```python
import asyncio

import dreamlake.lakeshore as dls

@dls.udf
async def fetch(scene: str) -> dict:
    await asyncio.sleep(0.01)
    return {"scene": scene, "frames": 12}

@dls.udf
async def calibrate(raw: dict) -> dict:
    await asyncio.sleep(0.01)
    return {**raw, "calibrated": True}

@dls.udf
async def summarize(doc: dict) -> str:
    await asyncio.sleep(0.01)
    return f"{doc['scene']}: {doc['frames']} frames, calibrated={doc['calibrated']}"

async def main() -> None:
    raw = await fetch("scene7")
    doc = await calibrate(raw)
    print(await summarize(doc))   # scene7: 12 frames, calibrated=True

asyncio.run(main())
```

## L3 — dynamic spawn off resolved values

The graph's shape is discovered while running: a probe resolves, its value
picks the fan-out width, `asyncio.create_task` spawns the wave, and
`asyncio.as_completed` consumes it in completion order. A second wave spawns
off the first wave's values — ordinary Python control flow is the DAG
language.

```python
import asyncio

import dreamlake.lakeshore as dls

@dls.udf
async def probe(scene: str) -> list[int]:
    await asyncio.sleep(0.01)
    return [3, 1, 4, 1, 5]          # width discovered at run time

@dls.udf
async def refine(seed: int) -> float:
    await asyncio.sleep(0.01 * seed)
    return seed * 1.5

async def main() -> None:
    seeds = await probe("scene7")                # resolved value picks the width
    tasks = [asyncio.create_task(refine(s)) for s in seeds]

    first_wave = []
    for fut in asyncio.as_completed(tasks):      # completion order, not submit order
        first_wave.append(await fut)

    # second wave, spawned off the first wave's values
    rework = [asyncio.create_task(refine(int(v))) for v in first_wave if v > 4.0]
    print(sorted(first_wave), [await t for t in rework])

asyncio.run(main())
```

Both `create_task` and `as_completed` here are stdlib asyncio over local
coroutines. Nothing crosses a queue, so nothing is durable; the moment you
want the wave to survive the process, the handles become ids and the row is
[F2](/python-sdk/architecture/f2-ids.md).

## What is awkward today

- **There is no async worker verb.** `dls.run_worker` takes the *sync*
  queue, so draining from async code means reaching through the async
  surface for its blocking sibling and offloading by hand:
  `await asyncio.to_thread(dls.run_worker, q.sync, once=True)`. The `.sync`
  property makes it a one-liner, but the composition is the caller's job. An
  `arun_worker(q)` — or `run_worker` accepting either sibling — would close
  the seam.
- **Waiting for a worker to pick up a job is a poll loop.** `once=True`
  returns 0 when the queue is empty, so "drain exactly one job, whenever it
  arrives" from async code is a retry loop around `to_thread`. There is no
  awaitable "a job was executed" signal.
- **The async offload is thread-per-verb.** Every `Queue` verb rides
  `asyncio.to_thread` over the blocking core — fine at notebook scale, but a
  native-async transport is what would make it honest at high concurrency.
  The `Queue` class docstring reserves exactly that: an httpx-native async
  transport slotting in behind this same surface when the remote control
  plane wire lands. It is not built.

All four body kinds under the one decorator — including the async pair —
run side by side in `examples/03_body_kinds.py` in the dreamlake-lakeshore
repo (`python examples/03_body_kinds.py`).

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths/08-bootstrap-daemon-remote

# 08 · Bootstrap a daemon on a remote host

The "turn a vanilla box into a worker" flow. One command mints a token,
SSHes in, drops the `nymph` binary and its config, starts it, and waits
for the `Worker` row to appear.

## Exercises

- `lakeshore daemon install <ssh-alias>` — the four-step bootstrap.
- Per-daemon token minting through `POST /v1/namespaces/:ns/tokens`.
- Target-triple detection on the remote host.
- Binary download from Cloudflare R2 (`latest/`, no checksum step — the
  `.sha256` sidecars are used by `scripts/install.sh` and the OTA path,
  not by this bootstrap).
- Outbound-only registration via `POST /v1/daemon/hello`.

## Requires

- Everything in [07 · SSH provider smoke](/cli/happy-paths/07-ssh-provider-smoke.md).
- An entry in `~/.ssh/config` — `daemon install` takes an **SSH alias**,
  not a provider name, and hands it straight to `ssh`.
- `LAKESHORE_ADMIN_TOKEN` exported (or `--admin-token`), because the
  mint route is admin-gated.
- A writable home directory on the remote plus `curl` on `PATH`.
- Outbound HTTPS from the host to the control plane and to R2.

## Verifies

A fresh box becomes a registered Lakeshore worker through one command,
with no inbound port opened on the host.

## Run

```bash
export LAKESHORE_URL=http://localhost:8080
export LAKESHORE_ADMIN_TOKEN=…

lakeshore daemon install my-box --tag gpu --runner process
lakeshore daemon list
```

Useful flags:

| Flag | Effect |
| ---- | ------ |
| `--label <name>` | Worker label prefix. Defaults to the alias; a launch ULID is appended. |
| `--tag <name>` | Repeatable capability tag. |
| `--runner <name>` | Repeatable; defaults to `process`. |
| `--keep-alive-s <n>` | `-1` pool mode (the default here), `0` single job, `N` seconds idle. |
| `--binary <path>` | rsync a local nymph build instead of curling R2. Mutually exclusive with `--nymph-base-url`. |
| `--slurm` / `--slurm-partition <p>` | Wrap the launch in an `sbatch` heredoc instead of `setsid nohup`. |
| `--no-wait` | Skip the registration poll. |

## What it does on the host

```text
~/.local/share/nymph/launches/<launchId>/
  nymph             executable
  udf-daemon.toml   config
  token             minted per-daemon token
~/.local/share/nymph-<launchId>.log   stdout + stderr
```

Everything lives under `$HOME` and runs as whoever you SSH'd in as — the
script never sudos into another uid. It does make one best-effort
`sudo -n usermod -aG docker` attempt so the daemon can reach the Docker
socket, and ignores the failure when there is no sudo or no docker group.

The daemon is started detached with `setsid nohup …` in a background subshell (or an
`sbatch` heredoc under `--slurm`). Target-triple detection maps
`uname -s` to `unknown-linux-gnu` / `apple-darwin` and `uname -m` to
`x86_64` / `aarch64`, then curls
`<base>/dreamlake/nymph/latest/nymph-<triple>`. Any other OS or arch
exits with an error.

## Expected output

```text
→ [1/4] minting per-daemon token from http://localhost:8080…
  ✓ token minted (name 'daemon-my-box-<launchId>')
  launch ID: <launchId>
  log: ~/.local/share/nymph-<launchId>.log
→ [2/4] ssh my-box — bootstrapping (binary + config + start)…
  ✓ [3/4] daemon process started on my-box (label 'my-box-<launchId>')
→ [4/4] waiting (up to 60s) for registration ···
  ✓ daemon registered (worker 6712a…, label 'my-box-<launchId>')
```

The registration poll caps at 60 seconds and then fails; the daemon may
still come up afterwards, so check `daemon list` before re-running.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `daemon install needs LAKESHORE_ADMIN_TOKEN` | Exit 2 before anything happens. Export it or pass `--admin-token`. |
| `ssh my-box exited with code …` | The alias does not resolve, or the remote shell rejected the heredoc. Try `ssh my-box true` on its own first. |
| `curl: command not found` on the host | Install `curl`, or use `--binary` to rsync a local build instead. |
| `unsupported OS` / `unsupported arch` | Only linux and macOS on x86_64 / aarch64 have published binaries. |
| Process started but never registers | Outbound HTTPS blocked, or the control-plane URL baked into the config is unreachable from the host. SSH in and read `~/.local/share/nymph-<launchId>.log`. |
| Registers, then vanishes minutes later | `keep_alive_s`. `-1` is pool mode; `0` exits the moment it goes idle; `N` exits after N idle seconds. |

## Status

Manual.

## Next

→ [09 · Storage round-trip](/cli/happy-paths/09-storage-round-trip.md)

---

Source: https://lakeshore.dreamlake.ai/nymph/status-tui

# `nymph tui`

A live, read-only status monitor for **one** local daemon. It renders a
status header and a scrolling, level-filterable event log by polling the
daemon's loopback [introspection endpoint](/nymph/telemetry.md).

```bash
nymph tui
nymph tui --url http://127.0.0.1:9876 --interval 1
```

`tui` is the only subcommand the `nymph` binary has — running `nymph`
with no subcommand runs the daemon itself.

This is a different tool from [`lakeshore tui`](/cli/tui.md): that one is a
control-plane dashboard over **all** your workers and can act on them.
`nymph tui` watches a **single daemon on this host** and is strictly
read-only — it does not even install the daemon's tracing subscriber. It
is the tool you run next to a daemon you are debugging.

## Prerequisites

The daemon's introspection server must be on — `[introspect] enabled =
true` in `udf-daemon.toml`. See
[Telemetry and introspection](/nymph/telemetry.md). When the daemon is
unreachable the monitor renders a waiting banner and keeps retrying on
the interval; it never panics.

## Flags

| Flag                   | Default             | Meaning                                                                                                          |
| ---------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `--url <url>`          | derived from config | Introspection base URL. When omitted, derived as `http://<[introspect] bind>:<port>`, falling back to `http://127.0.0.1:9876`. |
| `--config <path>`      | config precedence   | Path to `udf-daemon.toml`, used **only** to derive the default `--url`. Same precedence as the daemon; also honours `LAKESHORED_CONFIG`. |
| `--interval <seconds>` | `2.0`               | Poll interval, clamped to a minimum of `0.1`. Each tick fetches `/status` and `/events` together.                  |

A config that fails to load is not fatal here — the TUI falls back to
built-in defaults so it still launches.

## Layout

Three stacked sections.

**1. Status header.** A connection banner plus the daemon's current
state, from [`GET /status`](/nymph/telemetry.md#get-status): `run-state`
(colour-coded), `version`, `queue`, `worker`, `uptime`, `last poll`,
`backoff`, and `in-flight`. The banner reads one of:

```text
● connected
◌ waiting for daemon at <url> (retrying…)
✗ daemon unreachable at <url> — <reason> (retrying every Ns)
```

Status fields keep rendering from the last good snapshot even while
unreachable, so a blip does not blank the context. Before the first
successful poll the header reads *"no status yet — the header populates
on the first successful poll"*.

**2. Events log.** The stream from
[`GET /events`](/nymph/telemetry.md#get-events), one row per event as
`HH:MM:SS  LEVEL  target  message`, with a `[+N fields — enter]` hint
when the selected event carries structured fields. The pane title shows
the active filter, `shown/capacity`, and whether you are in `⏵follow` or
`⏸manual` mode (plus `⏸ PAUSED` when polling is frozen).

Filtering is applied at **render** time against the events already
fetched, so switching levels is instant and never waits for a re-poll.

**3. Footer legend.** Always visible.

## Keybindings

The footer prints them verbatim:

```text
[e]rror [w]arn [i]nfo [d]ebug [t]race  [f]ollow  [space]pause  ↑/↓ scroll  [enter]expand  [q]uit
```

| Key                         | Action                                                                        |
| --------------------------- | ------------------------------------------------------------------------------- |
| `e` / `w` / `i` / `d` / `t` | Set the minimum severity shown. `e` = errors only; `t` = everything. Default `i`. |
| `f`                         | Toggle **follow** — snap to the newest event vs. stay put.                      |
| `Space`                     | Toggle **pause** — freeze polling and the view.                                 |
| `↑` / `↓` (or `k` / `j`)    | Scroll the event log. Any manual scroll drops follow mode; scrolling back to the bottom re-enables it. |
| `Enter`                     | Expand / collapse the selected event's structured fields.                       |
| `q` / `Esc` / `Ctrl-C`      | Quit.                                                                           |

If the log looks empty, you are probably filtered above the events that
exist — drop to `[d]ebug` or `[t]race`.

The terminal is restored on drop, including on panic, so a crash never
leaves your shell in raw mode.

## Read next

- [Telemetry and introspection](/nymph/telemetry.md) — the `/status` and
  `/events` endpoints this monitor renders, and how to enable them.
- [`lakeshore tui`](/cli/tui.md) — the control-plane dashboard over all
  workers. It acts on resources; this monitor only observes one daemon.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — the remote equivalents
  (`lakeshore daemon list --watch`, `daemon show`).

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f5-async-generators

# F5 — async generators

The streaming row of the async column: values arrive one at a time, the
consumer pulls with `async for`, and the event loop stays free between
pulls. F5 is [F3](/python-sdk/architecture/f3-generators.md) with the loop underneath —
same journal at system level, same cursor reconnect — but the consumer can
hold many streams open at once without a thread per stream *in its own
code*. Where the SDK itself still burns a thread per pull is this page's
named gap.

| Level | Spelling |
| --- | --- |
| L0 one call | `async for y in f(x)` — the body is an async iterator |
| L1 systems | `async for f in Queue.stream(inv)` — offloaded sync, not native async |
| L2 static pipeline | async-gen stages composed; `async for` all the way down |
| L3 dynamic graph | `TaskGroup` plus `asyncio.Queue` fan-in — works, hand-rolled, **GAP** |

## L0 — an async-gen body is an async iterator

`@dls.udf` wraps async-generator bodies natively —
`inspect.isasyncgenfunction` detects the kind at decoration, and a plain
call returns an async iterator. Nothing runs until the first `__anext__`;
each step may await real I/O.

```python
import asyncio
import dreamlake.lakeshore as dls

@dls.udf
async def ego_views(traj: str, n: int):
    for i in range(n):
        await asyncio.sleep(0)          # real async I/O goes here
        yield {"traj": traj, "frame": i}

async def main():
    frames = [f async for f in ego_views("scenes/0007", 3)]
    assert [f["frame"] for f in frames] == [0, 1, 2]

asyncio.run(main())
```

Laziness and backpressure work exactly as in F3; the only difference is that
the pull is an `await`, so the consumer can interleave other work between
frames. Yields of string keys are still partial manifests — the
return-your-keys contract concatenates them into one manifest and checks it
when the body finishes, same as the sync row.

## L1 — `Queue.stream`: what it actually is today

`dls.Queue` is the async sibling of `dls.SyncQueue` — a facade holding the
blocking queue as `._q`, with every verb offloaded. `Queue.stream` is a real
async generator with correct semantics, but its engine is the blocking
`SyncQueue.stream` iterator, pulled one item at a time:

```python file="dreamlake/lakeshore/queues.py"
async def stream(self, inv_id, *, since=-1, timeout=30.0):
    it = self._q.stream(inv_id, since=since, timeout=timeout)
    while True:
        item = await asyncio.to_thread(next, it, _SENTINEL)
        if item is _SENTINEL:
            return
        yield item
```

So: **async iterator, yes; native async I/O, no.** Each pull parks one
executor thread inside the blocking journal poll until a frame lands. The
event loop is never blocked — that part of the contract holds — but the
concurrency ceiling is the thread pool, not the loop. The `Queue` class
docstring reserves the fix: an httpx-native async transport slots in behind
this same surface when the remote control plane wire lands. It is not built.

It works today, end to end, including *live* consumption — worker and
consumer overlapping in time:

```python
import asyncio
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

@dls.udf(queue="render")
async def render(n: int):
    for i in range(n):
        await asyncio.sleep(0)
        yield {"frame": i}

async def main():
    plane = Dispatch(":memory:")
    sync_q = dls.SyncQueue("render", dispatch=plane)
    q = dls.Queue("render", dispatch=plane)

    inv = render.submit(4, _dispatch=plane)      # durable string id

    async with asyncio.TaskGroup() as tg:
        tg.create_task(asyncio.to_thread(dls.run_worker, sync_q, once=True))
        frames = [f async for f in q.stream(inv, timeout=10)]

    assert [f["frame"] for f in frames] == [0, 1, 2, 3]

asyncio.run(main())
```

On the worker side the async-gen body is driven under `asyncio.run` and each
yield becomes a VALUE frame on the invocation's journal, with END on return
— identical frames to the sync-generator row. The cursor works unchanged
through the async surface (`since` is the last 0-based seq you saw):

```python
tail = [f async for f in q.stream(inv, since=1, timeout=5)]
assert [f["frame"] for f in tail] == [2, 3]
```

## L2 — async-gen stages chained

Stages compose the way generators always have; the outermost `async for`
pulls through the whole chain, one item at a time, awaiting at every hop. No
stage runs ahead of demand.

```python
import asyncio
from typing import AsyncIterator
import dreamlake.lakeshore as dls

@dls.udf
async def decode(path: str):
    for i in range(4):
        await asyncio.sleep(0)
        yield {"frame": i, "src": path}

@dls.udf
async def embed(frames: AsyncIterator[dict]):
    async for f in frames:
        await asyncio.sleep(0)
        yield {**f, "emb": [f["frame"] * 0.5]}

@dls.udf
async def dedupe(embs: AsyncIterator[dict]):
    seen = set()
    async for e in embs:
        key = tuple(e["emb"])
        if key not in seen:
            seen.add(key)
            yield e

async def main():
    pipeline = dedupe(embed(decode("clips/0001.mp4")))
    out = [x async for x in pipeline]
    assert len(out) == 4

asyncio.run(main())
```

This chaining is **local-tier only**: an async iterator is in-process state
and cannot ride the msgpack wire as an argument. To chain across queues you
put a stream at each stage boundary — stage A yields onto its invocation
journal, stage B consumes `q.stream(inv_a)` and yields onto its own. The
frames compose; the iterators do not.

## L3 — merging a set of streams (GAP)

The dynamic-graph cell: `async for` over several invocation streams at once,
taking frames in whatever order they land. There is no merge primitive in
the SDK — the legacy `as_completed` helper is gone and nothing stream-shaped
replaced it — so today you hand-roll fan-in with a `TaskGroup`, an
`asyncio.Queue`, and a sentinel per stream. Clunky, but it runs:

```python
import asyncio
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

@dls.udf(queue="render")
async def render(scene: str, n: int):
    for i in range(n):
        await asyncio.sleep(0)
        yield {"scene": scene, "frame": i}

async def main():
    plane = Dispatch(":memory:")
    sync_q = dls.SyncQueue("render", dispatch=plane)
    q = dls.Queue("render", dispatch=plane)

    invs = [
        render.submit("scenes/0007", 3, _dispatch=plane),
        render.submit("scenes/0008", 2, _dispatch=plane),
    ]

    DONE = object()
    merged: asyncio.Queue = asyncio.Queue()

    async def pump(inv: str) -> None:
        async for frame in q.stream(inv, timeout=10):
            await merged.put(frame)
        await merged.put(DONE)

    async with asyncio.TaskGroup() as tg:
        for _ in invs:                                   # one drain per job
            tg.create_task(asyncio.to_thread(dls.run_worker, sync_q, once=True))
        for inv in invs:
            tg.create_task(pump(inv))

        open_streams = len(invs)
        got = []
        while open_streams:
            item = await merged.get()
            if item is DONE:
                open_streams -= 1
                continue
            got.append(item)

    assert len(got) == 5

asyncio.run(main())
```

Naming the gap precisely:

- **No fan-in verb.** There is no `merge(*invs)` or `as_completed(invs)`
  over streams, so every caller re-invents the pump-and-sentinel dance above
  — and re-invents error propagation with it. A failed stream raises inside
  its pump task and `TaskGroup` surfaces it, but partial frames already sit
  in the merge queue.
- **A thread per live stream.** Each open `Queue.stream` occupies one
  executor thread inside the blocking journal poll, per the L1 shape above.
  Two streams is fine; hundreds saturate asyncio's default thread pool. The
  honest ceiling today is tens of concurrent streams, and the fix is the
  native-async transport, not user code.
- **No growing-set spelling.** Adding a stream to the merge after it started
  is `tg.create_task(pump(new_inv))` plus manual bookkeeping of
  `open_streams` — workable, but the counting is on you.

What holds even in the clunky version: the merge inputs are invocation ids,
so a crashed merger restarts from persisted `(inv, cursor)` pairs and loses
nothing. The gap is convenience and thread economics, not durability.

## Read next

- [F3 — generators](/python-sdk/architecture/f3-generators.md) — the sync row this column
  mirrors; journal frames and cursors in full.
- [Queue API](/python-sdk/queue.md) — `stream`, `Topic`, and the worker verbs.
- `examples/03_body_kinds.py` in the dreamlake-lakeshore repo — all four
  body kinds, including this row's L0, as one runnable script.
- `examples/05_streams_topics.py` — the journal and cursor mechanics this
  page's L1 rides, zero infra.

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths/09-storage-round-trip

# 09 · Storage round-trip

Proves the control plane can mint working presigned URLs and temporary
credentials against a real bucket through a logical storage handle.

## Exercises

- `lakeshore storage add` — backend registration, and optionally
  `CreateBucket` via `--provision`.
- `lakeshore storage presign` — the control plane's presigned-URL mint
  path, both GET and PUT.
- `lakeshore storage credentials` — the STS `GetSessionToken` path.
- AWS / R2 credential resolution from a Lakeshore secret.

## Requires

- An S3 (or R2 / MinIO) bucket, or permission to create one.
- Credentials for it registered as an `aws_keypair` secret.
- For AWS, the bucket in the region you declare on the storage entry.

> **Warning:** `lakeshore storage` has no `put` / `get` / `ls`. The object surface is
> the presigned URL — use `curl` (or any S3 SDK, with
> `storage credentials`) against it. `add`, `list`, `show`, `update`,
> `remove`, `presign`, and `credentials` are the complete verb set.

## Run

```bash
# 1. Register the credentials, then the storage entry.
#    An aws_keypair secret is either colon-delimited
#    <access_key_id>:<secret_access_key> or JSON {accessKeyId, secretAccessKey}.
printf 'AKIAEXAMPLE:wJalr...' | lakeshore secrets add aws-prod-keys --kind aws_keypair

lakeshore storage add my-bucket \
  --kind s3 \
  --bucket lakeshore-test-bucket \
  --region us-east-1 \
  --creds aws-prod-keys

lakeshore storage show my-bucket

# 2. Upload through a presigned PUT.
echo hi > /tmp/hello.txt
PUT_URL=$(lakeshore storage presign my-bucket hello.txt --put --expires-in 600)
curl -sS -X PUT --upload-file /tmp/hello.txt "$PUT_URL"

# 3. Download through a presigned GET.
curl -sS "$(lakeshore storage presign my-bucket hello.txt)"

# 4. List with temporary credentials.
eval "$(lakeshore storage credentials my-bucket --env)"
aws s3 ls s3://lakeshore-test-bucket/
```

To have the control plane create the bucket for you, add `--provision`
on step 1 — it resolves the `aws_keypair` secret and calls
`CreateBucket`, idempotently if the bucket already exists under that
account.

## Expected output

Step 3 prints `hi`. Step 4 lists `hello.txt`. `storage credentials
--env` emits eval-able `export AWS_...` lines with a default lifetime
of 3600 seconds (range 900–129600).

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `error: unknown command 'put'` / `'get'` / `'ls'` | Those verbs don't exist — see the callout above. |
| `403 Forbidden` from S3 on the presigned PUT | The credential secret is wrong, or the IAM policy lacks `s3:PutObject` / `s3:GetObject`. |
| `301 Moved Permanently` | Region mismatch — the bucket isn't in the region you declared. |
| The storage entry has a surprise config field | `storage add` allows unknown options and folds them into the config. A typo'd flag becomes a config key. Check `storage show`. |
| `storage credentials` errors | The secret isn't an `aws_keypair`, or STS isn't reachable from the control plane. |

## Status

Manual.

## Next

→ [10 · Code mount](/python-sdk/happy-paths/10-code-mount.md)

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f6-scopes

# F6 — scopes

Row F6 of the [execution matrix](/python-sdk/architecture/execution-matrix.md): scoped
contexts. The other rows are about invocation *shape* — this row is about
the **ambient context every call runs inside**. `dls.scope` sets a key
prefix; `dls.run` resolves prefix-relative string keys to local paths. A UDF
body never sees a bucket, a credential, or an absolute path — string keys
in, string keys out, file bytes never on the msgpack wire.

| Level | Shape | Spelling |
| --- | --- | --- |
| L0 | one call | `with dls.scope(p):` plus read, write, return keys |
| L1 | fan-out | each call gets its own key ledger |
| L2 | static pipeline | `with dls.scope(f"scenes/{i:04d}"):` per item |
| L3 | dynamic graph | a scope per data-discovered branch |

> **Warning:** Despite the name, `dls.run` is not an entry point. It is the author-facing
>   proxy onto the ambient run context, and its whole surface is `prefix`,
>   `root`, `read(key)`, `write(key, dir=False)`, `read_all(keys)`, and
>   `write_all(keys, dir=False)`. `read` and `write` return
>   `pathlib.Path` objects. The things that actually run work are
>   `f(...)`, `f.submit(...)`, `f.remote(...)`, `f.local(...)`, the queue
>   verbs, and `dls.run_worker(...)`.

Everything on this page is the local tier — bare `@dls.udf`, no queue, no
worker. Scopes work with zero infrastructure and touch no network.

## L0 — the contract

A data UDF does I/O through `dls.run.read(key)` and `dls.run.write(key)` and
**returns the string keys it wrote**. Keys are prefix-relative POSIX
strings; the ambient context maps them to real locations. In local mode a
key resolves to `root / prefix / key` — `write` hands out that path with
parents created, `read` resolves it and raises if it does not exist.

The root defaults to the current working directory; the outermost
`dls.scope(..., root=...)` of a run re-anchors it.

```python
import tempfile
from pathlib import Path

import dreamlake.lakeshore as dls

root = Path(tempfile.mkdtemp())

@dls.udf
def normalize(raw: str) -> str:
    text = dls.run.read(raw).read_text()
    out = dls.run.write("process/clean.txt")
    out.write_text(text.strip().lower())
    return "process/clean.txt"          # the keys you wrote ARE the result

with dls.scope("scenes/0007", root=root):
    dls.run.write("source/raw.txt").write_text("  Hello Lakeshore  ")
    key = normalize("source/raw.txt")

assert key == "process/clean.txt"
assert (root / "scenes/0007/process/clean.txt").read_text() == "hello lakeshore"
```

`normalize` names nothing outside its scope — the scene number lives in the
ambient prefix, the machine location lives in the root.

The return-your-keys contract is enforced after every call: each key handed
out by `dls.run.write` must appear in the returned manifest. The accepted
manifest shapes are `str`, `list[str]`, `tuple[str, ...]`, and
`dict[str, str]` (named outputs, where the values are the keys). Return the
wrong thing and the error states the fix:

```python
@dls.udf
def sloppy(raw: str) -> str:
    dls.run.write("process/mesh.ply").write_text("x")
    return "done"                       # a string, but not the key
```

```
KeyError_: UDF wrote key(s) it did not return: ['process/mesh.ply'];
include them in the returned manifest (or don't write them)
```

Return nothing at all and the message spells out the accepted shapes:

```
KeyError_: UDF wrote 1 file(s) via dls.run.write (process/mesh.ply) but
did not return their keys; return the key string(s) — str, list[str],
or dict[str, str] — so the outputs are wired into the graph
```

`KeyError_` is a subclass of `ValueError`, not of the builtin `KeyError`.

The check is **one-directional**: a returned string that was never written
is allowed, because it may be plain data or an upstream key passed through
by a selector, and the two are indistinguishable by inspection. A silent
write is not allowed — lineage and downstream wiring key off the returned
manifest, so an unreturned key is an invisible artifact. (A typo'd return
still gets caught, from the other side: the correctly written key goes
unreturned and fails the check.)

## L1 — a ledger per call

Every `@dls.udf` call runs in a **fresh child context**: same root, prefix,
storage, and ids as the ambient scope, but its own write ledger. The unit of
the return-your-keys check is one call, so sequential calls under one scope
never cross-contaminate.

```python
@dls.udf
def stamp(name: str) -> str:
    dls.run.write(f"out/{name}.txt").write_text(name)
    return f"out/{name}.txt"

with dls.scope("batch", root=root) as ctx:
    keys = [stamp(n) for n in ["a", "b", "c"]]

assert keys == ["out/a.txt", "out/b.txt", "out/c.txt"]
assert ctx.writes == {}                 # each call kept its own ledger
```

If the ledger were shared, the second call would fail its check for not
returning the first call's key. It is not — `stamp("b")` answers only for
`out/b.txt`. The scope's own context stays clean; it accumulates nothing
from the calls inside it.

Generator bodies follow the same contract stretched over time: each `yield`
is a partial manifest, and together the yields are THE manifest, checked
once when the body finishes.

```python
@dls.udf
def shards(n: int):
    for i in range(n):
        dls.run.write(f"shards/{i:02d}.txt").write_text(str(i))
        yield f"shards/{i:02d}.txt"     # partial manifests concatenate

with dls.scope("gen", root=root):
    keys = list(shards(3))

assert keys == ["shards/00.txt", "shards/01.txt", "shards/02.txt"]
```

Yields that are not key-shaped concatenate to nothing, so a generator that
streams plain data has no manifest. That is fine as long as it also wrote no
files — the check only fires when the ledger is non-empty. A body that both
writes files and yields plain data fails, with the same "did not return
their keys" message.

## L2 — scope per scene

Nesting joins prefixes. A pipeline scopes the run once at the top and each
item once inside the loop; the stage chain in the middle is written entirely
in relative keys.

```python
import json

@dls.udf
def simulate(params: str) -> str:
    seed = json.loads(dls.run.read(params).read_text())["seed"]
    frames = dls.run.write("process/frames.txt")
    frames.write_text("\n".join(f"frame-{seed}-{t}" for t in range(3)))
    return "process/frames.txt"

@dls.udf
def encode(frames: str) -> str:
    n = len(dls.run.read(frames).read_text().splitlines())
    clip = dls.run.write("outputs/clip.txt")
    clip.write_text(f"clip[{n} frames]")
    return "outputs/clip.txt"

with dls.scope("dataset-v2", root=root):
    for i in range(3):
        with dls.scope(f"scenes/{i:04d}"):
            dls.run.write("source/params.json").write_text(json.dumps({"seed": i}))
            frames = simulate("source/params.json")
            clip = encode(frames)

assert (root / "dataset-v2/scenes/0002/outputs/clip.txt").read_text() == "clip[3 frames]"
```

Inside the loop, `dls.run.prefix` is `dataset-v2/scenes/0002` — the inner
scope joined onto the outer. Neither UDF nor the stage chain mentions a
scene number or a directory. Move the run by changing `root=`, re-home it in
a dataset by changing the outer scope string, and every key still reads the
same.

For a stage that takes or produces several keys at once, `dls.run.read_all`
and `dls.run.write_all` mirror the shape you hand them: a `str` maps to a
`Path`, a list to a list, a dict to a dict.

## L3 — scope per branch

When the graph's width is discovered at run time, open a scope per branch.
Each branch gets its own namespace, and keys inside stay short and identical
across branches.

A branch often needs an input from its *parent* scope. Keys may climb with
`../` — resolution joins prefix and key, so `../` climbs the scope, and the
result is checked against the run root: a key may climb out of its prefix,
never out of the run. Absolute keys are rejected outright.

```python
import posixpath

@dls.udf
def detect(frame: str) -> list[str]:
    labels = dls.run.read(frame).read_text().split()
    keys = []
    for j, label in enumerate(labels):
        dls.run.write(f"crops/{j:02d}.txt").write_text(label)
        keys.append(f"crops/{j:02d}.txt")
    return keys

@dls.udf
def refine(crop: str) -> str:
    label = dls.run.read(crop).read_text()
    dls.run.write("refined.txt").write_text(label.upper())
    return "refined.txt"

manifest = []
with dls.scope("scan-17", root=root):
    dls.run.write("source/frame.txt").write_text("cat dog heron")
    crops = detect("source/frame.txt")      # width decided by the data
    for j, crop in enumerate(crops):
        with dls.scope(f"objects/{j:02d}"):
            key = refine(f"../../{crop}")   # climb to the parent scope
            manifest.append(posixpath.join(dls.run.prefix, key))

assert manifest == [
    "scan-17/objects/00/refined.txt",
    "scan-17/objects/01/refined.txt",
    "scan-17/objects/02/refined.txt",
]
assert (root / manifest[2]).read_text() == "HERON"
```

Three labels in the frame, three branch scopes — the structure of the key
space mirrors the structure the data dictated. `refine` is the same function
in every branch; only the ambient prefix differs. The manifest is
scope-qualified by joining `dls.run.prefix` onto each returned key — plain
strings, ready for a journal line.

Climbing past the root is refused:

```
KeyError_: key '../../outside' escapes the run root from prefix
"scan-17"; ../ may climb to a parent scope, not out of the run
```

## Why an ambient context

The alternative is threading a path or a context argument through every
call, at which point UDF signatures grow a parameter that is not data and
every caller becomes responsible for plumbing. With scopes, the *pattern*
rows (F1 through F5) stay pure: a sync call is a call, a generator is a
generator, and where the bytes land is decided entirely by the `with` blocks
around them. The same body runs under a temp directory in a test, under
`scenes/0042` in a pipeline, and under a bound storage on a worker —
unchanged.

---

Source: https://lakeshore.dreamlake.ai/api/auth-and-secrets

# Auth and secrets

Two related concerns. **Auth** is how a caller proves itself to the
control plane. **Secrets** are server-stored credential material the
control plane decrypts in memory when it launches hosts on your behalf.

Both share the same dev ergonomic: in production the relevant env var is
required and the server refuses to boot without it; in dev, omitting it
degrades to an insecure fallback with a loud one-time warning.

## Credential types

Four distinct bearer credentials. Do not mix them up.

| Credential | Format | Who holds it | What it grants |
| --- | --- | --- | --- |
| Admin token | operator-chosen, from `LAKESHORE_ADMIN_TOKEN` on the server | the operator | Everything, in every namespace. Required to mint the other two token types. |
| Namespace token | `dlk_<base64url-32-bytes>` | CLI, SDK, dashboard | CRUD in its own namespace (plus org siblings). |
| Enroll token | `dle_<base64url-32-bytes>` | a daemon, once | One-shot daemon enrollment on `/v1/daemon/hello`. |
| Daemon signature | Ed25519 over `X-LS-*` headers | a daemon, ongoing | Poll / ack / event. Covered on [Key rotation](/nymph/key-rotation.md). |

The `dlk_` / `dle_` prefixes are greppable so an accidental paste is
auditable, and they make the two token types impossible to confuse in a
log.

## Where the auth hook fires

The `onRequest` auth hook matches exactly two path shapes:

- `/v1/admin/*` — admin token or open mode only. Anything else gets `403`.
- `/v1/namespaces/:ns/*` — the tiered check below.

**Every other route is outside the hook**, including all of
`/v1/daemon/*` and `/v1/producer/*`. Those either carry their own
in-route verification (daemon signatures) or are authenticated by an
unguessable id in the URL.

One explicit exemption inside the namespaced space:
`/v1/namespaces/:ns/workers/:id/exec/:exec_id/result` bypasses namespace
auth entirely — it is a daemon callback authenticated by the `exec_id`.

### The tiered check

1. **Open mode.** When `LAKESHORE_ADMIN_TOKEN` is unset, every request
   passes with `authMode = "open"`.
2. **Missing bearer** → `401 missing bearer token`.
3. **Admin.** Bearer equals `LAKESHORE_ADMIN_TOKEN` (constant-time
   compare) → `authMode = "admin"`, full access to any namespace.
4. **Namespace token.** Look up a `Token` row by `hash =
   sha256(bearer)`. Unknown → `401 invalid token`. Revoked → `403 token
   revoked`. Namespace mismatch → `403`, *unless* both namespaces share a
   non-null `orgId`. On success `authMode = "token"` and `lastUsedAt` is
   bumped asynchronously.

`authMode` is one of `admin`, `token`, `open`, `daemon-sig`,
`daemon-bearer` — note the hyphens. `whoami` only ever reports the first
three.

Only four routes require the admin tier beyond the hook itself: minting
a namespace token, minting an enroll token, deleting an enroll token, and
uploading a nymph release. `requireAdmin()` returns true for `authMode`
`admin` **or** `open`.

## Server env config

```bash file="lakeshore-controlplane/.env"
# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 24 | tr '+/' '-_' | tr -d '='
LAKESHORE_ADMIN_TOKEN=""

# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 32
SECRETS_KEY=""
```

- **Production** (`NODE_ENV=production`) — the server logs a fatal and
  exits when either is missing.
- **Dev** — leaving `LAKESHORE_ADMIN_TOKEN` blank enters **open mode**
  with one loud warn at startup. Leaving `SECRETS_KEY` blank falls back
  to a hard-coded `DEV_FALLBACK_KEY`, also with a warn.

> **Note:** The server formerly read `ADMIN_TOKEN` with no prefix. It is still
> accepted, with a one-time stderr deprecation warning. Migrate `.env`
> files and Heroku config vars to `LAKESHORE_ADMIN_TOKEN` before the
> back-compat goes away.

## Client-side env vars

These are easy to conflate — they are read by different programs and are
not interchangeable.

| Var | Read by | Effect |
| --- | --- | --- |
| `LAKESHORE_URL` | every CLI command except `auth`; the Python SDK | Names the control plane. **When set, the CLI sends no bearer at all** — it deliberately bypasses the saved auth file. |
| `LAKESHORE_SERVER` | `lakeshore auth login \| status` only; the `nymph` daemon | Server URL for the auth commands and for the daemon's `--server`. |
| `LAKESHORE_NAMESPACE` | CLI (with `LAKESHORE_URL`), Python SDK | Namespace slug. Defaults to `default`. |
| `LAKESHORE_CLIENT_TOKEN` | the Python SDK's default dispatch | Bearer for `HttpDispatch`. `lakeshore worker` passes the token to its Python child this way — never on argv. |
| `LAKESHORE_ADMIN_TOKEN` | the CP server (validation); the CLI's `admin` group and `auth login` | On the server host, and once in your shell when bootstrapping. |

> **Warning:** `lakeshore worker start` / `worker once` deliberately do not consult
> `auth.yml`. The plane a worker drains must come from `--url` or
> `LAKESHORE_URL`. Running `lakeshore auth login` is not sufficient setup
> for a worker.

## CLI auth commands

```bash
lakeshore auth login [--server <url>] [--namespace <slug>] [--token <plaintext>] [--name <label>]
lakeshore auth status [--server <url>] [--namespace <slug>]
lakeshore auth logout
```

Server URL precedence for these three commands, highest first:
`--server`, `LAKESHORE_SERVER`, the saved auth file, then
`http://localhost:8080`.

`login` resolves a token, validates it by calling
`GET <server>/v1/namespaces/<ns>/whoami` with a Bearer header, and saves
on success. `status` reloads the saved auth and re-validates. `logout`
deletes the file.

Credentials land at `$XDG_CONFIG_HOME/lakeshore/auth.yml` (defaulting to
`~/.config/lakeshore/auth.yml`), written `chmod 600`:

```yaml file="~/.config/lakeshore/auth.yml"
server: https://api.lakeshore.dreamlake.ai
namespace: default
token: dlk_xxxxxxxxxxxx…
```

### First-time bootstrap via the admin token

If `--token` is absent and `LAKESHORE_ADMIN_TOKEN` is set in your shell,
`login` mints a fresh namespace token through the admin path and saves
**that** — not the admin token — to `auth.yml`:

```bash
export LAKESHORE_ADMIN_TOKEN=...          # same value as the server's .env
lakeshore auth login --server http://localhost:8080 --namespace dev --name cli-$USER
lakeshore auth status                     # → authMode: token
```

The label actually stored is `<--name>-<timestamp>`; the suffix keeps a
re-run from colliding with the per-namespace unique-name constraint. With
neither `--token` nor `LAKESHORE_ADMIN_TOKEN`, `login` prompts on a TTY
and fails with exit 2 otherwise.

## `whoami`

`GET /v1/namespaces/:ns/whoami` returns the caller's view. Reaching `200`
is what `auth login` treats as validation.

```json
{
  "namespace": "default",
  "tokenPrefix": "dlk_xxxx",
  "tokenName": "cli-ge-2026-05-18T05-40-00",
  "lastUsedAt": "2026-05-18T05:40:00.000Z",
  "authMode": "token"
}
```

For an admin or open-mode caller there is no backing `Token` row, so the
route returns a synthetic envelope: `tokenPrefix: null` and `tokenName`
of `<admin>` or `<open>`.

## Token CRUD

| Verb | Path | Notes |
| --- | --- | --- |
| `POST` | `/v1/namespaces/:ns/tokens` | Admin only. `201`. Returns plaintext once. |
| `GET` | `/v1/namespaces/:ns/tokens` | Metadata only. |
| `DELETE` | `/v1/namespaces/:ns/tokens/:id` | Soft delete — sets `revokedAt`. |

Mint body is `{ "name": "cli-ge" }`. Labels match
`^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$` (underscores allowed, unlike the
lowercase-and-hyphen regex used for resource names) and are unique per
namespace. The mint response is `{id, name, prefix, plaintext,
createdAt}`.

Only `hash = sha256(plaintext)` (globally unique) and `prefix` (the first
8 characters of the plaintext, for display) are stored. Revocation is a
soft delete so audit listings keep the row; a revoked token returns
`403`.

> **Warning:** The mint response is the only time a token's plaintext leaves the server.
> There is no recovery path — lose it and the remedy is revoke plus mint a
> new one.

The CLI wraps the same routes under `lakeshore admin tokens create |
list | revoke` (alias `ls` for `list`), all of which hard-require
`LAKESHORE_ADMIN_TOKEN` in the environment.

## Organizations

Namespaces carry an optional `orgId`. Namespaces sharing a non-null
`orgId` are treated as siblings.

- **Cross-namespace access**: a token scoped to `alice` (org `acme`) may
  call `/v1/namespaces/bob/...` when `bob` is also in org `acme`. A
  namespace with no `orgId` is reachable only by its own tokens.
- **Org-scoped listing**: `GET /v1/namespaces/:ns/providers` and
  `GET /v1/namespaces/:ns/queues` aggregate rows across every namespace
  in the org. Provider rows carry a `namespace` field so you can tell
  them apart; queue summaries do the same when the org has more than one
  namespace. Provider names must be unique **within the org**, not just
  within the namespace.

There is no `User` or `Organization` model — `orgId` is a plain string on
the `Namespace` row, set at create time or patched later (admin only):

```bash
curl -X POST $CP/v1/admin/namespaces \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "alice", "orgId": "acme"}'

curl -X PATCH $CP/v1/admin/namespaces/alice \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"orgId": "acme"}'
```

Namespace slugs match `^[a-z0-9][a-z0-9-]{0,63}$`. Creating a duplicate
returns `409`.

## Enroll tokens

A one-shot credential a daemon presents as a Bearer on
`POST /v1/daemon/hello` while enrolling its Ed25519 public key.

| Verb | Path | Notes |
| --- | --- | --- |
| `POST` | `/v1/namespaces/:ns/enroll-tokens` | Admin only. Returns `dle_…` plaintext once. |
| `GET` | `/v1/namespaces/:ns/enroll-tokens` | Metadata only. |
| `DELETE` | `/v1/namespaces/:ns/enroll-tokens/:id` | Admin only. |

Stored as `sha256` hex plus an 8-character display prefix and an operator
label (same label regex as namespace tokens). Single-use via
`consumedAt`, time-limited via `expiresAt` — default TTL 3600 s, hard cap
30 days.

The consume is atomic and happens **after** signature verification, so a
daemon that presents a bad self-signature does not burn the token.
Enrollment failures come back as HTTP `200` with a
`{ "reject": { "code", "message" } }` msgpack body, not as a 4xx — codes
include `enroll_token_required`, `invalid_enroll_token`,
`enroll_token_consumed`, `enroll_token_expired`, `key_id_mismatch`, and
`enroll_signature_invalid`. Full flow: [Key rotation](/nymph/key-rotation.md).

## Secrets

Server-stored credentials, AES-256-GCM-encrypted at rest, referenced from
provider kwargs as `{ "$secret": "<name>" }`. **Plaintext only flows
in.** No `GET` route ever returns it.

### Kinds

`kind` tells the consumer how to read the plaintext. Exactly four values
are accepted:

| `kind` | Plaintext shape | Consumer |
| --- | --- | --- |
| `ssh_key` | PEM / OpenSSH private key | SSH and SLURM launchers |
| `aws_keypair` | `<access_key_id>:<secret_access_key>` — colon-delimited, split on the **first** `:` | EC2 launcher, storage provisioning |
| `gcp_sa_json` | Raw service-account JSON | GCE launcher |
| `opaque` | Anything — the consumer decides | Any |

> **Warning:** The `$secret` substitution used by every launcher (`shapeBySecretKind`
> in `launchers/ec2.ts`) reads `aws_keypair` plaintext as
> `<access_key_id>:<secret_access_key>` and throws
> `aws_keypair secret must be formatted as <access_key_id>:<secret_access_key>`
> on a value with no `:`. The storage-provisioning path is the one
> exception: it *also* accepts a JSON object whose keys are
> `accessKeyId` / `access_key_id` / `AccessKeyId` (and the matching
> secret-key spellings), falling back to the colon form. Store the colon
> form and both paths work.

Names match `^[a-z0-9][a-z0-9-]*$` and are unique per namespace — a
duplicate returns `409`. Plaintext is capped at **64 KiB**; a larger body
returns `413`. An unknown `kind` or a malformed name returns `400`.

The row stores `ciphertext`, a 12-byte `nonce`, a 16-byte `tag`, plus
`name` / `kind` / `description` / timestamps. `GET` list and show return
metadata only — never `ciphertext`, `nonce`, or `tag`.

### Encryption at rest

AES-256-GCM via Node's `crypto`. Each encryption generates a fresh
12-byte nonce; the 16-byte auth tag is stored alongside. Decrypt throws
on a tag mismatch — GCM's tamper detection — and callers treat the throw
as "corrupted blob or wrong key" and fail closed.

The master key comes from `SECRETS_KEY`, base64-encoded 32 bytes:

```bash
openssl rand -base64 32
```

A `SECRETS_KEY` that decodes to anything other than exactly 32 bytes
throws rather than silently truncating.

> **Warning:** Rotating `SECRETS_KEY` without re-encrypting orphans every existing row.
> A rotation today means: mint the new key, decrypt and re-encrypt every
> row with it, then switch the env var. Plan the downtime.

### CLI

```bash
lakeshore secrets add <name> --kind <ssh_key|aws_keypair|gcp_sa_json|opaque> \
    [--from-file <path>] [--description <text>]
lakeshore secrets list
lakeshore secrets show <name>
lakeshore secrets rotate <name> [--from-file <path>]
lakeshore secrets remove <name>
```

`--kind` is required on `add`. Plaintext comes from `--from-file <path>`
or from piped stdin when the flag is absent; on a TTY with neither, the
command exits 2 with `use --from-file <path> or pipe plaintext via
stdin`.

```bash
$ printf '%s:%s' "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" \
    | lakeshore secrets add aws-prod-keys --kind aws_keypair
ok  secret "aws-prod-keys" created  (aws_keypair)  2026-05-18T05:42:00.000Z
```

> **Warning:** Both the CLI and the server treat the plaintext as a UTF-8 string. PEM,
> OpenSSH, and JSON credentials round-trip losslessly; arbitrary binary
> does not. There is also no `--from-file -` convention — omit the flag and
> pipe instead.

### `$secret` references

A provider's kwargs may carry secret references inline:

```json
{
  "name": "aws-us-east",
  "launcher": "EC2",
  "kwargs": {
    "region": "us-east-1",
    "aws_credentials": { "$secret": "aws-prod-keys" }
  }
}
```

At create and update time the server walks the kwargs tree with
`findSecretRefs()` (depth-capped at 20; a malformed marker or a depth
overflow is a `400`) and verifies every referenced name exists in the
namespace. A typoed name fails with `422 {"error": "unknown secret",
"secret": "<name>"}` at provider-create time rather than silently at
launch time.

`lakeshore providers show` and `lakeshore modes show` mask references as
`"<secret: name>"` in their YAML preview, so a copy-pasted output cannot
leak a name into the wrong place.

> **Warning:** `readSecretPlain()` is internal-only. It decrypts in memory for the
> launch path and the result is never written to disk, returned over the
> API, or logged. Never call it from an HTTP handler.

## Operator setup

```bash
heroku config:set \
  SECRETS_KEY="$(openssl rand -base64 32)" \
  LAKESHORE_ADMIN_TOKEN="$(openssl rand -base64 24 | tr '+/' '-_' | tr -d '=')" \
  --app <your-app>
```

For a local stack, leave both blank in `lakeshore-controlplane/.env`. The
control plane boots in open mode with the dev fallback key, and the CLI,
SDK, and dashboard all skip auth. Secrets stored under the fallback key
survive restarts but give zero security — a dev database is not
load-bearing.

To exercise the real auth path locally, set both, restart, then bootstrap
with `lakeshore auth login` as shown above.

## See also

- [Configuration](/api/configuration.md) — where `$secret` references live inside `.dreamrc` provider entries.
- [Key rotation](/nymph/key-rotation.md) — Ed25519 daemon identity, enrollment, and rotation.
- [Token lifecycle](/cli/happy-paths/15-token-lifecycle.md) — the mint-login-inherit walkthrough.
- [Auth and setup](/cli/examples/auth.md) — the CLI cookbook.

---

Source: https://lakeshore.dreamlake.ai/get-started/mounts

# Mounts

A **Mount** is a per-namespace registry entry describing a shared
filesystem — an NFS export, an SMB share, an S3 bucket, a host
directory, a Kubernetes ConfigMap — that the runner attaches into a
job's workdir before dispatch. The server stores the kind-specific
config and validates the required fields at write time; everything
beyond that (mount-option syntax, credential enforcement, the actual
`mount -t <kind>` / rclone / aws-cli invocation) belongs to the runner.

> **Warning:** Mount **storage** — add / list / show / update / remove — is live.
> **Activation** — the runner actually mounting NFS / s3fs / bind /
> configmap into the sandbox before user code runs — is a daemon-side
> follow-up. Build the registry today; the wire-up at launch time lands in
> a later release.

For shipping *code* and moving *objects* in and out of jobs, see
[Storages](/get-started/storages.md) and
[Code mounts](/cli/examples/storage-code-mounts.md) — those paths work
end-to-end today. The Mounts registry is for network filesystems the
runner will attach wholesale.

## Kinds

Exactly ten kinds are accepted. The server validates required-field
presence per kind (422 on a miss) and stores the rest as-is.

| Kind | What it is | Required | Optional |
| --- | --- | --- | --- |
| `nfs` | NFS export. | `server`, `path` | `options` |
| `samba` | SMB/CIFS share (runner uses `mount -t cifs`). | `server`, `share` | `options`, `creds` |
| `ftp` | FTP server. | `server` | `path`, `port`, `creds` |
| `sftp` | SFTP over SSH (runner uses sshfs or curlftpfs). | `server` | `path`, `port`, `creds`, `key` |
| `s3` | S3-compatible bucket via a userspace client (aws-cli, boto, `rclone copy`). Objects pulled into the workdir on demand; no FUSE. | `bucket` | `endpoint`, `region`, `prefix`, `creds` |
| `s3fs` | The same bucket, FUSE-mounted with s3fs-fuse — POSIX-y file ops on a real path. | `bucket` | `endpoint`, `region`, `prefix`, `creds`, `options` |
| `google_drive` | A Drive folder (rclone-backed). `folderId` may be `root` for My Drive. | `folderId` | `creds` |
| `dropbox` | A path inside a Dropbox account (rclone-backed). | `path` | `creds` |
| `bind` | Host directory bind-mounted into the container. | `hostPath` | `readOnly` |
| `configmap` | A Kubernetes ConfigMap materialized into files. | `kubeNamespace`, `name` | `items` |

Note the spellings: `google_drive` takes an underscore, and `s3fs` has
no separator.

Three things worth knowing:

- **`s3` vs `s3fs`** — same required fields, different activation. Use
  `s3` for throughput and simple copy-on-read semantics; use `s3fs` when
  the job code expects a real filesystem path.
- **Non-AWS S3** — `endpoint` is the hook for Ceph, MinIO, R2, and
  Backblaze B2. Omit it and the runner assumes AWS S3.
- **Anonymous access** — credential refs are deliberately not required
  on kinds that can mount anonymously (public buckets, public NFS
  exports, anonymous FTP). The runner enforces credentials at activation
  time, where the error can be clearer than "field required".

> **Warning:** Some design notes describe a `git` mount that clones a repo at a
> resolved commit. It is not implemented: neither the control plane nor
> the CLI accepts `--kind git`. For shipping a working tree today, use
> `lakeshore code push` — see
> [Code mounts](/cli/examples/storage-code-mounts.md).

## CLI

```bash
lakeshore mounts add <name> --kind <kind> [--kwarg k=v ...] [--config-file <path>]
lakeshore mounts list [--json]
lakeshore mounts show <name> [--json]
lakeshore mounts update <name> [--kwarg k=v ...] [--config-file <path>]
lakeshore mounts remove <name>
```

The name must match `^[a-z0-9][a-z0-9-]*$`. Reusing a name inside a
namespace is a 409.

Config can come from repeatable `--kwarg key=value` entries (dotted keys
nest), from `--config-file` pointing at YAML or JSON, or from both on
`add` — the file seeds the config and the `--kwarg` entries override.
Values parse like the rest of the CLI: `true` / `false` / `null` and
numbers coerce, values starting with `[`, `{`, or `"` parse as JSON, and
everything else stays a string.

On `update` the two flags are mutually exclusive: `--kwarg` entries
deep-merge into the existing config (`--kwarg foo=null` drops a key),
while `--config-file` replaces the config wholesale.

## Secret references

Credentials belong in [Secrets](/api/auth-and-secrets.md#secrets), not in
mount config. A `{"$secret": "<name>"}` marker anywhere inside the config
resolves through the same plumbing that providers and tunnels use. The
dotted-key form on the CLI is:

```bash
lakeshore mounts add team-cache \
  --kind s3 \
  --kwarg bucket=my-team-cache \
  --kwarg 'creds.$secret=aws-team-keys'
```

Every referenced secret must already exist in the same namespace — the
server rejects the write with a 422 otherwise. A `$secret` marker is a
leaf: the walker does not recurse into sibling keys once it sees one,
and the value must be a non-empty string.

## Worked example

Register an NFS export of a dataset, mark it read-only for the runner,
and inspect it:

```bash
lakeshore mounts add imagenet \
  --kind nfs \
  --kwarg server=10.0.0.12 \
  --kwarg path=/exports/imagenet

# Runner-side mount options ride in `options` — deep-merged in.
lakeshore mounts update imagenet --kwarg options=ro

lakeshore mounts list
lakeshore mounts show imagenet --json
```

```json
{
  "name": "imagenet",
  "kind": "nfs",
  "config": {
    "server": "10.0.0.12",
    "path": "/exports/imagenet",
    "options": "ro"
  }
}
```

Larger configs read better from a file:

```yaml file="app-config.yaml"
kubeNamespace: training
name: app-config
items:
  - key: config.json
    path: config.json
```

```bash
lakeshore mounts add app-config --kind configmap --config-file ./app-config.yaml
lakeshore mounts remove app-config
```

## HTTP

```
POST   /v1/namespaces/:ns/mounts
GET    /v1/namespaces/:ns/mounts
GET    /v1/namespaces/:ns/mounts/:name
PATCH  /v1/namespaces/:ns/mounts/:name
DELETE /v1/namespaces/:ns/mounts/:name
```

`PATCH` replaces `config` wholesale — the CLI's deep-merge happens
client-side (fetch, merge, PATCH the result) — and the new config must
pass the same per-kind required-field gate as create.

These routes auto-create the namespace on first write, unlike the queue
and worker routes which 404 on an unknown namespace.

## Read next

- [Storages](/get-started/storages.md) — named S3 backends for moving data
  and code in and out of jobs (works end-to-end today).
- [Code mounts](/cli/examples/storage-code-mounts.md) — shipping git
  snapshots for reproducible remote execution.
- [Tunnels](/get-started/tunnels.md) — the parallel registry for WireGuard
  tunnels, with the same metadata-only status.
- [Secrets](/api/auth-and-secrets.md#secrets) — storing the credentials
  mounts reference via `$secret`.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/architecture/f7-run-object

# F7 — the run object (gap)

Row F7 of the [execution matrix](/python-sdk/architecture/execution-matrix.md) is empty.
There is no `run = pipeline.submit(source)` handle you can poll, stream, or
draw. The SDK's own `docs/execution-patterns.md` marks the entire row GAP.

> **Warning:** The "what exists today" sections below are verified, runnable code. The
>   `PipelineRun` sketch further down is a surface proposal: no `PipelineRun`
>   class, no `run.status`, no `run.events`, and no DAG object exists in
>   `dreamlake-lakeshore`. Do not write code against it.

## The gap, precisely

A pipeline today is ordinary Python driving `submit()` calls. When it runs,
exactly two kinds of run record come into existence:

1. **Invocation ids you keep by hand.** Every `submit()` returns a durable
   string id. If you want the graph the run took, you append
   `(src_inv, dst_inv, verb)` tuples to a list yourself.
2. **Journal frames.** Every generator invocation writes VALUE frames and a
   terminal END or ERROR frame to its journal; `Topic` is the same journal
   spelled pub/sub. That is live, durable, cursor-addressable status — per
   invocation.

Nothing joins them. The *structure* (which node feeds which) lives in your
hand-kept edge list or in an author-time trace; the *status* (queued,
running, done) lives in the journal, keyed by invocation id. No object holds
both, so there is nothing to ask "how far along is the run?" and nothing to
hand a canvas to draw live.

There is one unwired seam worth naming: the plane already has a parentage
column. `Dispatch.submit` and `HttpDispatch.submit` both accept `parent=`,
and `wire.Envelope` carries a `parent` field documented as "parent
invocation id (pipeline lineage)". Nothing in the SDK populates it —
`UDF.submit` passes only `id`, `key`, and `priority`. The storage for a run
tree exists; the writer does not.

## What exists today: the hand-kept edge list

`examples/06_dynamic_graph.py` already produces a full run record — by hand.
Every edge is a pair of invocation ids plus a verb. Compact version,
runnable as-is:

```python
import dreamlake.lakeshore as dls
from dreamlake.lakeshore.dispatch import Dispatch

plane = Dispatch(":memory:")
q = dls.SyncQueue("run-demo", dispatch=plane)

@dls.udf(queue="run-demo")
def propose(seed: int) -> list[int]:
    return list(range(seed, seed + (seed % 3) + 1))

@dls.udf(queue="run-demo")
def score(x: int) -> float:
    return (x * 37 % 10) / 10

def run(inv: str):
    dls.run_worker(q, once=True)
    return q.result(inv, timeout=5)

edges: list[tuple[str, str, str]] = []      # (src_inv, dst_inv, verb)

p_inv = propose.submit(2, _dispatch=plane)
candidates = run(p_inv)                     # width discovered at run time

accepted = []
for x in candidates:
    s_inv = score.submit(x, _dispatch=plane)
    edges.append((p_inv, s_inv, "score"))   # the edge, kept BY HAND
    if run(s_inv) >= 0.5:
        accepted.append(x)

assert len(edges) == len(candidates)        # the run's actual graph
```

The full example adds a rework loop — a failed candidate goes through
`refine` and around again, up to three attempts — and prints the edge tally.
A three-seed run produces 18 edges (12 `score`, 6 `refine`), a shape the
author could not have known. The edge list *is* the dynamic graph. It just
lives in a local variable that dies with the process, while the ids inside
it are durable.

## What exists today: the journal as live status

The other half already streams. A generator UDF's yields become journal
frames any consumer can pull with a cursor, and `Topic` is the same
mechanism for ad-hoc events (see `examples/05_streams_topics.py`). Runnable
as-is, continuing the setup above:

```python
@dls.udf(queue="run-demo")
def render_frames(n: int):
    for i in range(n):
        yield {"frame": i, "px": i * i}

inv = render_frames.submit(4, _dispatch=plane)
dls.run_worker(q, once=True)

frames = list(q.stream(inv, timeout=5))          # live per-invocation feed
assert [f["frame"] for f in frames] == [0, 1, 2, 3]

tail = list(q.stream(inv, since=1, timeout=5))   # reconnect from a cursor
assert [f["frame"] for f in tail] == [2, 3]

t = q.topic("progress")                          # same journal, pub/sub
t.post({"node": "score", "state": "running"})
t.post({"node": "score", "state": "ok"})
t.post(None, final=True)
assert [m["state"] for m in t.stream(since=-1, timeout=5)] == ["running", "ok"]
```

Everything a run object needs for liveness is here: durable frames, a
0-based `since=` cursor, reconnect from any process. What is missing is the
map from journal frames back to *nodes in a structure*.

## What exists today: the traced graph (author time)

The [dataflow-primitives draft](/python-sdk/architecture/primitives.md) supplies the
third piece — structure without execution. It lives at `design/primitives/`
in `fortyfive-ai/macrodata-subtask_yanbing`, not in the SDK. There, a `@udf`
call under an active trace does not run its body: it appends a node to the
active `Graph` and returns a `Col`, a lazy handle. The result algebra
extends the graph — `x > 0.5`, `a & ~b`, `review.all(axis=0)`, and
mask-indexing `labels[mask]` each become **filter** nodes; `batch(...)` is a
**source** node, `to_dataset(...)` a **sink**, and `requeue(...)` a dashed
loop-back edge. `@udf(kind=...)` sets the node's bucket, and each bucket
owns one hue of the shared six-hue palette: source green, transform blue,
model purple, filter amber, sink red, idle grey — the same colors the canvas
renders.

So the traced `Graph` knows every node and edge of the pipeline before
anything runs. It has no invocation ids and no status: it is the exact
complement of the journal.

## The PipelineRun sketch

The run object is the join. `pipe.submit(source)` would return a handle
holding three things: the traced structure, a node-to-invocation-id map, and
a journal cursor.

```python
# SKETCH — not shipped. Surface proposal only; names and shapes will move.

run = image_object_annotation.submit(source)     # -> PipelineRun

run.id                    # durable string, like any invocation id
run.graph                 # the traced Graph: nodes, edges, kinds
run.ids                   # {node -> invocation id | None (not submitted yet)}

# Per-node status, colored by the same six-hue palette the graph uses:
run.status                # {node -> "idle" | "queued" | "running" | "ok" | "error"}
run.status["score"]       # one node

run.result("detect_objects")        # q.result on the node's invocation(s)

# Live events ride the journal, cursor and all:
for ev in run.events(since=cursor):          # sync
    ...                                      # (seq, node, frame)
async for ev in run.events(since=cursor):    # async, same shape
    ...

run.graph.to_dot()        # render: DOT out, canvas draws it live
                          # (nodes colored by kind, tinted by status)

# Ids are durable, so reattaching from another process is natural:
run = PipelineRun.attach(run_id)
for ev in run.events(since=run.cursor):      # resume where you left off
    ...
```

Nothing in the sketch invents new machinery. `run.ids` is the hand-kept dict
from the dynamic-graph example, kept by the system instead; `run.events` is
`q.stream` with node labels attached; `run.graph` is the trace the
primitives draft already produces; `attach` is the same reconnect story
`q.result(inv)` and `since=` cursors already have for single invocations.

## Design questions

- **Trace-to-id binding.** The traced graph exists at author time; ids exist
  at submit time. When a node fans out — one `score` node, N invocations —
  is `run.ids["score"]` a list, and who appends to it: the driver, or the
  plane?
- **Dynamic edges.** Rework edges are discovered mid-run. Does the run
  object grow its graph live, with edges arriving as events, or does
  `run.graph` stay the static trace with rework shown only in status?
- **One journal or many.** Does `run.events()` merge the per-invocation
  journals, and if so how are they ordered across journals? Or does the run
  own a single run-level `Topic` that workers post to alongside their own
  journals?
- **Is the run itself an invocation?** If `pipe.submit()` returns a durable
  id and writes journal frames, the pipeline is just an invocation whose
  body submits others — `attach`, `result`, and `stream` come for free,
  `parent` gets its writer, and nested pipelines recurse. That economy is
  attractive; the open question is whether driver-side control flow can run
  *on* a worker without occupying one for the whole run.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths/10-code-mount

# 10 · Code mount

The "your local imports just work on the worker" story. On submit, the
SDK archives the current git tree, uploads it under a content-addressed
key, and stamps the archive onto the envelope. A worker that cannot
import your module fetches and extracts the archive, then retries the
import.

## Exercises

- `HttpDispatch.push_code()` — called automatically by `UDF.submit` for
  reference-transport functions when the plane has a code channel.
- `code_archive.push_if_needed` — `git archive HEAD` → presigned PUT on
  the `code-staging` storage → register on `POST /v1/namespaces/:ns/code-repos`.
- `lakeshore code push` / `lakeshore code list` — the same channel, by hand.
- Worker-side: fetch by `archiveKey`, extract under
  `$LAKESHORE_HOME/code/<commit>`, prepend to `sys.path`, re-import.
- Dedup: a `(remote, commit)` already registered on the control plane is
  never re-uploaded, and the worker reuses an already-extracted commit.

## Requires

- Everything in [05 · Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md),
  against a **control plane** (`--url` / `LAKESHORE_URL`). The local
  SQLite plane has no code channel — it assumes a shared filesystem.
- Everything in [09 · Storage round-trip](/cli/happy-paths/09-storage-round-trip.md):
  a storage entry named `code-staging` for the archives.
- A **git repository with a clean working tree**. `git archive HEAD`
  ships committed content only; `push_if_needed` returns `None` on a
  dirty tree unless it is called with `allow_dirty=True`
  (`lakeshore code push --dirty`).

## Verifies

A `@udf` whose module is not installed on the worker still runs there:
the worker's `ImportError` triggers a fetch of the submitted archive and
the import succeeds on the retry.

## Run

Push the archive explicitly and confirm it registered:

```bash
lakeshore code push
lakeshore code list
```

Then submit against the control plane, from inside the repo:

```bash
export LAKESHORE_URL=http://localhost:8080
export LAKESHORE_CLIENT_TOKEN=<token>
python my_project/submit.py
```

with a worker running on the same plane:

```bash
lakeshore worker start --queue default --url http://localhost:8080 --token <token>
```

> **Note:** `lakeshore-examples/_legacy/11-code-mount` exercises the archive channel
> through `lakeshore exec`, not through `@udf`. It is still a valid check
> of the upload half; the worker-side import path is exercised by the SDK
> submit above.

## Expected output

The UDF's return value comes back, and the worker imported a module it
did not have on disk when it started. The first submit on a new commit
uploads (slow); a second submit on the same commit skips the upload
entirely — that is the dedup signal.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `WorkerError: cannot import module ...` | No archive was stamped on the envelope. Dirty working tree, not a git repo, or the plane is the local SQLite one (no code channel). Commit, then resubmit. |
| Your edits do not take effect on the worker | `git archive HEAD` ships committed content only — uncommitted changes never travel. |
| `WorkerError: ... source differs from the submitted version` | The worker resolved a different revision of the function. Update its checkout, or pass `--runtime-policy off` for same-machine dev. |
| Upload fails or hangs | The `code-staging` storage entry is missing or its credentials are wrong. `lakeshore code push` shows the raw failure. |
| Slow on every run (no dedup) | The commit changed every run (amend/rebase loop), or the worker's `LAKESHORE_HOME/code` cache is not persistent. |

## Status

Manual.

## Next

→ [11 · Storage access from a UDF](/python-sdk/happy-paths/11-storage-from-udf.md)

---

Source: https://lakeshore.dreamlake.ai/get-started/storages

# Storages

A **Storage** is a named reference to an S3-compatible bucket (or a
scoped prefix inside one) registered under a namespace. The control
plane keeps the long-lived credentials encrypted in its Secret store and
hands out **presigned URLs** or **short-lived STS credentials** on
request. Your scripts see a storage name; they never see an access key.

Two kinds, and only two:

| Kind | Config | What it is |
| --- | --- | --- |
| `s3` | `{ bucket, region?, endpoint?, creds }` | A whole bucket. |
| `s3-prefix` | `{ bucket, prefix, region?, endpoint?, creds }` | A scoped directory inside a bucket. |

Note the **hyphen** in `s3-prefix`. Omitting `endpoint` means AWS S3;
setting it targets Ceph, MinIO, R2, or Backblaze B2 (the client
switches to path-style addressing when an endpoint is present).

## Immutability

`bucket`, `prefix`, and `endpoint` are denormalized out of the config
onto the row and are **immutable after creation**. Repointing a storage
would silently break every reference to it, so you create a new entry
instead. Only `description`, `creds`, and `region` are mutable.

Two unique indexes back this: `(namespace, name)` and
`(namespace, bucket, prefix, endpoint)` — the second de-duplicates two
names pointing at the same target.

## Register a storage

```bash
lakeshore storage add my-data \
  --kind s3 \
  --bucket my-lab-datasets \
  --region us-east-1 \
  --creds aws-lab
```

`--creds <name>` is shorthand for `--kwarg 'creds.$secret=<name>'`. The
named secret must already exist in the namespace:

```bash
printf '%s:%s' "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" \
  | lakeshore secrets add aws-lab --kind aws_keypair
```

The **storage** code path accepts an `aws_keypair` secret in either the
JSON form (`{"accessKeyId": …, "secretAccessKey": …}`) or the
colon-delimited `<access_key_id>:<secret_access_key>` form. Everywhere
else — the EC2/GCE/SSH `$secret` substitution — only the colon form
parses, so store the colon form and the same secret works for both.

The name must match `^[a-z0-9][a-z0-9-]*$`. You may prefix it with a
namespace — `lakeshore storage add team-robotics/training-data …`
targets that namespace instead of your default. The same
`<namespace>/<name>` form works on `show`, `update`, `remove`,
`presign`, and `credentials` (but not `list`).

### Provisioning a bucket

`--provision` has the control plane create the bucket for you. It is
only supported for `kind: s3`, requires a `creds.$secret` pointing at an
`aws_keypair`, and needs an explicit `--bucket` — the name is **not**
derived from the namespace.

```bash
lakeshore storage add scratch \
  --kind s3 \
  --bucket dreamlake-geyang-scratch \
  --region us-east-1 \
  --creds aws-lab \
  --provision \
  --s3-option ACL=private
```

The call is idempotent: a `HeadBucket` runs first, and an existing
bucket you own is adopted rather than recreated (the response reports
`bucketCreated: false`). `--s3-option key=value` entries are forwarded
verbatim as `CreateBucket` parameters.

## The verbs

`storage` is **singular**, unlike every sibling group. There are seven
subcommands:

| Command | What it does |
| --- | --- |
| `storage add <name>` | Register (and optionally provision). |
| `storage list [--json]` | List entries in the current namespace. |
| `storage show <name> [--json]` | Print one entry's config. |
| `storage update <name>` | `--kwarg k=v` (deep-merge, repeatable), `--config-file` (replace wholesale), `--description`. |
| `storage remove <name> [--purge]` | Delete the record. `--purge` also deletes the S3 bucket. |
| `storage presign <name> <key>` | Mint a presigned URL. |
| `storage credentials <name>` | Vend short-lived STS credentials. |

> **Warning:** The CLI does not ship `storage upload`, `download`, `duplicate`, `mv`,
> `ls`, or `sync`. Data movement goes through a presigned URL or STS
> credentials — use `curl`, the AWS CLI, `rclone`, or boto3 with the
> credentials the control plane vends. `add` takes bare `--<key> <value>`
> pairs as pass-through config fields, so an unknown flag will be swallowed
> as config rather than rejected.

## Presigned URLs

```bash
# Download URL (default), 1 hour
lakeshore storage presign my-data runs/exp-01/model.pt

# Upload URL
lakeshore storage presign my-data runs/exp-01/model.pt --put --expires-in 900
```

`--expires-in` defaults to 3600 seconds and is clamped to 86400. The key
you pass is relative to the storage's configured `prefix`; the server
joins them and returns the full key alongside the URL.

Then move the bytes yourself:

```bash
URL=$(lakeshore storage presign my-data runs/exp-01/model.pt --put)
curl -X PUT --upload-file ./model.pt "$URL"
```

## Temporary credentials

For anything that wants a real S3 client — boto3, the AWS CLI, `rclone`,
`torch.save` straight to S3 — ask for STS session credentials instead:

```bash
# Shell exports
eval "$(lakeshore storage credentials my-data --env)"
aws s3 sync ./checkpoints/ s3://my-lab-datasets/runs/exp-01/

# Or JSON, for scripting
lakeshore storage credentials my-data --json --duration 7200
```

`--duration` defaults to 3600 seconds and accepts 900–129600. The
control plane calls STS `GetSessionToken` with the storage's own
credentials.

## From Python

```python
from dreamlake.lakeshore import Storage

store = Storage("my-data")          # server/namespace/token resolved from env or auth.yml

url = store.presign("runs/exp-01/model.pt")                 # presigned GET
url = store.presign("runs/exp-01/model.pt", operation="put")

data = store.get("runs/exp-01/config.json")                 # -> bytes
store.put("runs/exp-01/out.json", b'{"loss": 0.01}')
text = store.get_text("runs/exp-01/notes.md")

creds = store.credentials(duration_seconds=7200)
session = boto3.Session(**creds.to_boto3_kwargs())
env = creds.to_env()      # AWS_ACCESS_KEY_ID / …_SECRET_ACCESS_KEY / …_SESSION_TOKEN / AWS_REGION
```

`Storage` resolves the control plane the same way the rest of the SDK
does: an explicit `base_url=`, then `LAKESHORE_URL`, then
`~/.config/lakeshore/auth.yml`, then `http://localhost:8080`. The token
comes from `LAKESHORE_CLIENT_TOKEN` (or `auth.yml`).

## Shared storage across namespaces

Register the storage under the team namespace, then reference it by
`<namespace>/<name>`:

```bash
# Admin registers it under team-robotics
lakeshore storage add team-robotics/training-data \
  --kind s3 --bucket dreamlake-team-robotics-data \
  --region us-east-1 --creds aws-team

# Any member of the org mints a URL against it
lakeshore storage presign team-robotics/training-data v2/dataset.tar
```

Cross-namespace access requires the two namespaces to share an `orgId` —
a token scoped to one namespace in an org authorises requests against
its siblings.

## HTTP

```
POST   /v1/namespaces/:ns/storages
GET    /v1/namespaces/:ns/storages
GET    /v1/namespaces/:ns/storages/:name
PATCH  /v1/namespaces/:ns/storages/:name
DELETE /v1/namespaces/:ns/storages/:name
POST   /v1/namespaces/:ns/storages/:name/presign      { key, operation?, expiresIn? }
POST   /v1/namespaces/:ns/storages/:name/credentials  { durationSeconds? }
```

Deletion is a soft delete: the row is renamed with a tombstone suffix
and `deletedAt` is stamped, so the unique index stays intact.

## See also

- [Payload store](/get-started/payloads.md) — the *other* S3 path, owned by
  the control plane and used for job arguments and results.
- [Mounts](/get-started/mounts.md) — the registry for filesystems the runner
  will attach into a job's workdir.
- [Auth and secrets](/api/auth-and-secrets.md) — how `$secret` references
  and `aws_keypair` secrets work.
- [Storage, code, and mounts examples](/cli/examples/storage-code-mounts.md)
  — runnable end-to-end commands.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/happy-paths/11-storage-from-udf

# 11 · Storage access from a UDF

A worker moves bytes through a **named** storage entry without holding
any AWS credentials. The control plane mints a presigned URL (or STS
temporary credentials) at call time; the code only ever names the
storage and the key.

## Exercises

- `dls.Storage(name)` — the handle for a storage entry registered on the
  control plane.
- `presign(key, operation="get"|"put", expires_in=...)` →
  `POST /v1/namespaces/:ns/storages/:name/presign`.
- `credentials(duration_seconds=...)` →
  `POST /v1/namespaces/:ns/storages/:name/credentials`, returning a
  `TemporaryCredentials` with `to_boto3_kwargs()` and `to_env()`.
- The convenience pairs `get` / `put` and `get_text` / `put_text`, which
  presign and then do a plain HTTP transfer.
- Credential rotation: nothing is embedded in the code or the envelope,
  so rotation is transparent to running workloads.

## Requires

- Everything in [09 · Storage round-trip](/cli/happy-paths/09-storage-round-trip.md)
  — a registered storage entry with a known object in it.
- Everything in [05 · Hello, queue-bound UDF](/python-sdk/happy-paths/05-hello-queue-udf.md)
  against a control plane, plus
  [10 · Code mount](/python-sdk/happy-paths/10-code-mount.md) if the worker
  does not already have your module.
- `LAKESHORE_URL` + `LAKESHORE_CLIENT_TOKEN` in the worker's environment
  (or saved `lakeshore auth login` credentials in `~/.config/lakeshore/auth.yml`,
  which the `Storage` side channel does read).

## Verifies

A worker reads and writes data it holds no static credentials for, by
name, through the control plane.

## Run

```python
# storage_udf.py
import dreamlake.lakeshore as dls

@dls.udf(queue="default")
def peek(storage: str, key: str) -> dict:
    with dls.Storage(storage) as store:
        blob = store.get(key)
        store.put_text(f"{key}.size", str(len(blob)))
    return {"key": key, "bytes": len(blob)}

inv = peek.submit("my-bucket", "test/hello.txt")
q = dls.SyncQueue("default")
print(q.result(inv, timeout=60))
```

The bundled example, `lakeshore-examples/_legacy/12-storage-access/access.py`,
walks the same three access modes (`presign`, `credentials`, `put`) from
a local script rather than from inside a UDF:

```bash
uv run access.py --storage my-bucket --mode presign
uv run access.py --storage my-bucket --mode credentials
uv run access.py --storage my-bucket --mode put
```

## Expected output

`{'key': 'test/hello.txt', 'bytes': N}` — with no AWS variables in the
worker's environment.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `403 Forbidden` on the presigned URL | The URL expired before the transfer, or the client and S3 clocks are skewed. Raise `expires_in` (`get` presigns for only 300 s by default; the cap is 86400). |
| `404` from the presign route | No storage entry by that name in this namespace. `lakeshore storage list`. |
| `NoSuchKey` | The object is not in the bucket. Upload it first. |
| The worker cannot reach the control plane | `Storage` resolves its server from `LAKESHORE_URL` / `DREAMLAKE_SERVER` / `~/.config/lakeshore/auth.yml`, then falls back to `http://localhost:8080` with no token. |
| A `boto3` import error | The presigned path is a plain HTTP transfer and needs no boto3. Only `credentials().to_boto3_kwargs()` implies boto3, and you supply it. |

> **Warning:** `dls.run.read` / `dls.run.write` resolve keys on the worker's local
> filesystem under its `--root`. Nothing binds a `Storage` to the run
> context today, so a data UDF that needs object storage must call
> `dls.Storage(...)` explicitly, as above.

## Status

Manual.

## Next

→ [12 · Compose a cluster declaratively](/admin/happy-paths/12-compose-cluster.md)

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths/12-compose-cluster

# 12 · Compose a cluster declaratively

Write one YAML, get a working fleet. `up` reconciles toward the declared
state and is idempotent — re-running with a delta applies only the delta.

> **Warning:** There is no `lakeshore compose` command. It is `lakeshore up`, `down`,
> `ps`, `status`, and `logs`, even though the source lives in
> `src/cli/compose/`.

## Exercises

- `lakeshore up` — parse `lakeshore.yaml`, diff against the control
  plane, apply.
- The daemon-group abstraction — N daemons sharing config and queue
  membership, labelled `<group>-0`, `<group>-1`, …
- `lakeshore ps` / `status` — fleet state scoped by the `project` slug.
- `lakeshore down` — drain (or terminate) the fleet.
- Reconciliation: state is derived from the control plane on every run.
  There is no lock file in v1.

## Requires

- A provider that can provision — EC2, GCE, or Kube. Its credentials
  registered as secrets and referenced from the provider kwargs.
- A `lakeshore.yaml` in the current directory. Lookup is cwd-only; there
  is no upward walk, though `-f, --file <path>` overrides it.

## The file

```yaml file="lakeshore.yaml"
version: 1
project: 08-queues-tutorial
provider: aws-us-east

queues:
  cpu-lane:
    kind: fifo
    description: "CPU pool — batch + smoke"
  gpu-t4:
    kind: priority
    description: "T4 GPUs (g4dn.xlarge)"

daemons:
  cpu:
    queues: [cpu-lane]
    count: 5
    instance_type: t3.medium
    keep_alive_s: 600
    setup:
      - apt-get update -y && apt-get install -y python3-pip
      - pip3 install --break-system-packages dreamlake-lakeshore
  gpu:
    queues: [gpu-t4]
    count: 5
    instance_type: g4dn.xlarge
    image_id: ami-012ba162b9cd2729c
    keep_alive_s: 1800
```

`version: 1`, `project`, and a non-empty `daemons` mapping are required.
`project` and every queue and group name must match
`^[a-z0-9][a-z0-9-]*$`. Inside a daemon group, `queues` and `count` are
required — `count` has no default on purpose, because a silent `1` is
more footgun than convenience — and any key outside the allowed set is a
hard error naming the allowed keys.

Queue `elasticity.kind` in YAML is **underscored**: `fixed`,
`fully_elastic`, `pool_with_threshold`, `max_count`. (The
`queues add --elasticity` CLI flag spells the same values with hyphens.
The two surfaces are not interchangeable.)

## Run

```bash
git clone --recurse-submodules \
  https://github.com/dreamlake-ai/lakeshore-workspace.git
export LAKESHORE_WORKSPACE=~/lakeshore-workspace
export LAKESHORE_URL=http://localhost:8080

cd $LAKESHORE_WORKSPACE/lakeshore-examples/08-queues-cpu-vs-gpu

lakeshore up --plan          # dry run — prints the diff, no side effects
lakeshore up --wait          # apply, block until every daemon is active
lakeshore ps                 # fleet state
lakeshore up cpu             # reconcile one group only
lakeshore down --terminate   # default is drain; --terminate kills instances
```

## Expected output

`--plan`:

```text
Plan for project '08-queues-tutorial':
  queue cpu-lane  (would create)
  queue gpu-t4  (would create)
  daemons cpu  0 → 5  (would launch: cpu-0, cpu-1, cpu-2, cpu-3, cpu-4)
  daemons gpu  0 → 5  (would launch: gpu-0, gpu-1, gpu-2, gpu-3, gpu-4)
```

Applying:

```text
✓ project '08-queues-tutorial': 10 daemon(s) launched, 2 queue(s) created
  + queue cpu-lane
  + queue gpu-t4
  + daemon cpu-0  (6712a…)
  …
```

A second `up` with nothing changed reports `(no change)` per group and
`already present` per queue.

`down`:

```text
✓ project '08-queues-tutorial': 10 worker(s) deleted, 10 cloud instance(s) terminated
  - cpu-0 (6712a…)
  …
```

`--prune-queues` also removes the declared queues; without it they are
kept and the reason is printed.

## How scoping works

Every daemon `up` spawns is stamped with `compose_project: <project>`.
`ps`, `status`, and `down` list workers with
`?compose_project=<project>` and match group membership by the
`<group>-` label prefix. Two projects in the same namespace never see
each other's daemons.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| Parse error naming a key path | The loader validates strictly. The message includes the offending path — e.g. `daemons.gpu.count — required, positive integer (got null)`. |
| `up` reports failures per daemon | The provider could not provision. Each failure line carries the launcher's error; `providers instances <name>` usually shows why. |
| Queue created but no daemons | A crash mid-reconcile. `down` cleans up; if it does not, `lakeshore daemon kill --prefix <group>-` and `lakeshore queues rm <name>`. |
| Re-running `up` re-creates things | A reconciliation bug — file an issue with the `--plan` output from both runs. |

## Status

Manual.

## Next

→ [13 · Cloud provider launch](/admin/happy-paths/13-cloud-provider-launch.md)

---

Source: https://lakeshore.dreamlake.ai/api/configuration

# Configuration

Four config files, owned by different processes and frequently confused
with each other. Nothing merges across them.

| File | Read by | Purpose |
| --- | --- | --- |
| [`.dreamrc`](#dreamrc) | CLI + Python SDK, producer-side | Named compute **modes** (RunConfigs) and **provider** connections. |
| [`udf-daemon.toml`](#udf-daemontoml) | `nymph`, on the host | Daemon bootstrap — server URL, runners, workdir, poll tunables, identity. |
| [`.lakeshore` / `.lakeshore.local`](#lakeshore-project-defaults) | CLI, `lakeshore daemon launch` only | Per-project launch defaults. |
| [`lakeshore.yaml`](#lakeshoreyaml-compose) | CLI, `up` / `down` / `ps` / `status` / `logs` | Declarative queue + daemon-group topology. |

## `.dreamrc`

YAML. Declares named **modes** (`RunConfig`s) and **providers** — the
account/cloud connections the control plane launches workers through.

### Resolution order

The CLI resolver returns the first hit:

1. `--dreamrc <path>` → source `explicit`
2. `$DREAMRC` → `env`
3. `./.dreamrc`, walking up from cwd → `workspace`
4. `$HOME/.dreamrc` → `home`
5. Server pull (needs saved auth or `LAKESHORE_URL`) → `server`
6. On-disk cache from a previous pull → `cache`
7. Nothing → `none` (an empty `DreamRc`)

Inspect with `lakeshore config show` (the resolved config as YAML),
`lakeshore config refresh` (force a server pull and update the cache), or
`lakeshore config source` (one word: `env`, `workspace`, `home`,
`server`, `cache`, `none`).

The server pull composes a `DreamRc` from
`GET /v1/namespaces/<ns>/providers` and `GET /v1/namespaces/<ns>/modes`.
A server mode literally named `default` is promoted to the root
`RunConfig`. Providers whose launcher the CLI doesn't recognize are
skipped silently rather than failing the pull. The cache lives at
`<cacheRoot>/<server-host-slug>/<namespace>/dreamrc.yml`, where
`cacheRoot` is `$XDG_CACHE_HOME/lakeshore` or `~/.cache/lakeshore`
(directories `0700`, files `0600`). A failed pull falls back to the cache
and prints `⚠ server unreachable, using cached config from <ts>`.

The Python SDK's own loader (`dls.load_config()`) is deliberately
shorter — explicit path, `$DREAMRC`, `./.dreamrc` walking up,
`~/.dreamrc`. It does **not** pull from the server.

### Example

```yaml file=".dreamrc"
default_mode: local

providers:
  bos14: !providers.SLURM
    host: login.bos14.example.edu
    user: ge
    partition: gpu

  aws-prod: !providers.EC2
    region: us-east-1
    aws_credentials: { $secret: aws-prod-keys }

modes:
  local:
    backend: local

  bos14-gpu:
    backend: fabric
    provider: bos14
    runner: process
    resources: { cpu: 4, gpu: 1 }
    time_limit: "02:00:00" # unknown key → forwarded to the launcher

  aws-h100:
    backend: fabric
    provider: aws-prod
    tags: [gpu:h100]
    runner: docker
    image: ghcr.io/example/torch:2.4
    resources: { gpu: 1, mem: 64Gi }
    timeout_s: 7200
```

### Provider entries

A provider entry **must** be a YAML-tagged mapping. The tag is the type
discriminator — there is no `type:` field.

```
!providers.SSH   !providers.SLURM   !providers.EC2   !providers.GCE   !providers.Kube
```

Those five are the whole set (`KNOWN_PROVIDER_TAGS`). An unknown
`!providers.X` tag is a hard parse error. The body is the launcher
kwargs, deep-merged with the referencing mode's launcher-shaped fields at
dispatch time.

Default dispatch per launcher, overridable with an explicit `dispatch:`
key inside the kwargs:

| Launcher | Default dispatch |
| --- | --- |
| `SSH` | `direct` |
| `SLURM` | `direct` |
| `Kube` | `direct` |
| `EC2` | `daemon` |
| `GCE` | `daemon` |

### Mode fields

Top-level keys: `default_mode` (default `"local"`), `modes` (alias:
`mode`), `providers`. Any *other* top-level key is parsed as a root
`RunConfig`. A synthetic `local` mode is injected when the file declares
none.

| Field | Type | Default | Notes |
| --- | --- | --- | --- |
| `backend` | string | `"local"` | `"local"` runs in-process; `"fabric"` dispatches through the fabric. |
| `server` | string \| null | _none_ | Control-plane URL override for this mode. |
| `tags` | list[string] | `[]` | Selector — matched against daemon tags. |
| `runner` | string | `"docker"` | `"process"`, `"docker"`, `"gvisor"`, `"slurm"`, `"kube"` (`"subprocess"` aliases `"process"`). |
| `resources` | table | `{}` | Reservation hints (`cpu`, `mem`, `gpu`, …). |
| `image` | string \| null | _none_ | OCI image. Required by the `docker` / `gvisor` runners. |
| `env` | table | `{}` | Extra env vars for the invocation. |
| `timeout_s` | int \| null | _none_ | Wall-clock cap. |
| `provider` | string \| null | _none_ | Name of a provider entry. |

Any key not in that list lands in `extras` and is forwarded to the
launcher at dispatch time (`instance_type`, `image_id`, `partition`,
`time_limit`, …).

### Server-stored modes

Modes also live as rows on the control plane, managed with
`lakeshore modes add | list | show | update | remove`. The override flag
is `--field`, not `--kwarg`:

```bash
lakeshore modes add aws-h100 \
  --field backend=fabric \
  --field provider=aws-prod \
  --field runner=docker \
  --field image=ghcr.io/example/torch:2.4 \
  --field resources.gpu=1 \
  --field timeout_s=7200
```

Dotted keys nest. `lakeshore modes update <name> --field …` deep-merges,
so you can patch one field; `--field foo=null` deletes a key.
`--config-file <path>` loads (on `add`) or replaces (on `update`) the
whole RunConfig from YAML or JSON.

> **Warning:** The Mode create handler defaults `backend` to `"fabric"` and `runner` to
> `"process"` when unsupplied, and otherwise accepts **any string**. A typo
> is stored and echoed back rather than rejected. Unknown config keys
> round-trip through `resources.__extras`.

## `udf-daemon.toml`

`nymph`'s bootstrap config. Read **once** at process start and never
re-read — everything mutable arrives over the long-poll channel.

Lookup order when `--config <path>` is absent:

1. `/etc/dreamlake/udf-daemon.toml`
2. `$HOME/.config/dreamlake/udf-daemon.toml`
3. Built-in defaults

`$LAKESHORED_CONFIG` (note the trailing `D`) is the env form of
`--config`.

The top-level `Config` is `#[serde(deny_unknown_fields)]`, so an unknown
**section** is a hard parse error. The sub-structs are not, so an unknown
**key inside** a known section is silently ignored.

```toml file="udf-daemon.toml"
[server]
url = "https://api.lakeshore.dreamlake.ai"
namespace = "default"
token_file = "/etc/dreamlake/token"
tls_verify = true

[daemon]
machine_id = "auto"
label = "gpu-box-01"
tags = ["gpu:h100", "region:us-east"]
lanes = []

[runtime]
workdir = "/var/lib/dreamlake/udf"
max_invocations = 100
runners = ["process", "docker"]
default_runner = "process"
keep_alive_s = 300
auto_update = false
auto_update_check_interval_s = 300

[poll]
wait_max_s = 20.0
retry_min_s = 1.0
retry_max_s = 60.0

[shutdown]
grace_s = 60

[log]
level = "info"
format = "human"
event_buffer = 1024

[introspect]
enabled = false
bind = "127.0.0.1"
port = 9876

[identity]
enabled = false
key_file = "/etc/dreamlake/daemon.key"
enroll_token_file = "/etc/dreamlake/enroll-token"

[mounts]
cache_dir = "/var/lib/dreamlake/cache"
cache_max_bytes = 21474836480
```

### `[server]`

Exactly four keys.

| Field | Type | Default | Notes |
| --- | --- | --- | --- |
| `url` | string | `http://localhost:8080` | Control-plane base URL. |
| `token_file` | path \| null | _none_ | Legacy bearer token, sent as `Authorization: Bearer` on every request. |
| `tls_verify` | bool | `true` | `false` sets reqwest's `danger_accept_invalid_certs`. Local dev only. |
| `namespace` | string | `"default"` | Used only for namespace-scoped routes (the payload presign endpoint). |

> **Warning:** `ServerConfig` has no `transport`, no `socket_path`, and no Unix-socket
> mode. Because `[server]` is not `deny_unknown_fields`, a `transport =`
> line would be silently ignored rather than honored.

### `[daemon]`

| Field | Type | Default | Notes |
| --- | --- | --- | --- |
| `machine_id` | string | `"auto"` | `"auto"` resolves to the hostname on `/hello`. |
| `label` | string | `""` | Display name; falls back to the worker id when empty. |
| `tags` | list[string] | `[]` | Merged with auto-detected host tags; config tags win on a `key=` collision. |
| `lanes` | list[string] | `[]` | Queue memberships, sent as `HelloRequest.lanes` (the back-compat wire name for `queues`). Omitted from the wire entirely when empty, which the control plane reads as default-queue membership. |

> **Warning:** The control plane's daemon-bootstrap renderer emits a
> `queues = [...]` key in `[daemon]`. The `nymph` source in this checkout
> has no `queues` field — only `lanes` — so that key parses without error
> and is then ignored. If you are hand-writing the TOML for this build,
> use `lanes`.

### `[runtime]`

| Field | Type | Default | Notes |
| --- | --- | --- | --- |
| `workdir` | path | `/var/lib/dreamlake/udf` | Root for per-invocation directories. See the [workdir contract](/nymph/daemons.md#shared-workdir-contract). |
| `max_invocations` | int | `100` | Concurrency cap, enforced by a semaphore. `0` = unbounded. |
| `runners` | list[string] | `["process"]` | Advertised on `/hello`. Detected runners (`slurm`, `kube`) are appended if not already present. |
| `default_runner` | string | `"process"` | Used when a run config names no runner. |
| `keep_alive_s` | int | `300` | `-1` = pool mode, never idle-exit. `0` = exit as soon as the idle clock starts. `N` = exit after N continuous idle seconds. |
| `auto_update` | bool | `false` | Converge to the namespace's `target_nymph_version`. |
| `auto_update_check_interval_s` | int | `300` | Floor on how often the daemon reacts to a version advertisement. |

Runner kinds accepted at dispatch: `process`, `subprocess` (alias for
`process`), `docker`, `gvisor`, `slurm`, `kube`. An unknown kind fails
the invocation with `unknown runner kind: <k>` — it does **not** fall
back to the default runner.

There is no `default_image` key and no `[runtime.runner]` section. The
image comes from `run_config.image` per invocation; the `docker` runner
fails immediately with `ImageRequired` when it is missing.

### `[poll]`, `[shutdown]`, `[mounts]`

| Field | Default | Notes |
| --- | --- | --- |
| `poll.wait_max_s` | `20.0` | Requested long-poll window. The control plane hard-caps its own side at 20 s. |
| `poll.retry_min_s` | `1.0` | Backoff floor; a successful poll resets to this. |
| `poll.retry_max_s` | `60.0` | Backoff ceiling. Growth is `min(backoff * 2, retry_max_s)` with full jitter. |
| `shutdown.grace_s` | `60` | Seconds in-flight work gets after SIGTERM before the kill token fires. |
| `mounts.cache_dir` | `/var/lib/dreamlake/cache` | Mount cache root. |
| `mounts.cache_max_bytes` | `21474836480` | 20 GiB. |

### `[log]`, `[introspect]`, `[identity]`

`[log]`: `level` (default `"info"`), `format` (`"human"` \| `"json"`; an
unrecognised value warns to stderr and falls back to human),
`event_buffer` (default `1024`; `0` disables the ring buffer).
`RUST_LOG` overrides `level`; `LAKESHORE_LOG_FORMAT` overrides `format`.

`[introspect]`: `enabled` (default **false**), `bind` (default
`127.0.0.1`), `port` (default `9876`; `0` = OS-assigned). The endpoint is
unauthenticated, so a non-loopback `bind` is refused at startup — the
failure is logged as a warning and introspection is skipped, not fatal.

`[identity]`: `enabled` (default **false**), `key_file` (default
`/etc/dreamlake/daemon.key`), `enroll_token`, `enroll_token_file`. The
inline token wins over the file. If `enabled = true` and the key cannot
be loaded, the daemon fails closed rather than silently degrading to the
bearer path.

Both are documented in full on
[Telemetry and introspection](/nymph/telemetry.md) and
[Key rotation](/nymph/key-rotation.md).

### Daemon env vars

| Var | Equivalent |
| --- | --- |
| `LAKESHORED_CONFIG` | `--config` |
| `LAKESHORE_SERVER` | `--server` |
| `LAKESHORE_TOKEN` | `--token` |
| `DREAMLAKE_QUEUE` | `--queue` (default `"default"`) |
| `RUST_LOG` | overrides `log.level` |
| `LAKESHORE_LOG_FORMAT` | overrides `log.format` |

> **Warning:** A comment in `config.rs` mentions `LAKESHORE_INTROSPECT` as a way to
> enable the introspection server. No code reads it. The only switch is
> `[introspect] enabled = true`.

## `.lakeshore` project defaults

Two files at the project root, read **only** by `lakeshore daemon
launch`: `.lakeshore` (checked in) and `.lakeshore.local` (gitignored
override). Discovery walks up from cwd and stops at the filesystem root
or at any directory containing `.git`. The first level where *either*
file exists wins; `.lakeshore.local` layers on top with a shallow
spread — arrays **replace**, they do not concatenate.

```yaml file=".lakeshore"
daemon:
  provider: aws-prod
  name: trainer
  runners: [process, docker]
  tags: [gpu:h100]
  queues: [training]
  keep_alive_s: 600
  capacity: { instance_type: p5.48xlarge }
  bootstrap_script: ./scripts/cloud-init.sh
  setup_scripts:
    - ./scripts/install-cuda.sh
```

Every key is optional and unknown top-level keys are ignored. `lanes` is
a legacy synonym for `queues`. `bootstrap_script` and `setup_scripts`
paths resolve relative to the **config file's** directory, not cwd. Full
walkthrough: [Daemon lifecycle → Project defaults](/nymph/daemon-lifecycle.md#project-defaults--lakeshore).

## `lakeshore.yaml` compose

The declarative topology file for `lakeshore up | down | ps | status |
logs`. Looked up as `./lakeshore.yaml` from cwd only — there is no
upward walk — or named explicitly with `-f, --file`.

Requires `version: 1`, a `project:` slug matching `^[a-z0-9][a-z0-9-]*$`,
and a non-empty `daemons:` mapping. Optional at top level: `namespace`,
`provider`, `queues`.

```yaml file="lakeshore.yaml"
version: 1
project: research
namespace: default
provider: aws-prod

queues:
  training:
    kind: fifo
    elasticity: { kind: pool_with_threshold, min: 0, max: 8, threshold: 0.7 }

daemons:
  trainers:
    queues: [training]
    count: 2
    instance_type: p5.48xlarge
    runner: docker
```

> **Warning:** `lakeshore.yaml` accepts only the **underscored** forms — `fixed`,
> `fully_elastic`, `pool_with_threshold`, `max_count`. The
> `lakeshore queues add --elasticity` flag documents the **hyphenated**
> forms — `fixed`, `fully-elastic`, `pool-with-threshold`, `max-count`.
> Neither spelling is universal.

Daemon-group keys: `queues` (required, non-empty), `count` (required
positive integer, no default), plus optional `provider`, `instance_type`,
`image_id`, `runner`, `keep_alive_s`, `setup[]`, `startup`, `python`,
`mounts[]`. Any other key is a hard error listing the allowed set. Full
reference: [Compose](/get-started/compose.md).

## See also

- [Architecture](/get-started/architecture.md) — how modes, providers, queues, and daemons fit together.
- [Auth and secrets](/api/auth-and-secrets.md) — `$secret` references inside provider kwargs.
- [Daemons](/nymph/daemons.md) — the host-side runtime.
- [Daemon protocol](/nymph/protocol.md) — what the daemon sends and receives.
- [Function protocol](/api/function-protocol.md) — the invocation wire format.

---

Source: https://lakeshore.dreamlake.ai/get-started/compose

# Compose

A `lakeshore.yaml` at the project root declares the cluster you want to
exist: which queues, which daemon groups, how many of each. One
`lakeshore up` reconciles the actual state to it; `lakeshore down` tears
it back down. Docker-compose for fleets of daemons.

The five verbs are **top-level commands**, not subcommands of a
`compose` group — there is no `lakeshore compose …`.

## When you want this

You have a script in your repo that does this:

```bash
lakeshore queues add cpu-queue --kind fifo
lakeshore queues add gpu-t4    --kind priority
lakeshore daemon launch --queue cpu-queue --provider ec2-cpu --count 5 --name cpu --wait
lakeshore daemon launch --queue gpu-t4    --provider ec2-gpu --count 5 --name gpu --wait
# ... work ...
lakeshore daemon kill --prefix cpu- --yes
lakeshore daemon kill --prefix gpu- --yes
lakeshore queues rm cpu-queue
lakeshore queues rm gpu-t4
```

That sequence is the literal serialization of a document you should
have. With compose it collapses to:

```bash
lakeshore up
# ... work ...
lakeshore down --terminate
```

And the document lives in your repo, version-controlled with the code
that depends on it.

## A minimal `lakeshore.yaml`

```yaml file="lakeshore.yaml"
version: 1
project: hello-cluster
provider: ec2-prod            # project-level default; groups can override

queues:
  cpu-queue:
    kind: fifo
    description: "CPU pool — batch + smoke"
  gpu-t4:
    kind: priority

daemons:
  cpu:                        # label prefix → cpu-0, cpu-1, …
    queues: [cpu-queue]
    count: 5
    instance_type: t3.medium
    keep_alive_s: 600
  gpu:
    queues: [gpu-t4]
    count: 5
    instance_type: g4dn.xlarge
    keep_alive_s: 1800
```

`version: 1` and a slug-shaped `project:` are required, as is a
`daemons:` block with at least one group. The file is looked up as
`./lakeshore.yaml` from the current directory — there is **no upward
walk** in v1.

## Schema

### Top level

| Key | Required | Meaning |
| --- | --- | --- |
| `version` | yes | Must be exactly `1`. |
| `project` | yes | Slug (`^[a-z0-9][a-z0-9-]*$`) tagged onto every Worker row this file spawns. |
| `namespace` | no | Overrides the namespace from `auth.yml`. |
| `provider` | no | Project-level default provider. |
| `queues` | no | Mapping of name → queue spec. |
| `daemons` | **yes** | Mapping of group name → group spec. At least one. |

### Queue spec

| Key | Meaning |
| --- | --- |
| `kind` | `fifo`, `priority`, `boltzmann`, or `filo`. |
| `description` | Free text. |
| `elasticity` | `{ kind?, min?, max?, threshold? }` — and only those four. |
| `daemon_template` | `{ instance_type?, image_id?, runner? }`. |

Elasticity kinds in YAML are **underscored**: `fixed`, `fully_elastic`,
`pool_with_threshold`, `max_count`. (The CLI's `queues add
--elasticity` flag takes the hyphenated spelling; the YAML validator
does not.)

> **Warning:** A queue spec is parsed for `kind`, `description`, `elasticity`, and
> `daemon_template` only, and inside `elasticity` only for `kind`, `min`,
> `max`, and `threshold`. An `admission:` block, or `pool` / `match` /
> `steal_delay_s` / `cost_weight` inside `elasticity`, is silently
> discarded on the way to the control plane rather than rejected. Set
> those by PATCHing the queue over HTTP — see
> [Elasticity](/get-started/elasticity.md#setting-the-knobs).

### Daemon group spec

`queues` (non-empty list of slugs) and `count` (positive integer, no
default) are both required. Everything else is optional:

| Key | Type | Meaning |
| --- | --- | --- |
| `provider` | string | Overrides the project default. |
| `instance_type` | string | e.g. `t3.medium`, `g4dn.xlarge`. |
| `image_id` | string | AMI / image id. |
| `runner` | string | `process`, `docker`, `gvisor`, … |
| `keep_alive_s` | number | Idle-exit seconds. |
| `setup` | **string[]** | Bash snippets run as setup commands on the host after the daemon registers. |
| `startup` | string | Python-environment startup script. |
| `python` | string | Python version, e.g. `"3.12"`. |
| `mounts` | list | `{ storage, mount_path, readonly? }` entries. |

Any other key inside a daemon group is a **hard error** listing the
allowed set — including `mode:`, which is not part of the compose
schema. Unknown *top-level* keys, by contrast, are silently ignored.

## The verbs

```bash
lakeshore up                    # converge to desired state (idempotent)
lakeshore up gpu                # only the `gpu` group
lakeshore up --wait             # block until every spawned daemon is active
lakeshore up --plan             # dry-run — print what would change

lakeshore ps                    # list daemons owned by this compose
lakeshore status                # current vs desired drift, read-only

lakeshore down                  # drain (default) — instances stay alive
lakeshore down --terminate      # also terminate cloud instances
lakeshore down --prune-queues   # delete declared queues that have zero members
lakeshore down --plan           # dry-run

lakeshore logs <group|label>            # multiplexed cloud-console output
lakeshore logs gpu --follow --poll-ms 5000
```

All five read `./lakeshore.yaml`; pass `-f <path>` to point elsewhere.
`up`, `down`, `ps`, and `status` all take `--json`.

## Project isolation

Each file is tagged with its `project:` slug, which is written to the
`composeProject` field on every Worker row it spawns. The compose verbs
scope to that slug, so daemons launched ad-hoc via `lakeshore daemon
launch` — which leave `composeProject` null — are invisible to `ps`,
`status`, and `down`. Same isolation pattern as docker-compose's
`com.docker.compose.project` label.

## Runtime configuration

Three fields shape the execution environment on each group:

```yaml file="lakeshore.yaml"
daemons:
  gpu:
    queues: [training]
    count: 4
    instance_type: g4dn.xlarge
    python: "3.12"
    setup:
      - apt-get update -y && apt-get install -y python3.12-venv
      - pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
    startup: "pip install -q transformers datasets"
    keep_alive_s: 1800
```

| Field | Runs |
| --- | --- |
| `setup` | On the **host**, once after the daemon registers. Delivered as setup commands, one per poll, drained FIFO. |
| `startup` | Inside the Python execution environment. |
| `python` | Target Python version for UDF execution. |

`setup` must be a **list** of strings. A single YAML block scalar is a
validation error — split the commands into list entries.

See [Daemons](/nymph/daemons.md) for how setup commands are delivered.

## What it does not do

- **Autoscaling.** `count` is a hard number, not a target. Reactive
  scaling is declared on the queue — see
  [Elasticity](/get-started/elasticity.md).
- **Live drift reconciliation.** No background process watches and
  converges. `up` runs only when you invoke it, and state is derived
  from the control plane every run (there is no lock file).
- **Cross-namespace composes.** One file, one namespace.

## See also

- [Compose a cluster](/get-started/tutorials/compose.md) — the hands-on
  walkthrough.
- [`/dev/compose`](/dev/compose) — the design doc: schema, resolution
  order, rollout phases.
- [Queues](/get-started/queues.md) — what `queues:` blocks declare.
- [Daemons](/nymph/daemons.md) — what each `daemons:` entry reconciles to.

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths/13-cloud-provider-launch

# 13 · Cloud provider launch

The "spin a real cloud VM, install nymph, register" flow. The most
complex path: cloud API call, secret decryption, bootstrap script,
binary download, and daemon registration on a fresh image.

## Exercises

- The server-side launchers — `EC2`, `GCE`, and `Kube` (the only three
  the launch routes support; `SSH` and `SLURM` return 422).
- Secret resolution at launch time: `{ "$secret": "…" }` markers in
  provider kwargs decrypted in the handler and never serialized back.
- `POST /v1/namespaces/:ns/daemons/launch` rendering the nymph bootstrap
  and handing it to the launcher as userdata / startup script.
- Outbound-only registration from a fresh instance.

## Requires

- Cloud credentials stored as a Lakeshore secret — kind `aws_keypair`
  (plaintext formatted `<access_key_id>:<secret_access_key>`) or
  `gcp_sa_json`.
- A provider registered with those credentials referenced by
  `{ "$secret": … }`.
- IAM / project permission to create, describe, and terminate instances.
- Outbound HTTPS from the instance's network, so nymph can reach the
  control plane and R2.
- `LAKESHORE_PUBLIC_URL` set on the control plane. The bootstrap script
  bakes in the URL from `LAKESHORE_PUBLIC_URL`, then
  `LAKESHORE_SERVER_URL`, then `LAKESHORE_URL`, then
  `http://localhost:$PORT` — and a daemon on EC2 pointed at localhost
  never registers.

## Verifies

Cloud provisioning end to end. When green, you can stand up ephemeral
workers on demand.

## Run

```bash
export LAKESHORE_URL=https://api.lakeshore.dreamlake.ai

# One EC2 daemon, named, waiting for it to come up:
lakeshore daemon launch --provider aws-us-east --name smoke --wait

# Four at once, into a queue, with a 30-minute idle timeout:
lakeshore daemon launch --provider aws-us-east --name gpu --count 4 \
  --queue training --runner docker --keep-alive 1800 --wait
```

With `--count > 1` the `--name` becomes a prefix (`gpu-00`, `gpu-01`, …)
and launches run at `--concurrency` (default 5, sized for the AWS
`RunInstances` rate limit, with retry on throttle). Omitting `--name`
falls back to `<provider>-<short-ulid>`.

`daemon launch` also reads project defaults from `.lakeshore` /
`.lakeshore.local`, walking up from cwd and stopping at the git root.
`--no-config` skips that discovery; `--no-setup` clears every setup
script from both sources.

Then confirm:

```bash
lakeshore daemon list
lakeshore providers instances aws-us-east
lakeshore daemon launch-log <worker-id>     # EC2 serial console
```

## Expected output

A `Worker` row appears in `daemon list` — first `pending` or
`setting_up`, then `joining` once `/hello` lands, then `active` on its
first poll. `providers instances` shows the instance transitioning
`starting → running`.

## Tearing it down

```bash
lakeshore daemon kill <id>              # terminates the cloud instance by default
lakeshore daemon kill --prefix gpu- --yes
lakeshore daemon kill <id> --no-terminate   # remove the row, leave the VM
```

> **Warning:** `daemon kill` terminates the cloud instance unless you pass
> `--no-terminate`. Getting this backwards costs either money or data.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| 502 with `awsCode: UnauthorizedOperation` | IAM. The role needs `ec2:RunInstances`, `ec2:Describe*`, `ec2:CreateTags`, `ec2:TerminateInstances`, plus `iam:PassRole` when attaching an instance profile. |
| 422 `only EC2, GCE and Kube supported` | The provider's launcher is `SSH` or `SLURM`. |
| `EC2 launch requires image_id` | No `image_id` / `ami` in the merged provider + mode kwargs. There is no default AMI. |
| Instance created, never registers | The bootstrap script failed, or it points at the wrong control plane. Check `/var/log/cloud-init-output.log` on the host and confirm `LAKESHORE_PUBLIC_URL`. |
| `secret '<name>' not found in namespace` | Secrets resolve in the **provider's own** namespace, which is not necessarily the one you called from. Providers are org-shared; secrets are not. |
| Launches succeed until they suddenly do not | A cloud account instance-count quota. Different problem. |

## Status

Manual. Costs real money — do not loop on failure.

## Next

→ [14 · Dashboard observability](/admin/happy-paths/14-dashboard-observability.md)

---

Source: https://lakeshore.dreamlake.ai/get-started/payloads

# Payload store

When work crosses the wire, both the **arguments** going in and the
**return value** coming out have to get there. Lakeshore keeps them in a
two-tier cache:

| Tier | Where | When |
| --- | --- | --- |
| Inline | On the job row, as raw bytes | `size ≤ 256 KiB` |
| S3 | Object store, reached by a presigned URL | `size > 256 KiB` |

The threshold is a control-plane env var (`INLINE_THRESHOLD_BYTES`), not
a per-queue knob — the same value across the fleet keeps the wire shape
predictable.

> **Warning:** The payload store is live on the **ExecJob** path — `POST
> /v1/namespaces/:ns/exec` and the `/jobs/:id/result` publish/subscribe
> pair, which carry `payloadInline` / `payloadRef` and `resultInline` /
> `resultRef`. The `Invocation` model declares matching `argsRef` /
> `resultRef` columns, but no route writes them yet; invocation arguments
> and results still ride inline as `argsBlob` / `resultBlob`.
> 
> Everything S3-side is also gated on `LAKESHORE_BLOBS_BUCKET`. With that
> env var unset, `bucketConfigured()` returns false and the control plane
> stays entirely on the inline-bytes wire.

## Why two tiers

The simple thing is "stick the bytes on the job row." That works for
`add(2, 3)` and breaks the moment you pass a numpy array, an image, or a
torch tensor.

The simple workaround — "spill big payloads to S3" — works, but proxying
the bytes through the control plane just to write them to S3 puts the
control plane on the bandwidth path. The fix: **clients upload and
download S3 directly via presigned URLs** the control plane mints on
demand. Bytes never touch the control plane; only the key does.

## The presign route

```
POST /v1/namespaces/:ns/payloads/presign
  { "op": "put", "size": <n>, "contentType"?, "jobId"?, "kind"? }
  { "op": "get", "key": "payloads/…" }
→ 200 { url, bucket, key, expires_at }
```

On `put` the **server** mints the key — the caller cannot choose it:

```
payloads////<job_id>/<kind>
```

`<job_id>` is the caller-supplied `jobId` (staged enqueue uploads before
the job row exists, so the key must be stable) or a freshly minted ULID.
`<kind>` defaults to `args`.

On `get` the server requires the key to start with the `payloads/`
prefix. That is the ownership gate today — it keeps callers out of the
`exec/` log-archive prefix, and the bearer-token gate in front of every
`/v1/namespaces/:ns/*` route does the rest. Stricter per-namespace
prefixing is a follow-up.

## The four flows

### Flow 1 — small payload in (`≤ 256 KiB`)

```
client                         control plane
  │ POST /v1/namespaces/:ns/exec     │
  │   { payloadInline: <base64> }    │
  │ ────────────────────────────────▶│
  │ ◀──── 202 { exec_id, worker_id } │
```

One HTTP call. The bytes ride inside the enqueue POST and are stored
inline on the job row. The daemon polls, gets the bytes back, runs the
command.

### Flow 2 — large payload in (`> 256 KiB`)

```
client                         control plane                   S3
  │ POST payloads/presign            │                          │
  │   { op: "put", size: N }         │                          │
  │ ────────────────────────────────▶│                          │
  │ ◀──── { url, bucket, key, … }    │                          │
  │                                                             │
  │ PUT <url>  (binary body) ─────────────────────────────────▶ │
  │ ◀──── 200 ──────────────────────────────────────────────────│
  │                                                             │
  │ POST /v1/namespaces/:ns/exec     │                          │
  │   { payloadRef: { bucket, key,   │                          │
  │                   size, … } }    │                          │
  │ ────────────────────────────────▶│                          │
  │ ◀──── 202 { exec_id, worker_id } │                          │
```

Three hops. `payloadInline` and `payloadRef` are mutually exclusive, and
the enqueue route rejects a `payloadRef.key` that does not start with
`payloads/` — so a client cannot point a job at an arbitrary S3 object.

### Flow 3 — the daemon pulls a job

```
daemon                         control plane                   S3
  │ POST /v1/daemon/poll             │                          │
  │ ────────────────────────────────▶│                          │
  │ ◀── { exec_id,                   │                          │
  │       payload_inline|payload_ref}│                          │
  │                                                             │
  │ (if payload_ref:)                                           │
  │ POST payloads/presign            │                          │
  │   { op: "get", key }             │                          │
  │ ────────────────────────────────▶│                          │
  │ ◀──── { url }                    │                          │
  │ GET <url> ────────────────────────────────────────────────▶ │
  │ ◀──── 200 (binary) ─────────────────────────────────────────│
```

The daemon never sees long-lived AWS credentials. The control plane
mints a short-lived presigned GET on demand; the daemon's fetch timeout
is 60 s.

### Flow 4 — the return value

Symmetric. The daemon presigns a PUT, uploads, and POSTs a `resultRef`
to `/v1/namespaces/:ns/jobs/:id/result`. The client subscribes with
`GET /v1/namespaces/:ns/jobs/:id/result?wait=N` and downloads via a
presigned GET.

> **Note:** The daemon's `ExecResultPayload` carries `result_inline` and
> `result_ref` fields, but the shell-exec path always sends both as
> `None` — a bash command has no typed return value. The bytes that flow
> back today are stdout/stderr and the exit code.

## Encoding

Payload bytes are msgpack (with the SDK's ExtType codec for numpy /
torch / PIL values). `payloadInline` is base64 on the JSON exec wire;
the daemon's own msgpack wire carries raw `bin`.

## Limits and safety

| Knob | Env var | Default |
| --- | --- | --- |
| Inline threshold | `INLINE_THRESHOLD_BYTES` | 256 KiB |
| Max single payload | `MAX_PAYLOAD_BYTES` | 100 MiB — larger requests get a 413 |
| Presigned URL TTL | `PAYLOAD_PRESIGN_TTL_S` | 900 s (15 min) |

- **Size cap, twice.** The route refuses to presign above
  `MAX_PAYLOAD_BYTES`, *and* the minted URL carries a `Content-Length`
  constraint so S3 itself rejects an oversized upload.
- **Auth.** The presign route sits behind the same namespace
  bearer-token gate as the rest of `/v1/namespaces/:ns/*`.
- **Integrity.** `sha256` is optional on a `payloadRef`. When present,
  the daemon verifies it after download; a mismatch raises
  `payload corrupt: sha256 mismatch (expected …, got …)`, short-circuits
  the exec with `exit_code = -1`, and never spawns the child process.

## What this is not

The payload store is only for job arguments and return values. Streamed
stdout/stderr is a different wire with a different bucket prefix
(`exec////<exec_id>/out`), a different retention policy,
and a different access pattern. See
[`/dev/log-storage`](/dev/log-storage).

## See also

- [`/dev/payload-store`](/dev/payload-store) — the full design spec: row
  schema, failure modes, rollout phases, and what was deliberately left
  out.
- [Storages](/get-started/storages.md) — user-registered buckets, which are
  a separate mechanism with their own credentials.
- [Daemon protocol](/nymph/protocol.md) — the daemon side of the
  poll/result wire.

---

Source: https://lakeshore.dreamlake.ai/admin/happy-paths/14-dashboard-observability

# 14 · Dashboard observability

The reactive surface. An invocation submitted from the CLI or SDK should
appear in the dashboard's Invocations view, transition through states,
and report its outcome — with no page refresh.

## Exercises

- The dashboard's long-poll loop against `GET /v1/admin/invocations`.
- Cursor-based change detection: the response carries a `cursor` and the
  next request holds until the server's cursor moves past it.
- CORS for cross-origin browser fetches (Netlify origin → Heroku control
  plane).

## Requires

- The control plane up, and the dashboard either running locally or the
  hosted one at `demo.lakeshore.dreamlake.ai`.
- At least one daemon registered and draining the queue you submit to,
  or the row will sit in `queued` forever.
- Admin auth — `/v1/admin/*` is gated on `LAKESHORE_ADMIN_TOKEN`, or
  passes freely in dev open mode when that variable is unset.

> **Warning:** `GET /v1/admin/invocations` resolves the **default** namespace and reads
> its invocations. There is no namespace query parameter. Submitting into
> some other namespace produces a row the dashboard's Invocations view will
> not show.

## Verifies

Submit → live state transitions visible in the browser without a manual
refresh.

## Run

In one terminal, submit:

```bash
export LAKESHORE_WORKSPACE=~/lakeshore-workspace
export LAKESHORE_URL=http://localhost:8080

uv run --project $LAKESHORE_WORKSPACE/lakeshore-py \
  $LAKESHORE_WORKSPACE/lakeshore-examples/09-hello-queue-udf/hello_udf.py
```

In the browser, open the dashboard and go to **Invocations**.

## Expected output

A new row appears, moves `queued → running → succeeded`, and settles
with a result preview. The delay before it appears is bounded by the
long-poll window, not by a fixed refresh interval.

## The long-poll contract

| Query param | Default | Cap |
| ----------- | ------- | --- |
| `cursor` | `0` | — |
| `timeout_s` | 30 | 60 |
| `limit` | 100 | 500 |

The request blocks until the server's `adminCursor` passes the supplied
`cursor`, or the timeout expires. The response is
`{ invocations, cursor }`; feed that `cursor` back into the next
request. Rows are ordered newest-submitted first, and each carries `id`,
`queueName`, `functionQualname`, `state`, `workerId`, the three
timestamps, and `progress`.

`queueName` is read off `runConfig.queue` — Phase A.5 removed
`Invocation.queueId`, so there is no join and no foreign key. Rows with
nothing there render as `default`.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| The row never appears | Submitted into a non-default namespace — see the callout above — or into a namespace the dashboard account is not pointed at. |
| The row appears but stays `queued` | Nothing is draining the queue. Check `lakeshore daemon list` and the daemon's `queues` column. |
| 401 on `/v1/admin/invocations` | The control plane has `LAKESHORE_ADMIN_TOKEN` set and the dashboard is not sending it. |
| CORS error in the browser console | The control plane registers `@fastify/cors` with `origin: true` (reflect any origin), so a CORS failure usually means the request never reached it — check the URL and TLS first. |
| The row appears but never updates | The long-poll hung. Look for the `/v1/admin/invocations?cursor=…` request in the Network panel; a request that returns without the cursor advancing is the tell. |

## Status

Manual.

## Next

→ [15 · Token lifecycle](/cli/happy-paths/15-token-lifecycle.md)

---

Source: https://lakeshore.dreamlake.ai/get-started/elasticity

# Elasticity

A queue's **elasticity** decides whether the control plane is allowed to
grow or shrink the fleet of daemons subscribed to it. Four named
policies cover the shapes we have seen in real workloads.

The controller is a single async loop on the control plane. Every tick
it reads each `state="active"` queue's `elasticity` object, gathers three
signals — daemon count, utilization, pending invocations — and produces
exactly one decision: `scale_up`, `scale_down`, or `tick_noop`. Every
decision, noops included, lands a row in the `ElasticityEvent`
collection.

> **Warning:** The controller starts only when the control plane is booted with
> `ELASTICITY_ENABLED=true`. The tick interval comes from
> `ELASTICITY_TICK_S` and defaults to **5 seconds**.
> 
> Each tick resolves the `default` namespace and iterates *its* queues.
> Queues in other namespaces are not scaled by the controller in this
> build.

## The four policies

| Policy | What the controller does | When to pick |
| --- | --- | --- |
| `fixed` | Nothing. Daemons are added and removed by hand. | You manage the fleet yourself or via [Compose](/get-started/compose.md). |
| `fully_elastic` | Targets `min(pending, max)` daemons, moving one step per tick. | Bursty jobs where spin-up cost is small next to runtime. |
| `pool_with_threshold` | Hysteresis on utilization — scale up when it stays above `threshold`, down when it stays below `threshold_low`. | Steady traffic with bursts; you want warm capacity. |
| `max_count` | Same target arithmetic as `fully_elastic`, framed as a hard cap. | Budget guardrails where the ceiling is the point. |

The stored values use **underscores**. The `lakeshore queues
--elasticity` flag accepts the hyphenated spelling (`fully-elastic`,
`pool-with-threshold`, `max-count`) and converts before the request;
`lakeshore.yaml` and the HTTP API accept only the underscored form.

One decision per queue per tick. The controller never launches or kills
more than one daemon per tick per queue, which caps the blast radius of
a single bad signal reading.

## Picking a policy

### `fixed`

The default when no `elasticity` object is present, and the simplest.
Workers are added with `daemon launch --queue …` and removed with
`daemon kill`. Pair it with [Compose](/get-started/compose.md) to keep the
fleet shape in source control while still naming an exact `count`.

```bash
lakeshore queues add hello --kind fifo --elasticity fixed
```

### `fully_elastic`

Target is `min(pending, max)`, floored at `min`. If the target exceeds
`daemon_count + in_flight`, the controller launches one daemon; if
`daemon_count` exceeds the target (and `min`), it kills one idle daemon.

```bash
lakeshore queues add training-burst \
  --kind fifo \
  --elasticity fully-elastic \
  --provider ec2-h100 \
  --max 32
```

### `pool_with_threshold`

Keep `min` daemons warm. When utilization — `sum(current_invocations) /
sum(max_invocations)` across members — stays at or above `threshold`
(default 0.8) for two consecutive ticks, launch one. When it stays at or
below `threshold_low` (default 0.3) for two consecutive ticks, kill one
idle daemon. In between, both counters reset and nothing happens.

```bash
lakeshore queues add training-h100 \
  --kind priority \
  --elasticity pool-with-threshold \
  --provider ec2-h100 \
  --min 2 --max 16 --threshold 0.7
```

### `max_count`

Arithmetically the same as `fully_elastic`: target is
`max(min(pending, max), min)`. The difference is intent — you set `max`
as a budget ceiling and let the pool drain to zero when work stops.

```bash
lakeshore queues add nightly-eval \
  --kind fifo \
  --elasticity max-count \
  --provider ec2-cpu \
  --max 4
```

## Scale-up requires a provider

`scaleUp` reads `queue.providerRef`, falling back to
`daemonTemplate.provider`. With neither set, the launch fails, the
decision is downgraded to `tick_noop`, and the reason recorded on the
audit row is `queue "<name>" has no providerRef / daemonTemplate.provider
— cannot launch`. Any non-`fixed` policy therefore needs `--provider`.

## Shared pools

Queues carrying the same `elasticity.pool` string are grouped and ticked
as one unit:

- **Signals aggregate.** `pending` is summed across member queues;
  workers are de-duplicated by id before counting and computing
  utilization.
- **Policy comes from the first member queue.** All members should
  declare the same `kind` / `min` / `max`.
- **Scale-up joins every member queue.** One launch, membership in all
  of them.
- **The audit row uses `pool:<name>`** as its `queueName`, so
  `queues events` on an individual member will not show pool decisions.

### Work-stealing

Work-stealing is a *poll-time* decision made by the dispatcher, not a
controller decision. When a worker polls its own queues and finds
nothing, the server resolves the sibling queues that share a pool with
them and tries those before sleeping.

### Match constraints

`elasticity.match` is a hard gate on stealing. A queue with
`match: { tags: ["gpu:h100"] }` only offers work to workers whose `tags`
contain every listed tag; `match.runners` works the same way against the
worker's `runners`. Queues with no `match` accept any worker in the pool.

### Steal delay

`steal_delay_s` is how long a worker must be idle before it will steal
from a non-native sibling. The value used is the **maximum** across the
worker's own queues. It gives cheaper native workers a head start — an
expensive GPU worker with `steal_delay_s: 5` will not grab a cheap CPU
job that a CPU worker might claim within five seconds. Default 0.

### Cost-aware scale-down

When the controller scales down it only considers **idle** workers
(`capabilities.current_invocations == 0`, or zero `running` invocations
when the daemon does not report the field). Candidates are then sorted
by cost weight descending, ties broken by oldest `lastSeenAt`.

Cost weight is read from `capabilities.cost_weight` when the daemon
reports it; otherwise a worker carrying any `gpu:*` tag is inferred as
`10` and everything else as `1`.

> **Note:** `ElasticitySpec` declares a `cost_weight` field, and it round-trips
> through the queue row, but the cull actually ranks workers using
> `Worker.capabilities.cost_weight` and the `gpu:*` tag heuristic. Setting
> `cost_weight` on a queue does not change which worker gets killed.

## Setting the knobs

Only a subset of the spec has CLI flags:

| Field | Default | CLI flag |
| --- | --- | --- |
| `kind` | `fixed` | `queues add --elasticity` |
| `min` | `0` | `queues add/patch --min` |
| `max` | unbounded | `queues add/patch --max` |
| `threshold` | `0.8` | `queues add/patch --threshold` |
| `threshold_low` | `0.3` | — HTTP only |
| `pool` | — | — HTTP only |
| `match.tags` / `match.runners` | — | — HTTP only |
| `steal_delay_s` | `0` | — HTTP only |
| `cost_weight` | `1` | — HTTP only |

For the HTTP-only fields, PATCH the whole `elasticity` object:

```bash
curl -X PATCH \
  -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"elasticity":{
        "kind":"fully_elastic","max":4,"pool":"gpu-fleet",
        "match":{"tags":["gpu:h100"],"runners":["docker"]},
        "steal_delay_s":5}}' \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/$NS/queues/training
```

`kind` is the only field the server validates; unknown keys inside
`elasticity` are stored and ignored.

The related admission knobs live on a separate object:

| Field | On | Meaning |
| --- | --- | --- |
| `max_depth` | `admission` | Depth at which `on_full` should kick in. |
| `on_full` | `admission` | `reject` or `block`. Stored, **not enforced** today. |

## Watching it work

```bash
lakeshore queues stats training-h100            # current signals
lakeshore queues events training-h100           # scale_up / scale_down rows
lakeshore queues events training-h100 --verbose # include tick_noop
```

`tick_noop` rows are hidden by default so a stable fleet does not bury
the audit log. If `events --verbose` shows nothing at all, the
controller is not running — check `ELASTICITY_ENABLED` on the server.

## Patching a live queue

The controller picks up a new spec on its next sweep — no relaunch:

```bash
lakeshore queues patch training-h100 --max 32 --threshold 0.6
```

To take a queue out of service without touching its daemons:

```bash
lakeshore queues drain training-h100      # controller skips non-active queues
lakeshore queues unarchive training-h100  # back to active
```

Because the controller only iterates `state="active"` queues, draining
is also the way to freeze scaling for one queue.

## See also

- [Scaling rules](/get-started/scaling-rules.md) — the complete rule book,
  signal by signal.
- [Queues](/get-started/queues.md) — the queue row the spec lives on.
- [Compose](/get-started/compose.md) — declarative fleet sizing when you
  want an exact `count`, not a policy.
- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — the launchers the controller calls.
- [`/dev/queues`](/dev/queues) — the controller design note.

---

Source: https://lakeshore.dreamlake.ai/cli/happy-paths/15-token-lifecycle

# 15 · Token lifecycle

The auth code path exercised end to end on a control plane that is
**not** in open mode. An admin token mints a client token, the CLI
saves it, and a Python worker consumes it from the environment.

Starting with a new server? Follow [Setting Up Lakeshore Service](https://docs.dreamlake.ai/lakeshore/setting-up-lakeshore-service) for the full setup sequence.

## Exercises

- `LAKESHORE_ADMIN_TOKEN` on the control-plane side — admin-bearer
  validation.
- `lakeshore admin tokens create` — the explicit mint path.
- `lakeshore auth login` — both the `--token` path and the
  admin-mint-on-the-fly path, saving to `auth.yml`.
- `LAKESHORE_CLIENT_TOKEN` — the env var the Python worker reads,
  independent of `auth.yml`.

## Requires

- A control plane with `LAKESHORE_ADMIN_TOKEN` set (so it is not in
  open mode). For details, see
  [Auth and secrets](/api/auth-and-secrets.md).
- `LAKESHORE_URL` **unset** while you test the token path — when it is
  set, non-`auth` commands send no bearer token at all.

## Run

```bash
export LAKESHORE_ADMIN_TOKEN=<admin-token-from-.env>

# ─── Option A: mint explicitly, then log in with the plaintext ───────
lakeshore admin tokens create cli-$USER
# prints the plaintext once, plus a ready-made `auth login` line

lakeshore auth login \
  --server http://localhost:8080 \
  --namespace dev \
  --token <plaintext-from-above>

# ─── Option B: let `auth login` mint and save in one step ────────────
# With LAKESHORE_ADMIN_TOKEN set and --token omitted, login mints a
# fresh per-namespace token itself. --name is the label base; the CLI
# appends a timestamp so re-running doesn't collide.
lakeshore auth login \
  --server http://localhost:8080 \
  --namespace dev \
  --name cli-$USER

lakeshore auth status     # expect authMode: token, namespace: dev
```

Then drive a worker from the environment alone — no `auth.yml`
involved:

```bash
unset LAKESHORE_ADMIN_TOKEN
export LAKESHORE_URL=http://localhost:8080
export LAKESHORE_NAMESPACE=dev
export LAKESHORE_CLIENT_TOKEN=$(grep '^token:' ~/.config/lakeshore/auth.yml | awk '{print $2}')

lakeshore worker once --queue default --dry-run   # prints the child argv
lakeshore worker once --queue default
```

> **Warning:** `lakeshore worker` never reads `~/.config/lakeshore/auth.yml`. The
> plane it drains has to come from `--url` / `LAKESHORE_URL`, and the
> token from `--token` / `LAKESHORE_CLIENT_TOKEN`. That is why this step
> re-exports the token explicitly rather than relying on step 1.

## Expected output

`lakeshore auth status`:

```text
Auth OK.
  config path: /Users/you/.config/lakeshore/auth.yml
  server:      http://localhost:8080
  namespace:   dev
  token:       dlk_abcd... (cli-you-2026-01-02T03-04-05)
  lastUsedAt:  2026-01-02T03:05:11.000Z
  authMode:    token
```

`worker once` starts, drains at most one job, and exits without
authentication errors.

## If it fails

| Symptom | Likely cause |
| ------- | ------------ |
| `error: LAKESHORE_ADMIN_TOKEN is required for admin commands.` | Not exported in this shell. Every `admin` subcommand hard-exits 1 without it. |
| `mint failed (401)` on `auth login` | The control plane doesn't recognize the admin token. Make sure your shell value matches what the CP read from `.env` at boot, and restart the CP after changing `.env`. |
| `whoami failed (403)` | The token is scoped to a different namespace than `--namespace`. |
| `authMode: open` | The CP booted with a blank `LAKESHORE_ADMIN_TOKEN` and enforces nothing. Set it and restart. |
| The worker authenticates as nobody | `LAKESHORE_CLIENT_TOKEN` is empty. Re-export it from `auth.yml`. |

## Status

Manual.

## Loop back

That's the full cascade. Once all 15 are green you have working
verification of every major surface. Re-run as features change.

→ Back to [Overview](/dev/happy-paths.md)

---

Source: https://lakeshore.dreamlake.ai/get-started/scaling-rules

# Scaling Rules

The elasticity controller runs a loop on the control plane. Every tick
it reads each active queue's `elasticity` spec, gathers signals, and
applies the rules below. This page is the reference for what the
controller does and why; [Elasticity](/get-started/elasticity.md) is the
narrative version.

> **Warning:** The loop starts only when the control plane is booted with
> `ELASTICITY_ENABLED=true`. Tick interval is `ELASTICITY_TICK_S`, default
> **5 seconds**. Each tick iterates the queues of the `default` namespace
> only.

## Tick lifecycle

```
partition active queues into (pooled by elasticity.pool) and (standalone)

for each standalone queue, then for each pool:
  1. Gather signals    →  daemon_count, utilization, pending
  2. Apply policy      →  scale_up | scale_down | tick_noop
  3. Execute decision  →  launch or kill exactly one daemon
  4. Write audit row   →  ElasticityEvent { action, reason, before, after }
```

A failed launch or a scale-down with no eligible victim is **downgraded
to `tick_noop`** and the failure reason is appended to the audit row's
`reason` — a broken provider never wedges the loop.

## Signal gathering

| Signal | Definition |
| --- | --- |
| `daemon_count` | Workers whose `queues` array contains the queue name. For the `default` queue, workers with an empty array count too. For pools, workers are de-duplicated by id across member queues. |
| `utilization` | `sum(capabilities.current_invocations) / sum(capabilities.max_invocations)` over those workers. `max_invocations` falls back to `1` per worker. If **no** worker reports `current_invocations`, the numerator falls back to counting `state="running"` invocations against those worker ids. `0` when there are no workers. |
| `pending` | `state="queued"` invocations whose `runConfig.queue` equals the queue name. Summed across members for a pool. |

## Policy rules

Let `in_flight` be the number of launches recorded by the
[launch tracker](#launch-tracking) for this queue.

### `fixed`

Do nothing. Every tick is a `tick_noop` with reason `fixed: no-op`.

An unrecognised `elasticity.kind` also lands here, with the reason
`unknown elasticity.kind="…" — defaulting to no-op`.

### `fully_elastic`

```
target           = max( min(pending, max), min )      # max defaults to unbounded
scale_up   when  target - daemon_count - in_flight > 0
scale_down when  daemon_count > target AND daemon_count > min
```

### `pool_with_threshold`

Hysteresis on utilization; the window is 2 consecutive ticks.

```
high = elasticity.threshold      or 0.8
low  = elasticity.threshold_low  or 0.3

utilization >= high  → consecutiveAbove += 1 ; consecutiveBelow = 0
utilization <= low   → consecutiveBelow += 1 ; consecutiveAbove = 0
otherwise            → both counters reset to 0        (dead band)

scale_up   when consecutiveAbove >= 2 AND daemon_count + in_flight < max
scale_down when consecutiveBelow >= 2 AND daemon_count > min
```

The acting counter resets to zero after a decision, so the next scale
step needs two more qualifying ticks. Hitting `max` (or `min`) with the
counter satisfied produces a `tick_noop` that names the bound, and does
**not** reset the counter.

### `max_count`

Same arithmetic as `fully_elastic`, with `max` defaulting to `0` instead
of unbounded — an unset `max` therefore scales nothing up.

```
target           = max( min(pending, max), min )
scale_up   when  target - daemon_count - in_flight > 0
scale_down when  daemon_count > target AND daemon_count > min
```

## Scale-up rules (all policies)

1. **Provider required.** `queue.providerRef` wins, falling back to
   `daemonTemplate.provider`. Neither set → launch fails → `tick_noop`.
2. **Launch body** comes from `queue.daemonTemplate`: `runners` (or a
   single `runner`, else `["process"]`), `capacity`, `cloudTags`,
   `bootstrap_script`, and `setup_scripts` mapped to setup commands.
3. **Queue membership** on the new worker is `[queue.name]`, or `[]` for
   the `default` queue.
4. **One per tick.** At most one daemon is launched per queue or pool
   per tick.

## Scale-down rules (all policies)

1. **Idle-only.** A worker is a candidate when
   `capabilities.current_invocations == 0`, or — when the field is
   absent — when it has zero `state="running"` invocations. Busy workers
   are never killed.
2. **Cost-first.** Candidates sort by cost weight descending.
   `capabilities.cost_weight` wins if present; otherwise a worker with
   any `gpu:*` tag is `10` and everything else is `1`.
3. **Then oldest.** Ties break by `lastSeenAt` ascending.
4. **One per tick.** At most one worker per queue or pool per tick.
5. **Kill is a row delete.** The default kill path deletes the `Worker`
   row; a still-running daemon that hellos again re-creates it.

## Shared pools

Queues with matching `elasticity.pool` values form a pool.

| Rule | Behavior |
| --- | --- |
| **Grouping** | Same `pool` string → one group. No `pool` → ticked individually. |
| **Signal aggregation** | `pending` summed across members; workers de-duplicated by id; utilization computed from the de-duplicated set. |
| **Policy source** | The **first** member queue's spec is canonical. Members should agree on `kind` / `min` / `max`. |
| **Scale-up** | The launched worker joins **all** member queue names (excluding `default`). Provider and template come from the first member. |
| **Scale-down** | Same idle-only / cost-first / oldest rules, over the de-duplicated worker set. |
| **Audit trail** | Events are written with `queueName = "pool:<name>"`. |

## Work-stealing rules

When a worker polls its own queues and finds no work, the dispatcher
tries sibling queues in the same pool before sleeping. This is a
poll-time decision in the request handler — the controller is not
involved.

| Rule | Behavior |
| --- | --- |
| **Eligibility** | Only `state="active"` queues sharing a `pool` with one of the worker's own queues. The worker's own queues are excluded. |
| **Match gate** | If a sibling declares `elasticity.match`, the worker's `tags` must include every entry in `match.tags` and its `runners` every entry in `match.runners`. |
| **Steal delay** | The worker must have been idle for at least `steal_delay_s` seconds, taken as the **maximum** across its own queues. |
| **Own-first** | Own queues are always tried first, every spin. Stealing is the fallback. |

## Dispatch latency

| Path | Mechanism |
| --- | --- |
| Work already queued when the worker polls | Immediate `findAndModify` claim. |
| Work arrives while the worker is long-polling | `POST /v1/producer/submit` calls `bumpQueue(queueName)`, which interrupts the idle wait. |
| Work arrives between polls | Picked up on the next poll. |

The poll's idle wait is `min(5000 ms + 0–500 ms jitter, remaining)`, and
the whole long-poll is capped at 20 seconds
(`min(body.timeout_s ?? 20, 20)`).

## Launch tracking

`LaunchTracker` prevents a double-spawn inside one launch window. When
the controller launches a daemon it records an in-flight token keyed by
queue name; later ticks subtract the in-flight count from the target.
Tokens auto-expire after **15 seconds** — long enough for the cloud
launch plus the first `/hello`, short enough that a failed launch does
not permanently consume a slot. The token is also resolved explicitly
when the launch call returns, success or failure.

## Configuration reference

| Field | On | Default | Description |
| --- | --- | --- | --- |
| `kind` | `elasticity` | `"fixed"` | Policy name. The only validated field. |
| `min` | `elasticity` | `0` | Floor for daemon count. |
| `max` | `elasticity` | unbounded (`0` for `max_count`) | Ceiling for daemon count. |
| `threshold` | `elasticity` | `0.8` | High watermark for `pool_with_threshold`. |
| `threshold_low` | `elasticity` | `0.3` | Low watermark for `pool_with_threshold`. |
| `pool` | `elasticity` | — | Shared pool name. |
| `match.tags` / `match.runners` | `elasticity` | — | Worker requirements for work-stealing. |
| `steal_delay_s` | `elasticity` | `0` | Idle seconds before stealing from siblings. |
| `cost_weight` | `Worker.capabilities` | `1` (`10` inferred for `gpu:*` tags) | Scale-down priority. Higher is killed first. |
| `providerRef` | queue row | — | Provider the controller launches under. Required for non-`fixed`. |
| `daemonTemplate` | queue row | — | `{ instance_type, image_id, runner, … }` for launches. |
| `ELASTICITY_ENABLED` | env var | `false` | Enable the controller loop. |
| `ELASTICITY_TICK_S` | env var | `5` | Tick interval in seconds. |

## See also

- [Elasticity](/get-started/elasticity.md) — concept overview and CLI examples.
- [Queues](/get-started/queues.md) — the queue primitive and lifecycle.
- [Compose](/get-started/compose.md) — declarative fleet sizing.
- Source: `src/elasticity/controller.ts` in `dreamlake-ai/lakeshore-controlplane`.

---

Source: https://lakeshore.dreamlake.ai/api/llm-readable

# LLM-Readable Docs

These docs are built to be read by agents as easily as by people. Every
page has a markdown twin, and the whole corpus is published in the
formats LLM tooling already looks for — so you can point Claude (or any
agent) at Lakeshore and have it answer accurately.

## Fetch a single page

Append `.md` to any docs URL to get the raw markdown — no nav, no chrome:

```bash
curl https://lakeshore.dreamlake.ai/get-started/quick-start.md
```

The path mirrors the HTML route exactly; the site root is `/index.md`.
Every page also advertises its markdown twin in the HTML head:

```html
<link rel="alternate" type="text/markdown" href="/get-started/quick-start.md" />
```

## The whole site, two ways

- **[`/llms.txt`](https://lakeshore.dreamlake.ai/llms.txt)** — a short,
  linked index of every page, grouped by section, each with its
  one-line description ([llmstxt.org](https://llmstxt.org) standard).
  The entry point an agent reads first to decide what to fetch.
- **[`/llms-full.txt`](https://lakeshore.dreamlake.ai/llms-full.txt)** —
  every page concatenated into one markdown file, each preceded by a
  `Source:` line naming its canonical URL. Drop the entire product into
  a context window in a single request.

## Import it as a skill

The docs are also packaged as an
[Agent Skill](https://lakeshore.dreamlake.ai/skills/lakeshore.zip) — a
`SKILL.md` index plus one markdown reference file per page. Install it
so your agent loads Lakeshore knowledge on demand:

```bash
# Claude Code: drop it into your project (or ~/.claude) skills directory
curl -L https://lakeshore.dreamlake.ai/skills/lakeshore.zip -o lakeshore.zip
unzip lakeshore.zip -d .claude/skills/
```

The archive expands to `lakeshore/SKILL.md` plus `lakeshore/reference/`,
so the command above lands it at `.claude/skills/lakeshore/`.

## How the surfaces are generated

All four surfaces come from one script, `docs/scripts/gen-llms.mjs`,
which reads `docs/pages/**/+Page.mdx` and nothing else. It runs during
`pnpm build`, after the prerender and before the search index.

Two behaviours worth knowing when you read the output:

- **MDX is flattened to plain markdown.** `import` lines are dropped,
  `> **Note:** ` becomes a `> **Note:**` / `> **Warning:**` blockquote,
>   `` is unwrapped to its children, and any other capitalized JSX
>   tag is stripped. Code fences and tables pass through untouched.
> - **Internal links are rewritten per target.** In the `.md` pages and
>   `llms-full.txt` a link to `/get-started/queues` becomes
>   `/get-started/queues.md`; in the skill it becomes
>   `reference/get-started-queues.md`. Anchors survive. Links to paths that
>   are not published pages are left alone.
> 
> Pages marked `noindex: true` in their frontmatter are internal and are
> excluded from every artifact — the `.md` twin, both `llms` files, and the
> skill.
> 
> 
> **Always current.** Every surface above is regenerated from the same
> source on each deploy, so none of them can drift from what you read on
> the site. `pnpm --filter docs check:llms` reports whether the committed
> `skills/lakeshore/` tree is stale relative to the pages.

---

Source: https://lakeshore.dreamlake.ai/nymph/daemons

# Daemons

A **daemon** is one `nymph` process registered with the control plane —
one Worker row in the database. It is a single static Rust binary; the
same binary and the same protocol run on a laptop, an EC2 instance, a
SLURM allocation, and a Kubernetes node. What differs is only *where* it
runs and *how* it got there.

For the CLI verbs that manage daemons, see
[Daemon lifecycle](/nymph/daemon-lifecycle.md). For the wire format, see
[Daemon protocol](/nymph/protocol.md).

## Getting one onto a host

| Path                              | Command                                       | Best for                                  |
| --------------------------------- | --------------------------------------------- | ----------------------------------------- |
| Hand install                      | [`install.sh`](/nymph/installation.md)           | A box you are already sitting on.         |
| SSH bootstrap                     | [`lakeshore daemon install <alias>`](/nymph/install-remote-daemon.md) | A host you can `ssh` to but the control plane cannot reach. |
| Provider launch                   | [`lakeshore daemon launch`](/nymph/daemon-lifecycle.md#launch-through-a-provider) | A VM the control plane provisions for you (EC2 / GCE / SLURM / Kube). |

Once registered, the control plane cannot tell the three apart.

> **Note:** `lakeshore worker start` supervises the **native Python queue worker**
> (`python -m dreamlake.lakeshore.daemon`) in the foreground on your own
> machine. It does not run nymph. Use it for local development against the
> Python SDK; use the paths above for fleet daemons.

## Boot sequence

```text
read config ─▶ detect host ─▶ [introspect server] ─▶ hello ─▶ poll loop ─▶ drain ─▶ exit
                                                                  │
                                                    backoff ◀─────┘ (on error)
```

1. **Config.** Read `udf-daemon.toml` once — from `--config`, else
   `/etc/dreamlake/udf-daemon.toml`, else
   `$HOME/.config/dreamlake/udf-daemon.toml`, else built-in defaults. It
   is never re-read.
2. **Host detection.** Probe the environment once on a blocking thread
   and derive extra tags and runners (see
   [Host detection](#host-detection)).
3. **Introspection.** If `[introspect] enabled = true`, bind the
   loopback status server — deliberately *before* `/hello`, so a daemon
   that cannot reach the control plane is still inspectable. A bind
   failure logs a warning and is not fatal.
4. **Hello.** `POST /v1/daemon/hello` with identity, tags, capabilities,
   versions, and (with an identity configured) the public key. A `reject`
   in the response aborts startup.
5. **Poll loop.** Long-poll, dispatch, repeat.
6. **Drain.** On `SIGTERM` / `SIGINT`, stop polling and let in-flight
   work finish. See [Shutdown](#shutdown-and-drain).

Before `/hello` returns a canonical id, the daemon uses a provisional
worker id of `{machine_id}-{ULID}` — or `{hostname}-{ULID}` when
`[daemon] machine_id` is `"auto"` or empty.

## The poll loop

Every iteration races three signals: the cancellation token, an internal
wake notification fired the instant any exec completes, and the poll
itself. The wake path exists because the control plane pops at most one
queued exec per poll — without it, back-to-back execs would stall a full
`wait_max_s` each.

**Concurrency.** Runs and execs share one task set, bounded by a
semaphore sized to `[runtime] max_invocations` (default 100; `0` means
unbounded). When zero permits are free the daemon stops long-polling
entirely and waits for one to be released, so it never claims work it
cannot start.

**Backoff.** A failed poll waits `backoff + rand(0,1) · min(backoff, 5)`
seconds — full jitter, so a fleet that lost the control plane together
does not come back in lockstep — then doubles up to `[poll] retry_max_s`.
A successful poll resets to `retry_min_s`. `lakeshore daemon reset`
queues a `reset_backoff` command to collapse a long backoff without a
restart.

### Idle policy

`[runtime] keep_alive_s` decides what happens when there is nothing to
do. The idle clock starts only when a poll returned zero commands *and*
nothing is in flight; a completing invocation restarts it.

| Value | Behaviour                                                       |
| ----- | --------------------------------------------------------------- |
| `-1`  | Pool mode. Never idle-exits. (Default for hand installs.)        |
| `0`   | Exit as soon as the idle clock starts — effectively single-job.  |
| `N`   | Exit after `N` continuous idle seconds.                          |

An idle exit routes through the same graceful-drain path `SIGTERM` uses.

### Shutdown and drain

`SIGTERM` or `SIGINT` fires one cancellation token: the poll loop stops
issuing polls, but in-flight work keeps running. The daemon then waits up
to `[shutdown] grace_s` (default 60) for natural completion. If the grace
expires, a **second** token fires; each in-flight dispatch holds a child
of it, and the runners `SIGTERM` their children and return `killed`.
Every outcome is still acked before exit.

## Runners

The runner kind comes from `run_config.runner`, falling back to
`[runtime] default_runner`. An unknown kind does **not** silently fall
back — it fails the invocation with `unknown runner kind: {other}`.

| Kind         | Isolation                                     | Notes                                              |
| ------------ | --------------------------------------------- | -------------------------------------------------- |
| `process`    | None — a child process on the host            | Default. Works anywhere Python is installed.       |
| `subprocess` | None                                          | An accepted **alias** for `process`.               |
| `docker`     | Namespaces + cgroups (`--runtime=runc`)       | Requires `run_config.image`.                        |
| `gvisor`     | User-space kernel (`--runtime=runsc`)         | A 26-line wrapper around the docker runner.        |
| `slurm`      | Whatever the cluster gives the job            | `sbatch` + `squeue` polling.                        |
| `kube`       | Pod                                           | `kubectl apply` + phase polling.                    |

A `mock` runner exists only under the `test-mock-runner` cargo feature.

### Shared workdir contract

`process`, `docker`, `gvisor`, `slurm`, and `kube` all follow the same
per-invocation contract:

- The daemon creates `{[runtime] workdir}/{invocation_id}` (default root
  `/var/lib/dreamlake/udf`).
- The daemon writes `RunBody.args_blob` to `args.msgpack` in that
  directory.
- The child writes `result.msgpack` on success, or `error.json` with
  `{ type, message, traceback }` on failure. `error.json` is surfaced as
  `"<type>: <message>"`.
- On success the workdir is deleted. **On failure it is left on disk** so
  you can inspect it.

Container runners bind-mount the workdir at `/workspace`.

### `process`

Spawns `python3 -m dreamlake.lakeshore.daemon.entry` with `cwd` set to
the per-invocation workdir, stdin null, stdout/stderr inherited into the
daemon log, and `kill_on_drop`. Environment added to the child:
`DREAMLAKE_INVOCATION_ID`, `DREAMLAKE_WORKDIR`, plus
`DREAMLAKE_PARENT_INVOCATION_ID` / `DREAMLAKE_ROOT_INVOCATION_ID` when
set.

Two escape hatches on `RunBody`:

- **`script`** — when set, the runner executes `bash -c <script>` instead
  of the Python entry. Success is decided by exit status alone; there is
  no `result.msgpack` contract and the outcome blob is empty. This is how
  `setup` commands are executed.
- **`as_user`** — wraps the spawn as `runuser -u <as_user> -- <program> …`.
  The username is a separate argv element, never interpolated into a
  shell string.

### `docker` and `gvisor`

`gvisor` is `docker` with `--runtime=runsc`; everything else — workdir
layout, image resolution, env propagation, registry auth, cancellation —
is identical. It needs gVisor installed and registered as a docker
runtime on the host.

The argv is built as:

```text
docker run --rm -i --runtime=<runc|runsc> --cidfile <workdir>/container.cid \
  --network=<network, default host> \
  -v <host_workdir>:/workspace [extra -v …] \
  [caller -e K=V …] \
  -e DREAMLAKE_INVOCATION_ID=… -e DREAMLAKE_WORKDIR=/workspace \
  [raw run_config.docker.args …] [--user <uid>:<gid>] \
  <image> python -m dreamlake.lakeshore.daemon.entry
```

Note the in-container interpreter is `python`, while the `process` runner
uses `python3` on the host.

`run_config.image` is required — without it the invocation fails
immediately with `{ type: "ImageRequired" }`. Recognised sub-keys under
`run_config.docker`: `network`, `env`, `volumes` (`"host:cont"` strings),
`args` (raw docker flags), and `registry_auth`. Anything else is ignored.

`--user` defaults to the daemon's own uid:gid *unless* the caller already
passed `--user` / `-u` in `run_config.docker.args`.

`registry_auth` is `{ server, username, password | password_env }` —
`password_env` wins over a literal `password`. Login runs
`docker login <server> --username <u> --password-stdin` with the secret
on **stdin**, never in argv or the environment, and the password is
redacted from captured stderr. Any auth failure returns
`{ type: "RegistryAuthFailed" }` and the run is not attempted.

Cancellation captures the container id via `--cidfile`, runs
`docker stop <cid>` (SIGTERM, then SIGKILL after 10 s), and kills the
local docker client. `--rm` plus `kill_on_drop` prevent leaked
containers.

### `slurm`

Writes `job.sbatch` into the workdir, runs `sbatch`, parses the id out of
`Submitted batch job 12345`, then polls `squeue -j <id> -h -o '%T'` every
`run_config.slurm.poll_interval_s` (default 5). `scancel <id>` on cancel.
`CD` maps to success; `F` / `CA` / `TO` map to failure; empty `squeue`
output means the job left the queue, so the result / error files decide.

Config keys: `partition`, `time` (default `"01:00:00"`), `gres`, `nodes`,
`ntasks-per-node`, `cpus-per-task`, `mem`, `poll_interval_s`.

### `kube`

Renders a Pod manifest, applies it with `kubectl … apply -f -` on stdin,
polls `kubectl … get pod <name> -o jsonpath='{.status.phase}'` every 5 s,
and `kubectl delete pod <name>` on cancel. Config keys: `image` (default
`python:3.12-slim`), `namespace` (default `default`), `context`, `gpu`,
`cpu`, `memory`, `poll_interval_s`.

> **Warning:** The per-invocation workdir is mounted into the Pod as a **hostPath**
> volume, which assumes the daemon and the scheduling node share a
> filesystem. That holds for single-node kind / minikube / k3d, or for a
> daemon running as a DaemonSet co-located with the workload. Multi-node
> clusters need an RWX PVC, which the source flags as a TODO.

## Host detection

At boot the daemon probes its environment once and merges the result into
the tags and runner list it announces on `/hello`. Tags are emitted as
`key=value` strings.

| Context   | Detection signal                                                                                  | Tags added                                             | Runner added |
| --------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------ |
| SLURM     | `sinfo` on `PATH` **and** `/etc/slurm/slurm.conf` exists                                           | `host=slurm`, `slurm:cluster=`            | `slurm`      |
| Kube      | `kubectl` on `PATH` **and** `kubectl version --client=false --request-timeout=2s --output=json` exits 0 | `host=kube`, `kube:context=<ctx>`                 | `kube`       |
| AWS       | IMDSv1 `GET http://169.254.169.254/latest/dynamic/instance-identity/document`                      | `host=aws`, `aws:instance-type=<t>`, `region=<r>`      | —            |
| GCP       | `GET http://metadata.google.internal/…/machine-type` with `Metadata-Flavor: Google`                 | `host=gcp`, `gcp:machine-type=<t>`                     | —            |
| Local     | fallback when no host tag fired                                                                    | `host=local`                                            | —            |

Merge rules: config tags **win** on a `key=` collision and detected tags
are appended (deduped by key); the configured `[runtime] runners` list is
authoritative and detected runners are only appended if not already
present.

## Setup commands

A `setup` command pushes a bash script at an already-running daemon,
delivered through the poll channel. It is the supported way to install
docker, gVisor, NVIDIA drivers, or anything else a host needs *after* the
daemon is up.

```bash
lakeshore daemon setup <worker-id> \
  --script ./host-setup/docker.sh --sudo --refresh-capabilities
```

The script content (not the path) is appended to the Worker row's
`pendingSetupCommands` FIFO. The daemon pops one entry per poll onto a
serial queue drained by a dedicated task, so the poll loop is never
blocked by a slow install. `--sudo` sets `as_user = "root"`;
`--refresh-capabilities` re-runs host detection after exit 0 and ships
the new capability snapshot on the next poll, which flips the Worker from
`setting_up` back to `active`.

**Two-phase provisioning.** `lakeshore daemon launch` accepts *two* kinds
of script, and the split matters:

| Phase         | Field              | When                                        | Debuggability                                     |
| ------------- | ------------------ | ------------------------------------------- | ------------------------------------------------- |
| **Bootstrap** | `bootstrap_script` | Cloud-init, as root, **before** nymph starts | Cloud-init log on the VM only. No control-plane visibility. |
| **Setup**     | `setup_commands[]` | After register, FIFO over the poll channel  | Live; the control plane records the outcome; re-runnable. |

Keep `bootstrap_script` to the bare minimum nymph needs to start.
Anything that can fail interestingly belongs in setup, where you can see
it fail and retry without re-launching the VM.

```bash file="host-setup/base.sh"
#!/usr/bin/env bash
# Cloud-init bootstrap. Runs as root, BEFORE nymph starts. Keep it tiny.
set -euo pipefail
apt-get update -y
apt-get install -y --no-install-recommends ca-certificates curl gnupg
```

```bash file="host-setup/docker.sh"
#!/usr/bin/env bash
# Post-register setup. Must be idempotent — it can be redelivered.
set -euo pipefail

if ! command -v docker >/dev/null; then
  apt-get update -y
  apt-get install -y docker.io
fi

systemctl enable --now docker
usermod -aG docker "$(id -un)" || true
docker --version
```

> **Warning:** The daemon sends `cursor: null` on every poll — there is no command
> de-duplication on the wire. A reconnect can redeliver a command the
> daemon already ran. Probe before installing, use `apt-get install -y`,
> prefer `systemctl enable --now`, and read-modify-write configs rather
> than blind-append.

## Mounts

The [`Mount` registry](/get-started/mounts.md) is **metadata only** on both
sides today. The control plane stores mount rows and validates their
config shapes, but nothing attaches them into a job's sandbox.

On the daemon side there is a content-addressed blob cache module
(`nymph/src/mounts.rs`): SHA-256-keyed, two-level directory fan-out,
atomic `rename(2)` inserts, mtime-based LRU eviction against
`[mounts] cache_max_bytes` (default 20 GiB under
`/var/lib/dreamlake/cache`). It has **no call sites yet** — it is
infrastructure waiting for the mount pipeline, not a shipping feature.

## Known gaps

- **No poll cursor.** `PollRequest.cursor` is hard-coded `null`.
- **No stdout tailing for runs.** `InvocationStatus.stdout_tail` is
  always empty. Streaming output exists for `exec` only, via `LogSink`.
- **No per-invocation cancel.** The protocol has no cancel command; the
  control plane returns 409 when you try to kill a *running* invocation.
- **Mounts are unwired**, as above.
- **`startup` (compose) is a no-op.** The `lakeshore.yaml` daemon-group
  `startup:` field is parsed by the CLI and forwarded on the launch body,
  but neither the control plane nor nymph reads it today.

## Read next

- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — install, launch, list,
  exec, update, kill, cleanup.
- [Daemon protocol](/nymph/protocol.md) — the exact wire contract.
- [Daemon configuration](/api/configuration.md) — the `udf-daemon.toml` schema.
- [Telemetry and introspection](/nymph/telemetry.md) — logs and the local
  status endpoint.
- [Compose](/get-started/compose.md) — declaring daemon groups in
  `lakeshore.yaml`.

---

Source: https://lakeshore.dreamlake.ai/nymph/daemon-lifecycle

# Daemon lifecycle

A **daemon** is one `nymph` process registered with the control plane —
one Worker row. This page is the operator's map of `lakeshore daemon …`:
how to bring one online, poke at it while it runs, upgrade it, and get
rid of it.

For what the daemon does on the host, see [Daemons](/nymph/daemons.md); for
the wire format, [Daemon protocol](/nymph/protocol.md).

## Bring one online

### SSH bootstrap

```bash
export LAKESHORE_ADMIN_TOKEN=…
lakeshore daemon install bos14
```

`<ssh-alias>` is anything `ssh <alias>` accepts. The CLI mints a
per-daemon token, SSHes in, drops the binary and config under `$HOME`,
starts the process detached, and (by default) waits up to 60 s for the
Worker row to appear. Full mechanics on
[Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md).

| Flag                       | Default                  | Meaning                                                    |
| -------------------------- | ------------------------ | ---------------------------------------------------------- |
| `--label <name>`           | the SSH alias            | Label prefix; a fresh launch ULID is appended.                          |
| `--tag <key:value>`        | (none)                   | Capability tag. Repeatable.                                |
| `--runner <name>`          | `process`                | Runner kinds advertised; the first becomes the default. Repeatable. |
| `--keep-alive-s <n>`       | `-1`                     | Idle policy: `-1` pool, `0` single-job, `N` seconds.       |
| `--nymph-base-url <url>`   | the R2 release bucket    | Override the binary download base URL.                     |
| `--binary <path>`          | (download)               | `rsync` a local ELF binary instead of downloading. Mutually exclusive with `--nymph-base-url`. |
| `--admin-token <token>`    | `$LAKESHORE_ADMIN_TOKEN` | Admin token used to mint the per-daemon token.             |
| `--no-wait`                | (waits 60 s)             | Skip the registration poll.                                |
| `--slurm`                  | off                      | Launch under `sbatch` instead of `setsid nohup`.           |
| `--slurm-partition <name>` | (none)                   | `#SBATCH --partition=` value; only meaningful with `--slurm`. |

```bash
# GPU host with tags and a 10-minute idle window
lakeshore daemon install h100-rig \
  --label gpu-rig --tag gpu:h100 --tag gpu-count:8 \
  --runner docker --runner process --keep-alive-s 600

# SLURM login node
lakeshore daemon install bos14-login --slurm --slurm-partition xeon-g6-volta
```

> **Warning:** `--slurm` runs nymph inside an `sbatch` job, so the daemon dies when the
> wall-clock runs out. Re-run `daemon install <alias> --slurm` for a fresh
> allocation — there is no auto-renewal.

Each install prints a launch ULID and a separate log path. The label is
`<prefix>-<launchId>`; under `--slurm`, the job name is
`lakeshore-nymph-<launchId>`. See [remote installation](/nymph/install-remote-daemon.md#slurm)
for config paths and commands to match a worker to its job and logs.

### Launch through a provider

When the control plane can reach the host it is provisioning, let it do
the work:

```bash
lakeshore daemon launch \
  --provider aws-prod \
  --name gpu-rig-east \
  --capacity '{"instance_type": "g5.2xlarge"}' \
  --tag gpu:h100 --queue training \
  --keep-alive 300 --wait
```

The control plane mints a bootstrap token, renders a cloud-init script
(binary into `/opt/dreamlake/bin/nymph`, config at
`/etc/dreamlake/udf-daemon.toml`, a `dreamlake-nymph.service` systemd
unit with `Restart=always`), and calls the provider's launcher. Any
`setup_commands` land on the Worker row as a FIFO, so the initial state
is `setting_up` when the queue is non-empty and `active` otherwise.

| Flag                        | Meaning                                                                                   |
| --------------------------- | ----------------------------------------------------------------------------------------- |
| `--provider <name>`         | Provider registered on the control plane.                                                 |
| `--name <text>`             | Friendly name — sets the daemon label **and** the cloud `Name` tag. With `--count > 1` it becomes a prefix (`name-00`, `name-01`, …). Defaults to `<provider>-<short-ulid>`. |
| `--count <n>`               | Launch N daemons in parallel.                                                             |
| `--concurrency <n>`         | Max in-flight launches when `--count > 1`. Default 5 (the AWS `RunInstances` rate limit); retries on throttle. |
| `--label <text>`            | Worker-row label; `--name` is usually what you want.                                      |
| `--capacity <json>`         | JSON object passed verbatim to the launcher.                                              |
| `--tag <key:value>`         | Capability tag. Repeatable. **Replaces** config tags when supplied.                        |
| `--queue <name>`            | Queue the daemon joins on first `/hello`. Repeatable. Empty means the default queue.       |
| `--runner <name>`           | Runner kinds advertised; the first becomes the default. Repeatable.                        |
| `--keep-alive <seconds>`    | Idle policy: `0` single-job, `N` seconds, `-1` pool.                                       |
| `--bootstrap-script <path>` | Cloud-init bash, runs as root before nymph. Content is read locally and sent inline.       |
| `--setup-script <path>`     | Post-register setup script. Repeatable; **appends** to the config list.                    |
| `--no-config`               | Skip `.lakeshore` discovery.                                                               |
| `--no-setup`                | Drop every setup script, from config and flags alike.                                      |
| `--wait`                    | Block until the daemon registers.                                                          |

Parallel launches are safe — each call mints its own bootstrap token, so
20 concurrent launches produce 20 distinct Workers.

```bash
seq 0 19 | xargs -P 20 -I {} lakeshore daemon launch --name fleet-{}
# or, equivalently:
lakeshore daemon launch --name fleet --count 20
```

### Project defaults — `.lakeshore`

`lakeshore daemon launch` reads defaults from a `.lakeshore` YAML file at
the project root, layered with a gitignored `.lakeshore.local`.

```yaml file=".lakeshore"
daemon:
  provider: my-ec2-pro
  runners: [docker, gvisor]
  tags: [example:docker-gvisor]
  queues: [training]
  keep_alive_s: 1800
  capacity: { instance_type: g5.2xlarge }
  bootstrap_script: ./host-setup/base.sh
  setup_scripts:
    - ./host-setup/docker.sh
    - ./host-setup/gvisor.sh
```

Discovery walks **up** from the current directory; the first level where
either file exists wins, and the walk stops at the git root (a directory
containing `.git`) or the filesystem root. Script paths resolve relative
to the **config file's directory**, not your cwd, and the CLI ships the
script *text* on the wire.

Precedence is `CLI flag > .lakeshore.local > .lakeshore > built-in
default`. Arrays **replace** rather than concatenate — the one exception
is `--setup-script`, which appends.

Every field under `daemon:` is optional: `provider`, `name`, `runners`,
`tags`, `queues` (legacy synonym `lanes`), `keep_alive_s`, `capacity`,
`bootstrap_script`, `setup_scripts`. Unknown keys are ignored.

## Inspect

```bash
lakeshore daemon list                 # every daemon in the namespace
lakeshore daemon list --json
lakeshore daemon list --watch 5       # live-refresh every 5s
lakeshore daemon show <id>            # one row: versions, tags, runners, queues
lakeshore daemon show <id> --json
```

Both accept `--watch [seconds]` (default 2; Ctrl-C exits).

A worker whose `lastSeenAt` is older than 60 s reads back as `stale` —
that is derived at read time and never stored. The persisted states are
`pending`, `setting_up`, `joining`, `active`, and `gone`.

`lakeshore daemon launch-log <id>` fetches the cloud serial-console
output for a control-plane-launched daemon (EC2 only today) — the tool of
choice when a daemon never reached its first `/hello`. Flags: `--json`,
`--tail <n>`, `--no-latest`.

There are also two local-only verbs: `lakeshore daemon status` (is a
daemon running on *this* machine?) and `lakeshore daemon stop`
(`--timeout <seconds>`, `--force`).

## Run something on a daemon

```bash
lakeshore daemon exec <daemon> nvidia-smi
lakeshore exec --queue training uname -a       # let the CP pick a worker
```

Everything after the daemon argument is the remote command — no `--`
separator, but CLI-side flags must come first. The CLI posts the command,
then long-polls the control plane until the daemon ships back
stdout/stderr/exit code.

| Flag                  | Meaning                                                              |
| --------------------- | -------------------------------------------------------------------- |
| `--id`                | Treat `<daemon>` as a literal worker id, skipping name lookup.       |
| `--timeout <seconds>` | SIGTERM after N seconds, then SIGKILL 5 s later.                     |
| `--env <K=V>`         | Extra environment variable, merged on top of the daemon env. Repeatable. |
| `--workdir <path>`    | Override the working directory.                                      |
| `--stdin <path\|->`   | Pipe a file (or the CLI's own stdin) to the command.                  |
| `--wait <seconds>`    | Total CLI wait budget. Default 600.                                  |
| `--json`              | Emit the raw envelope instead of streaming text.                     |

`lakeshore exec` (no `daemon` subcommand) is the same surface with the
daemon argument dropped: it pops an interactive picker, or skips the
picker entirely when you pass `--queue <name>` and lets the control plane
route to an eligible member.

## Push setup at a running daemon

```bash
lakeshore daemon setup <id> --script ./host-setup/docker.sh \
  --sudo --refresh-capabilities --name docker
```

`--script <path>` is required; the file's **contents** go on the wire.
`--sudo` runs it as root, `--refresh-capabilities` re-probes the host
afterwards so a freshly installed runner is advertised, `--name <label>`
is a human label, `--json` emits the envelope. See
[Daemons → Setup commands](/nymph/daemons.md#setup-commands) for the
idempotency requirements.

## Pause, unstick, upgrade

```bash
lakeshore daemon hibernate <id> --until +2h     # or an ISO-8601 timestamp
lakeshore daemon reset <id>                     # collapse a long poll backoff
lakeshore daemon update <id> --version 0.1.10
```

**Hibernate** accepts a relative duration (`+30s`, `+5m`, `+1h`, `+3d`)
or an ISO-8601 timestamp; a deadline in the past is a 400. The daemon
stops polling until the deadline, then re-enters the poll loop from
scratch.

**Reset** queues a `reset_backoff` command so a daemon stuck in
exponential backoff retries at `retry_min_s` on its next poll — cheaper
than a kill-and-relaunch.

**Update** queues an over-the-air binary update. With `ota-self-replace`
baked in (the default build), the daemon `exec()`s into the new binary —
no manual restart.

| Flag                    | Meaning                                                                            |
| ----------------------- | ---------------------------------------------------------------------------------- |
| `--version <v>`         | Target version. Omitted → the namespace's `target_nymph_version`.                  |
| `--all` / `--prefix <t>`| Fan out to every daemon, or every daemon whose label starts with `<t>`.             |
| `--yes`                 | Skip the confirmation prompt required by `--all` / `--prefix` on a TTY.             |
| `--url <u>`             | Override the download URL (the control plane normally derives it).                 |
| `--sha256 <h>`          | Override the checksum (normally fetched from the `<url>.sha256` sidecar).          |
| `--add-queue <name>`    | Add the daemon to a queue. Repeatable. Errors if the queue does not exist.          |
| `--remove-queue <name>` | Remove the daemon from a queue. Repeatable.                                         |

Queue-only edits skip the OTA path entirely and do not need `--version`.

The fleet-wide target is set separately:

```bash
lakeshore daemon target-version                 # show it
lakeshore daemon set-target-version 0.1.10      # set it
lakeshore daemon set-target-version --clear
```

Daemons with `[runtime] auto_update = true` converge on it on their own;
everyone else waits for an explicit `daemon update`.

## Kill

```bash
lakeshore daemon kill <id>
lakeshore daemon kill <id> --no-terminate       # leave the cloud instance up
lakeshore daemon kill --prefix fleet --yes      # bulk kill by label prefix
```

By default the control plane also terminates the cloud instance using
the provider credentials it holds; `--no-terminate` removes only the
Worker row. `--prefix` is mutually exclusive with a positional id and
prompts for confirmation unless you pass `--yes`.

Killing the row does not reach into the daemon process itself. A pool
daemon whose row disappeared keeps polling until you stop the process
yourself (`ssh HOST pkill nymph`) or its host goes away.

## Cleanup

```bash
lakeshore daemon cleanup                          # default 24h, prompts
lakeshore daemon cleanup --older-than 2h --dry-run
lakeshore daemon cleanup --older-than 5d --yes
lakeshore daemon cleanup --older-than 6h --json
```

Deletes every Worker row whose `lastSeenAt` is older than the threshold.

| Input | Parsed as   |
| ----- | ----------- |
| `30m` | 30 minutes  |
| `2h`  | 2 hours     |
| `5d`  | 5 days      |
| `24`  | 24 hours (a bare number is hours) |

The default is **24h** and the server cap is **168h** (one week) —
anything above that, or zero/negative, is rejected with a 400. Without
`--dry-run` and without `--yes` the CLI prompts before deleting.

Server side this is `POST /v1/namespaces/:ns/workers/cleanup` with
`{ olderThanHours, dryRun }`, returning `{ count, deleted: [...] }`.

## Common flows

**Clean up after an experiment.**

```bash
lakeshore daemon list | grep <label>
lakeshore daemon kill <id>
lakeshore daemon cleanup --older-than 1d
```

**Spot-check before pruning.**

```bash
lakeshore daemon cleanup --older-than 6h --dry-run --json | jq '.deleted'
```

**A daemon launched but never registered.**

```bash
lakeshore daemon launch-log <id> --tail 100
```

## Read next

- [Bootstrap a daemon on a remote host](/nymph/install-remote-daemon.md) —
  what `daemon install` runs on the far side.
- [Daemons](/nymph/daemons.md) — runners, workdir contract, host detection.
- [Daemon protocol](/nymph/protocol.md) — the commands these verbs queue.
- [Completion](/cli/completion.md) — tab-completion for worker ids and SSH
  aliases.

---

Source: https://lakeshore.dreamlake.ai/nymph/install-remote-daemon

# Bootstrap a daemon on a remote host

The most common question after the control plane is up: *I have a host I
want jobs to run on, but the control plane can't reach it. How do I get a
daemon there?*

The daemon never needs to be reachable — it long-polls **outbound** to
the control plane. Anywhere you can `ssh` from your laptop, you can drop
a daemon and walk away.

## When to use this

- **HPC login node** behind a VPN your laptop is on but your hosted
  control plane is not.
- **On-prem box** with no public IP.
- **A subnet reachable only from a jump host.**
- **Quick experiments** — a spare laptop, a colleague's dev box, a
  hand-started instance.

If the control plane *can* reach the host — a normal EC2 or GCE instance
you provision through a Provider — use
[`lakeshore daemon launch`](/nymph/daemon-lifecycle.md#launch-through-a-provider)
instead. That path gets you a systemd unit, cloud-init bootstrap, and
serial-console logs.

## The command

```bash
export LAKESHORE_ADMIN_TOKEN=…
lakeshore daemon install bos14
```

The full flag surface lives on
[Daemon lifecycle → SSH bootstrap](/nymph/daemon-lifecycle.md#ssh-bootstrap).
Exit codes: `0` success, `1` the ssh/rsync/mint step failed or the wait
timed out, `2` a configuration error (no admin token, no saved auth).

> **Warning:** The launch-specific paths below require the
> [unique-launch installer fix](https://github.com/dreamlake-ai/lakeshore/pull/25).
> Older versions print no launch ID, share `~/.local/share/nymph.log`, and may
> write flat TOML that nymph cannot parse. Upgrade before running multiple
> installs against the same host. This change does not stop older processes or
> move their files.

## What it does

**Laptop side, step 1 — mint a token.** `POST /v1/namespaces/:ns/tokens`
with your `LAKESHORE_ADMIN_TOKEN` as the bearer. The token name is
`daemon-<sanitized-alias>-<launchId>`, so re-installing the same host
never collides. The plaintext is captured in memory and never written
locally.

**Laptop side, step 2 (only with `--binary`) — stage a local build.**
The path is validated as a real, readable file whose first four bytes are
the ELF magic `\x7fELF` (a Mach-O build fails here with a clear error),
then `rsync -av --progress` copies it to `<alias>:/tmp/nymph-<launchId>`.

**Remote side.** The CLI spawns `ssh -T -o BatchMode=yes <alias> bash -s`
and writes a script to its stdin. That script:

1. Records the SSH user (`whoami`) — everything installs under that
   account's `$HOME`; nothing runs as root.
2. Creates a private `~/.local/share/nymph/launches/<launchId>/` directory
   with mode `0700`. A repeated ID is refused rather than overwriting files.
3. Best-effort `sudo -n usermod -aG docker "$INSTALL_USER"`, so the
   daemon can reach the docker socket without sudo. Never fails the
   install if sudo or the group is missing.
4. Detects the target triple: `uname -s` → `unknown-linux-gnu` /
   `apple-darwin`, `uname -m` → `x86_64` / `aarch64`. Anything else is a
   hard error.
5. Downloads `<base>/dreamlake/nymph/latest/nymph-<triple>` into that
   directory as `nymph` and marks it executable — or, with `--binary`,
   moves `/tmp/nymph-<launchId>` there instead.
6. Writes the minted token beside it as `token`, with mode `0600`.
7. Writes sectioned `udf-daemon.toml` beside the binary and token.
8. Changes into the launch directory and starts `./nymph --config udf-daemon.toml`
   through `setsid nohup`, with stdin disconnected and output redirected to
   `~/.local/share/nymph-<launchId>.log`. With `--slurm`, `sbatch` receives
   `--job-name=lakeshore-nymph-<launchId>`, the same log path, and
   `--chdir` pointing to the private launch directory.

**Laptop side, step 3 — wait.** Unless you passed `--no-wait`, the CLI
polls `GET /v1/namespaces/:ns/workers` every 2 s for up to 60 s, looking
for a row whose label matches **and** whose state is `active`.

Nothing on the remote host needs a package install, a systemd unit, or
root. The binary is self-contained and everything lives under `$HOME`.

The generated config uses nymph's `[server]`, `[daemon]`, and `[runtime]`
sections. The worker label is `<label-prefix>-<launchId>`; the prefix defaults
to the SSH alias, or comes from `--label`. Nymph preserves this explicit label
when registering, so the launch ID reaches the worker row unchanged.

```toml file="~/.local/share/nymph/launches/<launchId>/udf-daemon.toml"
[server]
url = "https://api.lakeshore.dreamlake.ai"
token_file = "token"
namespace = "default"

[daemon]
label = "bos14-<launchId>"
tags = ["installed-from-mac"]

[runtime]
runners = ["process"]
default_runner = "process"
keep_alive_s = -1
```

`token_file` is relative to the process working directory. Both generated
launch commands set that directory; a manual restart must do the same:

```bash
cd ~/.local/share/nymph/launches/<launchId>
./nymph --config udf-daemon.toml
```

See [Daemon configuration](/api/configuration.md) for every key.

## Bootstrap stays minimal — install runtimes afterwards

The SSH heredoc does the absolute minimum to get `nymph` polling. To add
docker, gVisor, an NVIDIA driver, or anything else the daemon should
advertise as a runner, push them as **post-register** setup scripts:

```bash
lakeshore daemon setup <worker-id> \
  --script ./host-setup/docker.sh --sudo --refresh-capabilities
```

The daemon picks each one up on its next long-poll, runs it, and — with
`--refresh-capabilities` — ships back a re-probed capability list so the
scheduler starts routing docker work to it.

Baking runtime installs into the SSH heredoc would be large, hard to
debug (no live log stream until nymph is up), and impossible to retry
without re-running the whole bootstrap. Get nymph up first; install
runtimes from the live control-plane channel. Same two-phase model as
[`daemon launch`](/nymph/daemons.md#setup-commands).

## SLURM

On a login node you usually do not want a long-lived process on the login
node itself. `--slurm` wraps the launch in `sbatch` so the daemon runs as
a regular cluster job: it survives login-node restarts, inherits the
partition's resource limits, and counts against fair-share like anything
else.

```bash
lakeshore daemon install bos14-login --slurm --slurm-partition xeon-g6-volta
```

One ULID is minted per install, including installs without `--slurm`.
The CLI prints it and the log path. For an allocation, use that ID to join
all three views:

| Surface | Name |
| --- | --- |
| Worker label | `<alias-or-label>-<launchId>` |
| Slurm job name | `lakeshore-nymph-<launchId>` |
| Log | `~/.local/share/nymph-<launchId>.log` |

```bash
# On the cluster; copy the ID printed by the installer.
launch_id=01K4G8AX000000000000000001
squeue --name="lakeshore-nymph-$launch_id" -o '%.18i %.48j %.10T'
tail -f "$HOME/.local/share/nymph-$launch_id.log"

# On the machine with CLI auth:
lakeshore daemon list | grep "$launch_id"
```

Two queued allocations keep separate binaries, configs, and tokens. Starting
the second install cannot change the first job's label while it waits.
A new install creates another daemon; it does not replace or stop an old one.

The daemon dies when the allocation's wall-clock expires. Re-run to start a
fresh allocation. Launch directories and logs remain for inspection; remove a
launch's files only after its process or Slurm job has stopped.

## Troubleshooting

| Symptom                                    | Likely cause                                                                                          |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `daemon install` needs `LAKESHORE_ADMIN_TOKEN` | No admin token in the environment. Export it or pass `--admin-token`.                              |
| Exits `2` before SSHing                    | No saved control-plane auth. Run `lakeshore auth login --server <url>` first.                          |
| Never registers within 60 s                | `ssh <host> tail ~/.local/share/nymph-<launchId>.log`. Check the server URL, outbound HTTPS, and token. A Slurm job may still be pending in the queue. |
| `unsupported arch` / `unsupported OS`      | The host is not one of the published targets. Cross-compile locally and use `--binary`.                |
| `--binary path does not look like an ELF executable` | You built for macOS. Cross-compile for `x86_64-unknown-linux-gnu` (or the host's triple).      |
| `rsync` exits non-zero                     | `rsync` is not installed on one side, or the SSH alias does not resolve for rsync's transport.          |

## Read next

- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — every `lakeshore daemon`
  verb with flags.
- [Daemons](/nymph/daemons.md) — runners, host detection, setup commands.
- [Installation](/nymph/installation.md) — the standalone `install.sh` path.
- [Daemon configuration](/api/configuration.md) — the `udf-daemon.toml`
  schema.

---

Source: https://lakeshore.dreamlake.ai/get-started/tunnels

# Tunnels

A **Tunnel** is a stored network passage the launcher will bring up
before reaching a Provider. Today the only accepted kind is
`wireguard` — supplied either as a wg-quick INI string or as the
structured `{ interface, peers }` equivalent. Providers point at a tunnel
by name.

> **Warning:** Tunnel storage — add / list / show / update / remove, plus attaching one
> to a Provider — is live. **Activation** is not: nothing in the control
> plane invokes `wg-quick up` before SSH'ing to a provider. You can
> register tunnels and attach them today; the launcher wire-up is a
> follow-up.

## CLI

```bash
# Register from a wg-quick config file.
lakeshore tunnels add ny-lab \
  --kind wireguard \
  --config-file ~/wg/ny-lab.conf

# …or from an inline INI string.
lakeshore tunnels add ny-lab \
  --kind wireguard \
  --conf '[Interface]
PrivateKey = …
Address = 10.0.0.2/32

[Peer]
PublicKey = …
Endpoint = vpn.example.com:51820
AllowedIPs = 10.0.0.0/24'

lakeshore tunnels list [--json]
lakeshore tunnels show ny-lab [--json]
lakeshore tunnels remove ny-lab      # refuses while a provider references it
```

`--kind` defaults to `wireguard`, which is also the only accepted value.
The name must match `^[a-z0-9][a-z0-9-]*$`.

## Attaching to a Provider

```bash
lakeshore providers update bos14 --tunnel ny-lab     # attach
lakeshore providers update bos14 --tunnel ""         # detach
```

Tab-completion on `--tunnel ` lists registered tunnels — see
[Completion](/cli/completion.md).

## Storage shape

The server treats `config` as opaque: it does not validate wg-quick
keys. Both shapes round-trip:

```json
// structured — what the CLI's INI parser emits (camelCase keys)
{
  "interface": { "privateKey": "…", "address": ["10.0.0.2/32"] },
  "peers": [
    {
      "publicKey": "…",
      "endpoint": "vpn.example.com:51820",
      "allowedIPs": ["10.0.0.0/24"]
    }
  ]
}

// pass-through INI
{ "conf": "[Interface]\nPrivateKey = …\n…" }
```

`tunnels add --config-file *.conf` parses the INI into the structured
shape; `--config-file *.yaml` or `*.json` is uploaded as-is. The parser
camelCases keys (`AllowedIPs` → `allowedIPs`, `PreUp` → `preUp`, and
all-caps keys like `DNS` → `dns`), splits `Address`, `DNS`, and
`AllowedIPs` on commas into lists, and keeps integer values
(`ListenPort`, `PersistentKeepalive`) as numbers. An unknown section
header is a parse error.

## Secret references

Private keys in a wg-quick config are sensitive. Store the key as a
Secret and reference it with a `$secret` marker:

```json
{
  "interface": {
    "privateKey": { "$secret": "wg-ny-lab-private" },
    "address": ["10.0.0.2/32"]
  },
  "peers": [ … ]
}
```

```bash
cat ~/wg/ny-lab.key | lakeshore secrets add wg-ny-lab-private --kind opaque
lakeshore tunnels add ny-lab --kind wireguard --config-file ./ny-lab.json
```

Every referenced secret must already exist in the same namespace — the
server rejects the write with a 422 otherwise.

> **Warning:** The reference format is the JSON object `{"$secret": "<name>"}`. A bare
> `PrivateKey = $secret:wg-ny-lab-private` line inside a `.conf` file is
> parsed as a literal string — the INI parser does not convert it, and the
> secret walker only recognises the object form. Supply a structured
> YAML/JSON config when you want a `$secret` reference.

## HTTP

```
POST   /v1/namespaces/:ns/tunnels
GET    /v1/namespaces/:ns/tunnels
GET    /v1/namespaces/:ns/tunnels/:name
PATCH  /v1/namespaces/:ns/tunnels/:name
DELETE /v1/namespaces/:ns/tunnels/:name
```

## Read next

- [Providers](https://docs.dreamlake.ai/lakeshore/providers) — attaching a tunnel to a provider.
- [Secrets](/api/auth-and-secrets.md#secrets) — storing WireGuard private
  keys.
- [Mounts](/get-started/mounts.md) — the parallel registry for shared
  filesystems, with the same metadata-only status.
- [Completion](/cli/completion.md) — `--tunnel ` dynamic completion.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/rewrite

Branch **`feat/native-wire-udf`** · rewrite of the lakeshore Python SDK from
first principles. msgpack (the library) and the `Storage` broker survive;
everything above them is new. No zaku code — its *lessons* are encoded as
conformance tests.

## Progress

| Phase | What | State |
|---|---|---|
| 0 | `wire.py` — ExtType codec, versioned `Envelope`, `Frame` + reader | ✅ |
| 1 | `context.py` — `dls.run` seam, string-key returns, key contract, lazy root | ✅ |
| 2 | `dispatch.py` minimal plane ✅ · `functions.py` native `@udf` ✅ · Queue/SyncQueue ✅ · httpx transport ✅ · worker loop ✅ · legacy deletion ✅ | ✅ |
| 3 | Real streaming (Node CP Redis-Streams journal) · e2e + examples rework | ⬜ |
| 4 | dreamlake collections merge + dispatch-plane split | ⬜ |

**Tests: 113 passed** (SDK unit; legacy tests deleted with the legacy
surface) · control plane baseline 599 ✅ · dreamlake-server 108/109
(1 pre-existing failure, documented).

## The locked design, in one breath

Zero new value types — data UDFs return **string keys**; file bytes never
ride the wire (worker↔storage side channel via `dls.run.read/write`); the
function's own shape picks execution (sync/async/generator/async-gen) and
transfer timing (return = upload-then-return, async-gen = stream partial
manifests); **no Future** — the durable handle is the invocation id;
`Queue` async-first with `SyncQueue` sibling; local tier imports nothing
distributed; every dispatch transition is one guarded atomic UPDATE
(fencing, idempotent submit, counts==contents by construction).

## Documents

- [Native API + type system design](/python-sdk/rewrite/design.md) — the synthesized
  design, dual-ergonomics audit, and concept-by-concept zaku comparison.
  *(Historical note: its `Ref` recommendation was superseded by the
  zero-types / string-keys decision recorded here and in the API pages.)*
- [Usage surface & test plan](/python-sdk/rewrite/test-plan.md) — 113 CP
  routes × 7 consumers × 3 encodings, and the automatable test strategy.

## New modules (API)

- [`wire`] — codec, envelope, frames
- [`context`] — the `dls.run` I/O seam
- [`dispatch`] / [`http_dispatch`] — the dispatch planes
  ([API page](/python-sdk/dispatch.md))
- [`functions`] — the native `@udf` ([API page](/python-sdk/udf.md))
- [`queues`] — Queue / SyncQueue / Topic ([API page](/python-sdk/queue.md))
- [`worker`] — the claim-and-execute loop

The user-facing [Python SDK section](/python-sdk.md) now documents this
native surface.

---

Source: https://lakeshore.dreamlake.ai/python-sdk/rewrite/design

Synthesis of the three proposals into one buildable design. Where the three converged (ExtType codec, one wire path, Redis-Streams journal, renamed queue verbs, gather-over-futures) I take the convergent answer. Where they split — the data-UDF file story — I resolve it with the dual-ergonomics filter as the tie-breaker, and say exactly why.

---

## 1. Recommended shape + guiding principles

### The stance on new runtime types: **exactly ONE**, and it is a `pathlib.Path` subclass

The three proposals split precisely here:

- **primitives-only** → *zero* types; a bare `Path` is the file tag, provenance kept in a module-global `_REFS: dict[str, FileRef]` keyed by local-path string.
- **annotation-driven** → *zero* runtime types; the return **annotation** (`File`/`Dir`/`Tuple(...)`) drives encoding; the wire carries **bare key strings**.
- **one-marker-type** → *one* type, `Ref`, a `Path` subclass that carries its own `(storage, key, is_dir)`.

The user's constraint is "default to NO new type; any new runtime type must justify itself against a stdlib primitive." The **dual-ergonomics lens is a hard filter**, and it eliminates the two zero-type options on footgun grounds:

- The `_REFS` global registry is a **hidden aliasing contract** — provenance lives in a process-global keyed by an ephemeral `/tmp/...` string, invisible to the reader, unserializable, and wrong the moment two artifacts collide on a scratch path. The lens explicitly bans "hidden ordering/aliasing contracts."
- The bare-key-string wire is **stringly-typed magic** — the caller receives `"process/poses.json"`, a plain `str` an agent will try to `open()` and get a `FileNotFoundError`, and a *generic* queue consumer can only interpret it if a schema fingerprint rides in the envelope. The lens explicitly bans "stringly-typed magic."

`Ref` is the minimal thing that carries wire identity **without** a hidden registry and **without** stringifying semantics. It passes the filter that the stdlib primitive fails:

```python
class Ref(pathlib.Path):        # concrete PosixPath/WindowsPath subclass
    key: str                    # prefix-relative, e.g. "process/poses.json"
    storage: str | None         # storage binding; None => purely local
    is_dir: bool = False
```

**Why it justifies itself against `Path` (the required justification):** a plain `Path` cannot answer the one question the wire must answer — *"do these bytes need to travel, and under what prefix-relative key?"* A worker-local `Path("/tmp/w7/poses.json")` is meaningless to the caller; a returned `str` is indistinguishable from data. `Ref` answers it as **instance state you can read** (`ref.key`, `ref.storage`) rather than as a side-table. And because it **is** a `Path`, it is invisible where it counts: `open(ref)`, `np.load(ref)`, `ref / "rgb" / f"{i}.png"`, `os.fspath(ref)`, and every `def f(x: str | Path)` signature accept it unchanged.

Everything above one file stays **stdlib primitives**:

| Shape | Type | not |
|---|---|---|
| one file / one directory | `Ref` (`is_dir=True` for a dir) | ~~FileBundle~~ |
| N **named** outputs | `dict[str, Ref]` | ~~FileBundle.rgb~~ |
| N **positional** outputs | `tuple[Ref, ...]` | ~~Tuple(...)~~ |
| a **stream** of outputs | generator / async-gen yielding `Ref` | ~~Bundles~~ |

So: **one new type (`Ref`), everything else is `dict` / `tuple` / `list` / generator.** The rejected `FileBundle`/`FileRef`/`Bundles`/`Tuple(...)` sketch is dropped in full.

### The other load-bearing principles (all three agreed; adopted verbatim)

1. **msgpack the library stays; everything above it is fresh.** A native `ExtType`-code codec replaces zaku's `{"ztype":...}` dict overlay and `_greedy` marker. `zdata.py` (the "Vendored from zaku" file) is deleted.
2. **One codec for args, kwargs, results, and stream frames.** This deletes the two-`@udf` split and the `msgpack.packb(result)` result-path bug (arrays/tuples/`Ref` now survive as returns, not just as args).
3. **Bytes move at the seam, never in the codec.** `pack(Ref)` emits `(storage, key, is_dir)` metadata **only**. Upload/download happens in the runner, the one place a `Storage` handle exists. This is the cleanest idea across the three and removes the codec's dependency on credentials entirely.
4. **Streaming is real:** a per-invocation Redis-Streams journal with an explicit `END`/`ERROR` frame and a monotonic `seq`, replacing zaku's EOF-only termination and lakeshore's fake `subscribe_stream` (loop + `break`).

### How the design serves dual ergonomics

- **Greppable, guessable verbs:** `submit` / `claim` / `ack`, `publish` / `subscribe`, `call` / `stream`, `gather` / `as_completed` / `map`. An agent's first guess is the real method.
- **`Ref` is a `Path`, so there is no new API to learn** to read a file — the agent uses `open`/`np.load` it already knows.
- **Introspectable returns:** `-> Ref`, `-> dict[str, Ref]`, `-> AsyncIterator[Ref]` are native hints an agent reads directly; no custom marker vocabulary to construct.
- **No footguns:** no consume-once (a `Ref` is immutable and re-readable), no hidden registry, no stringly-typed keys, no `_greedy` mode to forget.
- **Errors state the fix:** the codec raises `wire: no codec for  (register one via wire.register)`; the runner raises `Ref "<key>" was returned but never created via dls.run.write(...)`.

---

## 2. Native TYPE / CODEC layer — `dreamlake/lakeshore/wire.py`

Replaces `zdata.py` entirely.

### 2.1 Leaf tagging — `msgpack.ExtType` codes via `default` / `ext_hook`

```python
# wire.py
import msgpack
from dataclasses import dataclass
from typing import Any, Callable, Iterator

@dataclass(frozen=True)
class Codec:
    code: int                                # 0x01..0x7f reserved ext space
    name: str
    match:  Callable[[Any], bool]            # predicate on a live object
    encode: Callable[[Any], bytes]           # obj  -> ext-body bytes
    decode: Callable[[bytes], Any]           # bytes -> obj

_CODECS: list[Codec] = []                    # match-ordered
_BY_CODE: dict[int, Codec] = {}

def register(codec: Codec) -> None:
    _CODECS.append(codec); _BY_CODE[codec.code] = codec

def _default(obj: Any) -> msgpack.ExtType:
    for c in _CODECS:
        if c.match(obj):
            return msgpack.ExtType(c.code, c.encode(obj))
    raise TypeError(
        f"wire: no codec for {type(obj).__name__}; register one via wire.register(...)"
    )

def _ext_hook(code: int, data: bytes) -> Any:
    c = _BY_CODE.get(code)
    return c.decode(data) if c else msgpack.ExtType(code, data)   # forward-compat

def pack(value: Any) -> bytes:
    return msgpack.packb(value, default=_default, use_bin_type=True, strict_types=True)

def unpack(buf: bytes) -> Any:
    return msgpack.unpackb(buf, ext_hook=_ext_hook, raw=False)
```

`strict_types=True` is load-bearing: it forces `tuple` through `_default` (code `0x05`) instead of silently coercing it to a `list` — so `(rgb, depth)` comes back a tuple.

### 2.2 Code table (stable integers; deferred imports preserved)

| code | Python type | ext body (`msgpack.packb`) | decode |
|---|---|---|---|
| `0x01` | `numpy.ndarray` | `[dtype_str, shape, raw_bytes]` | `frombuffer(...).reshape(...)` |
| `0x02` | `torch.Tensor` | `[dtype_str, shape, raw_bytes]` (`.detach().cpu().numpy()`) | `torch.from_numpy(buf.copy())` |
| `0x03` | `PIL.Image.Image` | `[format or "PNG", raw_bytes]` | `Image.open(BytesIO(...))` |
| `0x04` | **`Ref`** | `pack({"s": storage, "k": key, "d": is_dir})` — **metadata only, no file bytes** | reconstruct `Ref`; runner resolves/downloads on first open |
| `0x05` | `tuple` | `pack([*t])` | `tuple(...)` |
| `0x06` | plain `pathlib.Path` | `str(p)` | `Path(...)`; **runner rejects a non-`Ref` local Path on a remote hop** (see §6) |

numpy/torch/PIL predicates try-import and return `False` on `ImportError` (exactly as `zdata.py` does today). Each is one `register(Codec(...))` call, so a downstream package can add its own leaf type without editing `wire.py`.

### 2.3 Envelope — typed, versioned, no marker-in-data

```python
# wire.py
@dataclass
class Envelope:
    v: int                       # wire version — lets the format evolve
    id: str                      # invocation_id (ULID)
    fn: dict                     # {module, qualname, version, source|pickle}
    prefix: str                  # item root -> becomes dls.run.prefix
    args: bytes                  # wire.pack(tuple(args))
    kwargs: bytes                # wire.pack(dict(kwargs))
    cfg: dict                    # {mode, timeout_s, resources, image, retry}
    parent: str | None = None

    def to_bytes(self) -> bytes: ...
    @classmethod
    def from_bytes(cls, b: bytes) -> "Envelope": ...
```

An explicit `v` field replaces zaku's `_greedy` boolean smuggled into user data. `args`/`kwargs` are a nested `wire.pack` so numpy/`Ref` arguments are codec-tagged; the body/meta split means a generic worker can read `prefix`/`fn`/`cfg` without decoding the payload.

### 2.4 Frame — one shape for scalar results, item streams, and chunked binary

```python
# wire.py — Frame = packed [kind, seq, body, meta]
VALUE, CHUNK, BEGIN, END, ERROR = 0, 1, 2, 3, 4

@dataclass
class Frame:
    kind: int
    seq: int = 0                 # monotonic per invocation; the journal cursor
    body: Any = None             # codec value (VALUE) | raw bytes (CHUNK) | None
    meta: dict | None = None     # {size, content_type, name} on BEGIN; {state, error} on END

    def to_bytes(self) -> bytes:            # single pack of the whole array
        return pack([self.kind, self.seq, self.body, self.meta])
    @classmethod
    def from_obj(cls, o: list) -> "Frame":
        return cls(o[0], o[1], o[2], o[3])

class FrameReader:
    """Feed raw chunks; yield decoded Frames. msgpack length-prefixes make each
    frame self-delimiting over ONE Unpacker — no separator byte."""
    def __init__(self) -> None:
        self._u = msgpack.Unpacker(raw=False, ext_hook=_ext_hook)
    def feed(self, chunk: bytes) -> Iterator[Frame]:
        self._u.feed(chunk)
        for o in self._u:
            yield Frame.from_obj(o)

class FrameWriter:
    def value(self, obj, seq=0)     -> bytes: return Frame(VALUE, seq, obj).to_bytes()
    def chunk(self, data, seq)      -> bytes: return Frame(CHUNK, seq, data).to_bytes()
    def begin(self, meta, seq=0)    -> bytes: return Frame(BEGIN, seq, None, meta).to_bytes()
    def end(self, meta=None)        -> bytes: return Frame(END, -1, None, meta).to_bytes()
    def error(self, exc)            -> bytes: return Frame(ERROR, -1, None, {"error": repr(exc)}).to_bytes()
```

`body` is packed once as part of the frame array, so a `VALUE` frame carries a `Ref`/ndarray directly and `ext_hook` decodes it in place — no double-pack.

---

## 3. Native `@udf` — `dreamlake/lakeshore/decorator.py`

One decorator, four body kinds, detected once at decoration; args **and** results share `wire.pack`.

```python
class UDF(Generic[T]):
    def __init__(self, fn, defaults, *, remote_by_default: bool):
        self.is_async     = inspect.iscoroutinefunction(fn)
        self.is_gen       = inspect.isgeneratorfunction(fn)
        self.is_async_gen = inspect.isasyncgenfunction(fn)
        self.ret          = typing.get_type_hints(fn).get("return")  # introspectable schema

    def __call__(self, *a, **k) -> Any:  ...   # dispatch per the matrix below
    def local(self, *a, **k) -> T: ...          # always in-process; still commits under a local prefix
    def submit(self, *a, **k) -> "Future[T]": ...        # non-blocking remote handle
    def stream(self, *a, **k): ...              # force Stream/AsyncStream for a generator body
```

Decorator surface (spellings kept — they are the right native words):

```python
@udf                                     # local; sync | async | gen | async-gen
@udf("gpu")                              # remote-by-default
@udf(mode="local" if DEBUG else "gpu")
@udf("gpu", timeout_s=..., resources={...}, image=..., retry=2)
@udf.pure / @udf.pure("gpu")             # re-invocation-safe (memoizable in-runtime)
```

Dispatch matrix:

| body | `fn(...)` local | `fn(...)` remote |
|---|---|---|
| `def` (sync) | result | `submit(...).result()` (blocks) |
| `async def` | coroutine | `await submit(...)` |
| `def … yield` (gen) | the generator | `Stream` (sync-iterable) |
| `async def … yield` (async-gen) | the async-gen | `AsyncStream` (async-iterable) |

Explicit entrypoints: `fn.local(...)`, `fn.submit(...) -> Future`, `fn.stream(...) -> Stream | AsyncStream`. `Stream`/`AsyncStream` are thin iterators over a `FrameReader` (§5) — **handles, never data types**; a consumer never constructs one.

**The one result path (fixes the bug):**

```python
# runner (callee)  — entry.py / _runners.run_in_process
bound  = udf.resolve_inputs(args, kwargs, ctx)     # Ref args -> local paths (download)
raw    = fn(*bound.a, **bound.k)                    # or asyncio.run(...) for async
result = ctx.commit(raw)                            # Ref outputs -> upload; returns metadata Refs
client.settle(id, "succeeded", wire.pack(result))  # was: msgpack.packb(result)  <-- the bug

# Future (caller)
def _unpack(self):
    return wire.unpack(self._blob)                  # was: msgpack.unpackb(...)
```

The **function reference** is content-addressed by source (`sha256(source)[:16]`, already computed for versioning) with `cloudpickle` kept only as an opt-in fallback for closures — because cloudpickle across a Python/deps mismatch is the classic remote-exec footgun and our args are already fully codec-able.

---

## 4. Native queues — `dreamlake/lakeshore/queue.py`

One `Queue(name)` carrying four faces; verbs renamed to name the actual job lifecycle. `submit`/`poll`/`ack` already exist in the control-plane `client.py`, so these align with the CP.

```python
class Queue:
    def __init__(self, name: str, *, server=None, namespace="default", **pool_opts): ...

    # ── task queue (durable) ─────────────────────────────
    def submit(self, value: dict, *, key: str | None = None,
               priority: float = 0.0, meta: dict | None = None) -> str          # -> job_id
    def claim(self, *, timeout_s: float = 30.0) -> "Job | None"
    def ack(self, job: "Job | str", *, result: Any = None) -> None              # settle success
    def nack(self, job: "Job | str", *, error: str, retry: bool = True) -> None
    def count(self, status: str = "pending") -> int
    def __len__(self) -> int
    @contextmanager
    def working(self):                     # claim() + auto-ack / auto-nack on exception
        j = self.claim()
        try: yield j; self.ack(j)
        except BaseException as e: self.nack(j, error=repr(e)); raise

    # ── pubsub ───────────────────────────────────────────
    def publish(self, value: dict, *, topic: str) -> int                        # fan-out count
    def subscribe_one(self, topic: str, *, timeout: float = 5.0) -> dict | None
    def subscribe(self, topic: str, *, timeout: float | None = None) -> Iterator[dict]   # REAL stream
    async def subscribe_async(self, topic, *, timeout=None) -> AsyncIterator[dict]

    # ── RPC (submit + reply on the invocation_id topic) ──
    def call(self, *args, timeout: float = 30.0, **kwargs) -> Any               # request -> one reply
    def stream(self, *args, timeout: float | None = None, **kwargs) -> Iterator[Any]
    def call_future(self, *args, **kwargs) -> "Future"
    async def call_async(self, *args, timeout=30.0, **kwargs) -> Any
```

```python
@dataclass(frozen=True)
class Job:
    id: str          # invocation_id == reply topic
    value: dict      # wire.unpack'd inline (S3 round-trip only for Ref leaves)
    attempt: int
    meta: dict
```

Gather + functional constructs (module-level; operate on `Future`/`Stream`):

```python
dls.gather(*futs, timeout=None)          -> list                  # ordered join (submit order)
dls.as_completed(futs, timeout=None)     -> Iterator[Future]      # ready order; RE-RAISES failures
dls.map(fn, iterable, *, unordered=False)-> list                  # ordered parallel map
dls.starmap(fn, arg_tuples)              -> list
dls.scatter(fn, iterable)                -> list[Future]          # submit-all, no wait
# async siblings: dls.gather_async / dls.as_completed_async / dls.map_async
```

Two concrete fixes over today's code, carried from the proposals: `submit(priority=)` **actually attaches** priority to the envelope (today's `PriorityQueue` has no submit-side field); `as_completed` uses `asyncio.wait` semantics with **no hardcoded 60 s fallback** and **no `except BaseException: pass`** — a failed future re-raises.

---

## 5. Streaming + chunked streaming

### Substrate — a per-invocation Redis-Streams journal in the Node control plane

Today `publish` is a *terminal* `ack` into a single result slot, so a "stream" physically cannot hold >1 message, and `subscribe_stream` fakes it with a `break`. The fix:

- Producer: each frame → `XADD ls:{inv}:out * f <frame_bytes>`.
- Terminal `END` (or `ERROR`) frame is the last entry.
- Consumer: `XREAD BLOCK <ms> STREAMS ls:{inv}:out <cursor>`, feed bytes to a `FrameReader`, advance the cursor by entry id.

Replayable, survives consumer reconnect, and distinguishes *producer finished* from *connection dropped* — the two things zaku's `write_eof()` cannot.

```python
# client.py additions (thin CP routes over the Redis Stream)
def journal_append(self, inv_id, frame: bytes) -> None                                  # XADD
def journal_read(self, inv_id, cursor="0", *, block_ms=5000) -> tuple[list[bytes], str, bool]
def journal_close(self, inv_id, state, error=None) -> None
```

### Author side — a UDF emits N frames by being a generator

```python
@dls.udf("gpu")
async def traj_to_ego_views(splats: str | Path, traj: str | Path) -> AsyncIterator[Ref]:
    steps = load_traj(dls.run.read(traj))
    for i, wp in enumerate(steps):
        p = dls.run.write(f"images/ind_{i:04d}.png")     # local scratch Ref, registered
        render(splats, wp).save(p)
        dls.run.progress(i + 1, len(steps))
        yield p                                          # -> one VALUE frame (a metadata Ref)
```

Each `yield` → `FrameWriter().value(commit(p), seq=i)` → `journal_append`. Exhaustion → `END{state:"succeeded"}`; an exception → `ERROR`.

**Raw chunked binary** (a single large value split rather than one giant msgpack `bin`) — yield `bytes`:

```python
@dls.udf("gpu")
def stream_tar(path: str | Path) -> Iterator[bytes]:
    yield dls.begin(content_type="application/x-tar")     # optional BEGIN metadata frame
    with open(dls.run.read(path), "rb") as f:
        while (b := f.read(1 << 20)):
            yield b                                        # a yielded `bytes` -> a CHUNK frame
```

A yielded `bytes` is **inherently** a raw chunk; anything else is a `VALUE`. `dls.begin(...)`/`dls.end(...)` are helper *functions* that emit `BEGIN`/`END` metadata frames — **not** types, so the "one new type" budget holds. All frames ride the same single `Unpacker`, so "raw chunked binary over one msgpack.Unpacker" is just consecutive `CHUNK` frames the reader concatenates until an `END`.

### Consumer side — iterate the call / the future

```python
async for p in traj_to_ego_views.stream(splats, traj):   # p is a Ref per rendered frame
    ...
data = b"".join(stream_tar(path))                         # CHUNK run rejoined by the reader
fut = traj_to_ego_views.submit(splats, traj)
for p in fut:                                             # Future.__iter__ drains the journal
    ...
```

`Future.__iter__`/`__aiter__` feed `journal_read` bytes into one `FrameReader`: `VALUE` → yield `frame.body`; a `BEGIN…CHUNK…END` run → yield joined bytes; `END` → stop; `ERROR` → raise the reconstructed exception.

---

## 6. Data-UDF returns with simple primitives — worked on the real functions

### The seam — `RunContext` gets `prefix` + `write`/`read`; bytes move here, not in the codec

```python
# context.py  (RunContext today: id, attempt, event_sink, parent_id, root_id)
@dataclass
class RunContext:
    ...
    prefix: str = ""                 # item root, e.g. "scenes/0007" (storage-relative)
    scratch: Path = field(default_factory=_mk_scratch)   # local working dir
    storage: str | None = None       # None => local mode, plain fs, no S3
    _writes: dict[str, Ref] = field(default_factory=dict)

class _RunProxy:                     # dls.run — the ENTIRE author-facing I/O surface
    @property
    def prefix(self) -> str: ...
    def write(self, key: str, *, dir: bool = False, eager: bool = False) -> Ref:
        """Declare an output at prefix/key. Returns a Ref (a Path) the body writes
        to with plain fs ops. Lazy(default): uploaded atomically on UDF success.
        eager=True: each file streams up as it lands (sequence UDFs)."""
    def read(self, ref_or_key: "Ref | str | Path") -> Ref:
        """Resolve an input to a local Ref, downloading prefix/key on first open
        if remote; identity in local mode."""
    def progress(self, i: int, total: int) -> None: ...

def scope(subpath: str):             # context manager pushing a sub-prefix (per-episode fan-out)
    ...
def ref(prefix: str, key: str, *, storage: str | None = None) -> Ref:   # reconstruct from (prefix,key)
    ...
```

`dls.run.write`/`dls.run.read` are the **only** place a `Storage` handle is reached — which is exactly why byte movement lives here and not in the codec (`pack(Ref)` is metadata-only). The `Storage` STS/presign brokering already in `storage.py` stays behind this seam; the body never sees a bucket, a credential, or an absolute path.

**Local mode** (`@dls.udf` with no mode, or `mode="local"`): `storage=None`, so `write`/`read` are plain filesystem ops under `scratch` — `generate_scene` runs end-to-end with no S3.

### The commit rule (in the runner, after the body returns)

`ctx.commit(value)` walks the return tree: a `Ref` leaf uploads to `storage[prefix/key]` and re-emits as a metadata `Ref`; a `Ref` that arrived from upstream is re-shipped as the same `(storage, key)` (no re-download/re-upload); `dict`/`tuple`/`list` recurse; anything else is inline data. A `Ref` returned but never created via `dls.run.write` raises `Ref "<key>" was returned but never created via dls.run.write(...)`.

### Worked examples on the real UDFs

**Single file — `splats_to_mesh`** (also `maps_to_keypoints`, `contact_sheets_to_segments`, `segment_sheets_to_labels`). Lazy/atomic: a body that raises publishes nothing.

```python
@dls.udf("gpu")
async def splats_to_mesh(splats: str | Path) -> Ref:
    mesh = dls.run.write("process/mesh.ply")        # was: out=f"{scene_root}/process/mesh.ply"
    extract_mesh(dls.run.read(splats)).save(mesh)
    return mesh
```

**Directory as one artifact — `mesh_to_maps`** (also `video_to_contact_sheets`, `segments_to_segment_sheets`). The `is_dir=True` Ref denotes the *completed* directory, so `maps_to_keypoints(maps)` never sees a half-populated `maps/`.

```python
@dls.udf
async def mesh_to_maps(mesh: str | Path) -> Ref:
    d = dls.run.write("process/maps", dir=True)
    build_maps(dls.run.read(mesh), out_dir=d)
    return d
```

**Named multi-output — `poses_to_views` → `rgb/` + `depth/`.** The name-carrying primitive is a plain `dict[str, Ref]`; downstream selects with `views["rgb"]`.

```python
@dls.udf("gpu")
async def poses_to_views(splats: str | Path, poses: str | Path) -> dict[str, Ref]:
    rgb   = dls.run.write("process/images/rgb",   dir=True, eager=True)
    depth = dls.run.write("process/images/depth", dir=True, eager=True)
    for pid, cam in enumerate(load(dls.run.read(poses))):
        render_rgb(splats, cam).save(rgb / f"{pid}.png")
        render_depth(splats, cam).save(depth / f"{pid}.png")
        dls.run.progress(pid, total=n)
    return {"rgb": rgb, "depth": depth}             # two stdlib primitives already say "N named files"
```

**Sequence (streaming) — `traj_to_ego_views`** returns `AsyncIterator[Ref]` (§5). Ordering lives in the key (`ind_{i:04d}.png`), recoverable from `ref.key` — so the row-alignment invariant lives in the key, not re-derived in two UDFs.

### Upstream → downstream threading via `_write`/`_prefix`

Pipelines thread the returned `Ref`s directly; the shared `prefix` makes refs resolvable across separately-decorated pipelines. `build_scene_graph` (today string-templates `f"{scene_root}/process/images"`) becomes:

```python
@dls.pipeline
async def build_scene_graph(scene_root: str) -> None:
    with dls.scope(scene_root):                          # prefix = scene_root
        views = await poses_to_views("source/splats.ply", "process/poses.json")
        await views_to_scene_graph(views["rgb"], "process/poses.json")   # select one named stream
```

Per-episode fan-out with a sub-prefix:

```python
@dls.pipeline
async def build_traj_tuples(scene_root: str, *, num_episodes: int = 100) -> None:
    with dls.scope(scene_root):
        for i in range(num_episodes):
            with dls.scope(f"outputs/{i:04d}"):
                traj = await goal_to_traj("goal.json", "../../process/maps")
                await traj_to_ego_views("../../source/splats.ply", traj)   # eats the traj Ref
```

Properties that fall out, each mapped to a requirement:

- **Cross-pipeline resolution by `(prefix, key)`** — a `Ref` is reconstructable from `prefix`+`key` alone (`dls.ref(...)`), so `poses_to_views` opens `process/poses.json` written by an earlier `extract_freespace_keypoints` run not in the current object graph. Also what makes migration incremental.
- **Fan-out / aliasing is free** — a `Ref` is immutable and re-readable; `source/splats.ply` feeds `splats_to_mesh`, `poses_to_views`, and `traj_to_ego_views` with **no consume-once semantics**.
- **`source/*` pre-existing inputs** are keys no UDF produced — addressed as `dls.run.read("source/splats.ply")` like any other key; the three-root `source/`·`process/`·`outputs/` layout is just prefix-relative keys.
- **Idempotent re-runs** — `(prefix, key)` is deterministic across `attempt`; a retry overwrites `outputs/{i:04d}/traj.npz` rather than forking.

### Migration — no flag day

The commit walk checks, in order: (1) an explicit `out=` the pipeline still passes (today's convention) → the runner infers `key = out.relative_to(prefix)` and wraps the returned bare `Path`/`str` as a `Ref`; else (2) the `Ref` the body returned from `dls.run.write`. So the eight pipelines keep running while UDFs migrate one at a time (`splats_to_mesh` first, `poses_to_views` when its two named streams matter) — each migrates by changing `out: str|Path` → `dls.run.write(...)` and its annotation → `-> Ref`, independently.

---

## 7. Concept-by-concept comparison vs zaku (primary deliverable)

| Concept | zaku's approach | Our native approach | Verdict + one-line why |
|---|---|---|---|
| **Envelope / codec pass** | `Payload(SimpleNamespace)`, greedy `ZData` walk, `_greedy` marker injected into user data | typed `Envelope{v,id,fn,prefix,args,kwargs,cfg}`, codec always-on, `default`/`ext_hook` recursion | **RENAME + NEW** — versionable header, no marker-in-data, no hand-walk (msgpack recurses for us) |
| **Type tags** | `{"ztype":"numpy.ndarray", b, dtype, shape}` dict overlay; tuple→list; `Path`≡`str` | `msgpack.ExtType` integer codes `0x01–0x06`; tuple + `Path` preserved | **RENAME** — ext codes can't collide with a user `"ztype"` dict; loud on unknown; keeps tuple/Path identity |
| **File / data returns** | none — S3 is an opaque queue transport; `take` returns one S3 ref for the whole payload | **one** `Ref` (Path subclass) + plain `dict`/`tuple`; bytes move at the `dls.run` seam, codec is metadata-only | **NEW (exactly one type)** — files are the data-UDF's actual results, threaded per-leaf; stdlib already gives named/positional access |
| **Result serialization** | (lakeshore) bare `msgpack.packb(result)` — drops ndarray/tuple/Ref | same `wire.pack` as args/kwargs/frames | **DROP the bug** — a result is just a value; one codec everywhere |
| **Task-queue verbs** | `add` / `take`(S3-ref) / `mark_done` | `submit` / `claim`(inline `Job`) / `ack` / `nack` | **RENAME** — names the real lifecycle; payload decoded inline, S3 only for `Ref` leaves |
| **PubSub** | `publish` = terminal `ack` (1 message max); `subscribe_stream` fakes it with `break` | `publish` → Redis-Streams journal; `subscribe`/`subscribe_async` read by cursor | **KEEP-LESSON + NEW** — real N-message fan-out, not a single-emit stub |
| **RPC** | mints `rpc-{uuid4()}` topic carried in `_request_id` | `call` / `stream` / `call_future`, reply topic **is** the `invocation_id` | **RENAME** — CP already indexes by invocation_id; no side-channel id field |
| **Gather** | dedicated `{name}.return-queue.{uuid}` + `gather-{uuid}` token set drained by `is_done` | `gather` / `as_completed` / `scatter` / `map` over `Future`s reading the journal cursor | **RENAME + DROP** — futures already carry completion; the second queue + token bookkeeping is redundant state |
| **Streaming framing** | self-delimited msgpack over chunked HTTP body, terminated by `write_eof()`; no seq | `Frame[kind,seq,body,meta]` over one `Unpacker`, Redis-Streams journal, explicit `END`/`ERROR` + `seq` | **KEEP-LESSON + NEW** — keep self-delimited framing; add real close signal + resumable cursor + chunked-binary frames |
| **`@udf` / client** | two client classes (`TaskQ` sync, `TaskQAsync` async); function is an opaque payload | one `@udf`; detects sync/async/gen/async-gen; local/remote; content-addressed fn | **NEW** — typed defs already declare their nature; no client class to pick |
| **Data-UDF I/O seam** | none — worker owns raw S3 keys | `dls.run.{prefix, write, read}` + `dls.scope` + `dls.ref` | **NEW** — lets a body return a plain `Ref`/`dict` and never see a credential |

---

## 8. Dual-ergonomics audit of the final public surface

| Surface | Greppable / guessable? | Precise hints? | Introspectable / self-doc? | Footguns removed | Error quality |
|---|---|---|---|---|---|
| `@udf` / `@udf("gpu")` | ✅ first guess for "remote function" | ✅ preserves the wrapped signature | ✅ `.ret`, `.is_async/.is_gen` exposed | one decorator, not two client classes | `mode "<x>" unknown; expected local\|<pool>` |
| `fn.submit / .local / .stream` | ✅ verbs an agent tries first | ✅ `Future[T]` / `Stream[T]` | ✅ handle types name themselves | `.local` always in-process (no accidental remote) | — |
| `Ref` | ✅ short, one concept | ✅ **is** a `Path`; every `str\|Path` sig accepts it | ✅ `.key`, `.storage`, `.is_dir` readable | **no** hidden `_REFS` global; **no** bare-string key; immutable → no consume-once | `Ref "<key>" returned but never created via dls.run.write(...)` |
| `dls.run.write / read / prefix` | ✅ `write`/`read` are the obvious pair | ✅ `write(key) -> Ref`, `read(x) -> Ref` | ✅ docstring states lazy-vs-eager + local mode | body never touches S3/creds/abs paths | `read("<key>"): not found under prefix "<p>"` |
| `Queue.submit / claim / ack / nack` | ✅ standard lifecycle verbs | ✅ `claim() -> Job \| None` | ✅ `Job.{id,value,attempt,meta}` | `working()` ctx-mgr auto-acks; `priority=` actually wired | `nack` requires `error=` (keyword-only) |
| `publish / subscribe` | ✅ | ✅ `subscribe() -> Iterator[dict]` | ✅ real stream, documented | not a 1-message stub; explicit `END` | — |
| `call / stream` (RPC) | ✅ | ✅ `call(...) -> Any`, `stream(...) -> Iterator` | ✅ reply keyed by invocation_id | no `rpc-{uuid}` side-channel to leak | `call` timeout raises `TimeoutError`, not `None` |
| `gather / as_completed / map / scatter` | ✅ mirrors `concurrent.futures` | ✅ typed over `Future[T]` | ✅ ordered-by-default documented | `as_completed` **re-raises** (no silent `except BaseException`) | failed future re-raises through the iterator |
| `wire.pack / unpack / register` | ✅ | ✅ `pack(Any) -> bytes` | ✅ `Codec` dataclass is the schema | unknown type raises at `pack`, not deep in msgpack | `wire: no codec for ; register one via wire.register(...)` |

**Things an agent would likely get wrong, and how the design prevents them:**

1. *Returning `str(path)` out of habit.* A bare `str` is data, not a file → it would silently not ship. Prevented: the `out=` migration bridge wraps a returned bare path when an `out=` key is present, and the primary path (`dls.run.write(...) -> Ref`) makes the file-ness a type the agent sees in the signature. A `Ref` never created via `write` raises with the fix in the message.
2. *Trying to `open()` a value received from a streaming/named output.* Because the value **is** a `Ref` (a `Path`), `open`/`np.load` just work — there is no bare key string to fumble.
3. *Assuming a stream is consume-once or that iterating twice re-reads.* The journal has a cursor and an explicit `END`; a `Future` re-iterated replays from cursor 0. `Ref`s are immutable and re-readable, so fan-out (`source/splats.ply` → three UDFs) is well-defined.
4. *Forgetting a serialization "mode" (zaku's `_greedy`).* There is no mode — the codec is always on and free for scalar payloads (`_default` fires only on non-native types).
5. *Picking the wrong queue verb.* `submit`/`claim`/`ack` are the CF-standard lifecycle words; `add`/`take`/`mark_done` are gone.

---

## 9. Open questions for the user

1. **`Ref` — accept the one-type exception?** The synthesis introduces exactly one runtime type, `Ref` (a `Path` subclass), on the grounds that the two zero-type alternatives each trip the no-footgun filter (hidden global registry / stringly-typed wire). Do you accept `Ref`, or do you want to hold the line at literally zero and live with a bare-`Path` + resolve-registry (accepting the aliasing footgun)?
2. **`Ref` subclassing `Path` on Python ≤3.11.** Direct subclassing of `pathlib.Path` is clean only on 3.12+. On ≤3.11 we need the `_flavour = type(Path())._flavour` trick. What is the floor Python version for lakeshore workers?
3. **Directory artifacts — upload granularity.** For `is_dir=True` Refs, do we upload the subtree as many keys under a prefix (listable, resumable) or as one packed object (atomic, one round-trip)? Affects `eager=True` sequence UDFs and cross-pipeline directory reads.
4. **Return annotations — plain hints only, or optional marker aliases?** The recommendation uses native hints (`-> Ref`, `-> dict[str, Ref]`, `-> AsyncIterator[Ref]`). The annotation-driven proposal offered richer markers (`File`/`Dir`/`Files`) that also encode the key template for the dashboard/canvas schema. Do you want those as *optional* introspection-only aliases, or keep the surface strictly primitive?
5. **Content-addressed fn vs cloudpickle default.** We propose shipping the fn by content-addressed source with cloudpickle as an opt-in closure fallback. Any UDFs today that rely on closures/partials that would force cloudpickle-by-default?
6. **`out=` bridge lifespan.** How long must the legacy `out=`-passing call sites keep working — one release, or indefinitely? Determines whether the bridge is a deprecation shim or a permanent second path.
7. **Journal retention.** Redis-Streams result journals need a trim/TTL policy (`XADD MAXLEN` or time-based). What retention do streamed results need for replay after a consumer reconnect?

---

Source: https://lakeshore.dreamlake.ai/python-sdk/rewrite/test-plan

Status: plan of record for the native rewrite (2026-07-16). Visual checks on the
dreamlake frontend are **explicitly out of scope** here (manual + design work);
this plan covers everything automatable.

## 1. The shape of the problem

The control plane serves **113 routes**; seven consumers sit on them with three
different body encodings. The rewrite replaces the msgpack dispatch wire and
the SDK surface — so the blast radius is exactly: *who parses which bytes*.

```
                 ┌────────────── JSON (field-name-sensitive) ──────────────┐
   TS CLI ───────┤                                                          │
   lakeshore-web ┤   resource CRUD routes (~90): providers/queues/storages/ │
   dreamlake-ai ─┤   workers/modes/secrets/... — UNCHANGED by the rewrite   │
    (via proxy)  └──────────────────────────────────────────────────────────┘
                 ┌────────────── msgpack (wire-rewrite blast zone) ─────────┐
   Python SDK ───┤   producer/submit|await, daemon/poll|ack|event,          │
   nymph (Rust) ─┤   payloads/presign + (new) result journal routes         │
                 └──────────────────────────────────────────────────────────┘
```

## 2. Consumer → surface matrix

| Consumer | Lang / encoding | Surface | CP routes touched | Rewrite risk |
|---|---|---|---|---|
| **Python SDK + daemon** (`dreamlake-lakeshore`) | msgpack on `_post`, JSON on GETs | `@dls.udf`, `Queue`, `gather`, `Storage`, daemon loop | `producer/submit\|await`, `producer/queues`, `daemon/poll\|ack\|event`, ns `queues` CRUD, `storages/:name/presign\|credentials`, `code-repos*`, `workers` | **Total** — this is the thing being rewritten |
| **nymph** (Rust daemon) | msgpack (`rmp_serde`); `/chunk` raw octet-stream | worker runtime: hello/poll/ack/exec-result/presign | `daemon/hello\|poll\|ack`, `daemon/exec/:id/result\|chunk`, `payloads/presign` | **Highest external risk** — field-name/type-sensitive msgpack structs; must stay compatible or be updated in lockstep (Phase 3) |
| **TS CLI + Node SDK** (`lakeshore`) | JSON only | ~20 command groups (`providers`, `queues`, `daemon`, `jobs`, `storage`, `exec`, `compose`, `admin`, …) | resource CRUD + `workers/*` + `invocations` list/show/kill + `exec` + `nymph-releases` | Low — JSON field renames only |
| **lakeshore-web** *(removed — the dashboard was dropped from the workspace; row kept for historical context)* | JSON, strongly typed | dashboard reads + worker ops | `admin/invocations\|pipelines\|providers`, ns `queues\|storages\|workers` (+hibernate/patch/delete) | n/a — no longer a consumer |
| **dreamlake-server** | JSON; proxy is verbatim *except it re-serializes mutating bodies as JSON* | `Lakeshore` connection CRUD + `proxy/*` passthrough; probe = `GET ns queues` / `healthz` | none with knowledge (passthrough) | Low; **caveat: proxy cannot carry msgpack request bodies** — fine while its only clients speak JSON |
| **dreamlake-ai** | JSON via proxy | lakeshore UI: invocations/queues/workers/modes/storages/mounts/secrets/tunnels/providers panels | ~30 ns routes + `admin/providers` through `proxy/` | Low — field renames only |
| **lakeshore-examples** | via Python SDK | `@dls.udf`, `Queue`, `gather`, `as_completed`, `Storage`, thunks | (through SDK) | Total — must be migrated with the SDK |

**Orphan routes** (no consumer found; candidates to drop or intentionally reserve):
`GET /readyz`, `PATCH admin/namespaces/:slug`, daemon key-rotation (`daemon/keys`,
`workers/:id/keys*`), `enroll-tokens*` (server-internal), the namespaced
`workers/:id/exec/:eid/result` alias, `jobs/:id/result` (both verbs),
`exec/:id/log[/stream]`.

## 3. What the rewrite changes vs preserves

**Preserved (no consumer action):** all JSON resource-CRUD routes; the
dreamlake proxy model (until Phase 4); `healthz`; exec/log paths.

**Changed:** the msgpack dispatch wire (`Envelope` replaces the ad-hoc
submit/poll/ack body shapes; unified `wire.pack` result path), SDK public
surface (`Queue`/`SyncQueue`, no `QueueFuture`, string-key data returns),
**new** journal routes (result-chunk append + cursor-read stream). During
Phase 2/3 the Python daemon and CP move in lockstep; **nymph poll/ack
compatibility is a named gate** — either the CP keeps the legacy poll/ack body
shape for nymph's queues, or nymph's `protocol.rs` structs update in the same
PR (decision at Phase 3 start, golden fixtures either way).

## 4. Test plan (automatable only)

### A. Unit — SDK (runs on every commit, no network)
- `wire.py`: codec round-trips (ndarray/tensor/image/tuple/nesting), unknown-type
  raise, forward-compat ext codes, Envelope versioning, Frame reassembly across
  arbitrary chunk splits. ✅ landed (`test_wire.py`, 19 tests)
- `context.py`: local read/write/scope, shape-mirroring, key escaping,
  return-your-keys contract (all violation classes), **local-simple invariant**
  (fresh-interpreter import check: no httpx/msgpack on local paths). ✅ landed
  (`test_context.py`, 17 tests)
- Phase 2 additions: `@udf` dispatch matrix (sync/async/gen/asyncgen ×
  local/remote), `Queue`/`SyncQueue` protocol core against a mock transport,
  idempotency-key semantics, `check_returned_keys` for async-gen accumulation.

### B. Wire-contract conformance suite (the centerpiece)
One parametrized pytest suite, run against **every dispatch-plane backend**:
1. the minimal Python CP (in-process; every commit), and
2. the Node CP (full-stack harness via `LAKESHORE_CP_DIR`; pre-merge + CI).

Coverage — each item is a named test, not an aspiration:
- submit → claim → ack lifecycle; nack + retry; attempt increments
- **atomic claim**: N concurrent claimers, one job → exactly one winner
- **fencing**: a stale worker (job re-dispatched after unstale) cannot ack
- **idempotent submit**: same key twice → same invocation id, no dup job
- ordering: priority desc / FILO / Boltzmann-sampling distribution (χ² over
  many claims), all implemented as sorts/samples over the primary store
- **counts == contents** (zaku-desync regression): counts derived from the same
  rows that claim; assert after every lifecycle transition, incl. crash-mid-claim
- journal: N frames in-order, `seq` cursor reconnect resumes without loss/dup,
  explicit `END` vs dropped-connection distinguishable, `ERROR` reconstructs the
  exception, CHUNK runs reassemble, MAXLEN/TTL trim degrades to CP re-fetch
- streaming UDF: async-gen yields → one frame per yield → consumer iterates
  live; partial-key-list manifests concatenate; contract violation surfaces

### C. Integration / e2e
- Existing self-provisioning full-stack e2e (Mongo + Node CP + Python daemon)
  updated to the new wire; runs pre-merge (`E2E_REQUIRE_FULLSTACK=1` in CI).
- CP's own suite stays green — baseline **599 pass / 0 fail** recorded 2026-07-16.
- **nymph compat**: golden msgpack fixtures for poll/ack request/response bodies,
  checked into both repos; a fixture diff = a deliberate, reviewed wire change.

### D. Cross-consumer contract snapshots (catches field renames)
- Golden JSON snapshots for every response body the TS CLI, lakeshore-web, and
  dreamlake-ai parse (queues/workers/storages/invocations/providers lists +
  details). Generated against the Node CP, replayed against the minimal CP.
- dreamlake-server proxy suite stays green — baseline **108/109** (1 failure
  pre-existing on main: `pipelines.test.ts` PATCH-artifacts, documented
  2026-07-16, unrelated to lakeshore).

### E. Consumer pipelines (this repo = the real customer)
- Local golden run: stub-UDF `generate_scene` + `annotate_subtasks` under
  `dls.scope`, asserting the exact `source/ · process/ · outputs/{i:04d}/` tree
  and manifest keys. Runs with zero network (the local fast path IS the test).
- One remote smoke per release: `splats_to_mesh`-shaped UDF through the minimal
  CP end-to-end (submit → daemon → storage side channel → keys back).

### F. Examples as tests
- `lakeshore-examples/test_runner.py`: the no-CP examples (02, 05) run in CI as
  plain scripts; queue/gather examples (08, 09, 10) run against the minimal CP.
  Examples are migrated with the SDK in Phase 2 and become its acceptance tests.

### G. Explicitly out of scope (manual / needs design)
- Visual checks on the dreamlake frontend (lakeshore panels, provider UI).
- TUI rendering (`lakeshore tui`).
- Cloud-provider launch paths (EC2/GCE/kube) — covered by existing CP tests only.

## 5. Gates per phase

| Phase | Merge gate |
|---|---|
| 0 wire ✅ | unit A green (landed: 61 pass) |
| 1 seam ✅ | unit A green + local-simple invariant (landed) |
| 2 @udf/queues + minimal CP | A + B(minimal CP) + E(local golden) + F(no-CP examples); CP suite untouched-green |
| 3 streaming + Node CP journal | B(both backends) + C incl. nymph fixtures + D snapshots |
| 4 dreamlake collections | D snapshots re-baselined + dreamlake-server suite green (108→109: fix the pre-existing failure while in there) |