Architecture

Understand the three layers, the objects the control plane stores, and how an invocation travels from your laptop to a remote worker.

Lakeshore ships as three independently-deployed pieces, each swappable without touching the others:

• lakeshore CLI — Node + TypeScript, distributed via npm (@dreamlake/lakeshore). The lakeshore binary registers providers, manages secrets / modes / queues / storages / mounts / tunnels, brings daemons up, and runs one-off commands against them.
• Lakeshore control plane — Node + Fastify 5 + Prisma 5 + Mongo, deployed on Heroku. The single brain that owns canonical state and hands Invocations to Workers.
• nymph daemon — Rust binary (the nymph submodule — dreamlake-ai/nymph), the host process that supervises Invocations under docker / gvisor / process runners. Closed-source for now; built from source.

┌─────────────────────────────────────────────────────────────────┐
│  Client surface                                                 │
│   ─ lakeshore CLI         (Node, on npm — operator verbs)       │
│   ─ Python SDK            (dreamlake-lakeshore — @udf, Futures) │
├─────────────────────────────────────────────────────────────────┤
│  Control plane  (Lakeshore, Node + Fastify + Prisma)            │
│                                                                 │
│   Durable store   Mongo    — every model below                  │
│   Hot lane        Redis    — result-chunk delivery, log cache   │
│   Blob store      S3       — spilled payloads, code archives,   │
│                              archived exec logs                 │
│                                                                 │
│   ─ Scheduling   Queue · Invocation · Function · ExecJob        │
│   ─ Compute      Worker · Provider · Mode · QueueServer         │
│   ─ Data         Storage · Mount · CodeRepo                     │
│   ─ Identity     Namespace · Token · EnrollToken · DaemonKey ·  │
│                  Secret · Tunnel                                │
│   ─ Audit        Event · ElasticityEvent · ResultChunk          │
├─────────────────────────────────────────────────────────────────┤
│  Runtime  (Rust daemon, on each compute host)                   │
│   ─ Daemon (nymph)        (long-polls; runs Invocations under   │
│                            process / docker / gvisor / slurm /  │
│                            kube runners)                        │
│   ─ Wire protocol         (msgpack long-poll; daemon-initiated) │
└─────────────────────────────────────────────────────────────────┘

prisma/schema.prisma defines exactly 21 models. All of them carry a namespaceId except three: Namespace itself (it is the boundary), QueueServer, and NymphRelease (a global binary registry, deduped by sha256).

Reading the layers

Client surface is what you talk to. The CLI is the operator surface: register providers, create queues, launch daemons, exec commands against them, inspect jobs. The Python SDK (dreamlake-lakeshore) is the developer surface — the @udf(queue=...) decorator, submit() returning a durable invocation id, and SyncQueue.result(id). See Python SDK.

Control plane is Lakeshore — a single Node.js + TypeScript service. Three stores sit behind it, each with a distinct job:

• Mongo (via Prisma) is the durable record and the source of truth for every model on this page. It also carries the dispatch hot path: a raw findAndModify on the Invocation collection — { state: "queued", "runConfig.queue": … } sorted by { priority: -1, submittedAt: 1 } — atomically flips one row to running and stamps the workerId. That is why v0 needs no Lua scripts and no row locks.
• Redis is a low-latency delivery channel, not the scheduler. It carries exec log streams (exec:<id>:stdout), per-invocation result journals (invocation:<id>:result), and the daemon-signature replay-nonce store. The ResultChunk rows in Mongo are the durable truth — trimming Redis never loses a frame, reads just degrade to Mongo. (A Redis hot lane for queue dispatch is described in the schema header as a designed-but-unbuilt v1+ optimization.)
• S3 holds anything too big to inline: spilled payloads, CodeRepo archives, and archived exec logs. See Payload store for the concept and /dev/payload-store for the spec.

Runtime is what executes your code. The daemon — nymph, a Rust binary — runs on every compute host; it speaks the wire protocol to the control plane and supervises the Invocation under one of the registered runner kinds: process (alias subprocess), docker, gvisor, slurm, or kube. Daemons are outbound-only: they long-poll the control plane and never accept inbound connections, so a compute host needs egress and nothing else.

The object model

Everything the control plane stores is namespaced. Start with the four you will meet first, then the supporting cast.

The core four

Queue

A named pipe carrying pending Invocations, and the first-class scheduling primitive. Workers subscribe by name — Worker.queues: string[] records which queues a daemon pulls from, and the empty array means the default queue (there is no sentinel string).

A Queue carries three policies:

kindParams holds type-specific knobs — { temperature: 0.5 } for boltzmann, { priority_field: "p" } for priority.

Lifecycle: the schema declares active, draining, paused, and archived, but only three are reachable through routes — /drain → draining, /archive → archived, /unarchive → active. Nothing writes paused.

See Queues for the concept page, Elasticity and Scaling rules for the autoscaling behaviour, and /dev/queues for the design note.

Worker

A daemon process registered with the control plane. Each Worker carries a stable machineId (reported on /hello and preserved across restarts), a queues array, a runners array, a capabilities object (runner list, max_invocations, workdir, plus the detected CPU/GPU inventory), and a tag set. Queue membership and elasticity match constraints decide which Invocations a Worker can claim.

Worker state runs pending → setting_up → joining → active → gone. pending is a row pre-created by daemons/launch before the host boots; setting_up means queued setup commands are still draining; joining is set by /hello; active is the first poll after that. draining is declared in the schema but no route writes it. A seventh label, stale, is derived at read time when lastSeenAt is older than 60 s — it is never persisted.

Two fields worth knowing: hostStatus tracks the cloud instance lifecycle independently of daemon poll state (reconciled against the provider's DescribeInstances), and composeProject records the project: slug when the Worker was spawned by lakeshore up — ad-hoc daemons leave it null and stay invisible to lakeshore ps / lakeshore down.

See Daemon runtime and Daemon lifecycle.

Invocation

One deferred execution of a Function. Keyed by a ULID generated by the producer — the same id that flows across the wire — with attempt, priority, submitted / started / finished timestamps, and a state:

queued → running → one of succeeded · failed · killed · timeout.

Arguments and results ride as inline msgpack in argsBlob / resultBlob. The model also declares argsRef / resultRef for spilling to S3, but no route writes them yet — the payload store is wired on the ExecJob path first. correlationId, parentInvocationId, and a denormalized rootInvocationId give you the pipeline tree — the root field equals the invocation's own id when it is the root.

There is no queueId column. Phase A.5 removed it; the queue name is denormalized onto runConfig.queue by POST /v1/producer/submit, and every read path (admin list, ?queue= filter, the dispatcher's claim) matches on that embedded path. Absent, it falls back to "default". idempotencyKey and codeRepoId exist on the model but no route writes either one yet.

Inspect via lakeshore jobs list / show / kill — see Jobs.

Worker host

The machine the daemon runs on. Brought online either by the control plane (via a Provider row — EC2 / GCE / SLURM / Kube) or by the CLI bootstrapping nymph onto an existing host over SSH. See Daemon launch and Bootstrap a daemon on a remote host.

Compute and placement

Data and code movement

Identity and secrets

Audit and observability

How dispatch flows today

operator                control plane                  worker host
────────                ─────────────                  ───────────

lakeshore daemon launch
  --queue training-h100 ───▶ Worker row created
                             Provider boots host

                             /hello (EnrollToken ───▶ Daemon enrols,
                              consumed, DaemonKey       registers
                              registered)               queues=["training-h100"]
                                       │
                                       ▼
                                /poll ─┘  ◀── long-poll

lakeshore exec / run
  ───▶ Invocation enqueued ───▶ Queue("training-h100")
                                       │
                                       └─ poll response  ───▶ Daemon runs
                                                                command in
                                                                runner
                                                                (process /
                                                                 docker / gvisor)
                                  ◀── /result-chunk (streamed frames)
                                  ◀── /event         (progress, logs)
                                  ◀── /ack           (terminal result)

The daemon-facing surface is eight POST endpoints, all under /v1/daemon/ and all msgpack:

Three things to notice:

1. The CLI never talks to a worker directly. Every operator action (exec, run, daemon update, daemon kill) goes through the control plane.
2. The worker never accepts an inbound connection. It long-polls, and control commands ride back on the poll response. The complete set of command kinds is run, setup, exec, hibernate, ota_update, reset_backoff, and rotate_key — note the underscores on the last three. Only run comes from claiming an invocation; the others short-circuit the claim loop.
3. Only /poll, /ack, and /event are covered by the signature pre-handler. /hello and /keys verify their own signatures in-route, and the two exec callbacks plus /result-chunk are authenticated by the unguessable id in the URL.

The Python SDK (/python-sdk) produces the same Invocation envelope and rides the same dispatch path.

Agents — not yet modeled

State machines at a glance

Glossary

Read next