# Daemon protocol

The nymph daemon talks to the control plane over **outbound HTTPS only**.
It opens a long-poll, receives commands in the response, runs them, and
posts results back. It never listens on a port and it never accepts an
inbound connection, so a host behind NAT, a VPN, or a corporate firewall
works with no extra plumbing.

This page is the wire-level reference. For the operator-facing commands
that *produce* this traffic, see
[Daemon lifecycle](/nymph/daemon-lifecycle.md).

## Transport and framing

Every daemon → control-plane body is msgpack, encoded as a **map with
string keys** (`rmp_serde::to_vec_named`) — not as positional arrays.
Responses are decoded the same way.

| Property        | Value                                                        |
| --------------- | ------------------------------------------------------------ |
| Method          | `POST` on every route                                        |
| `Content-Type`  | `application/x-msgpack`                                       |
| `Accept`        | `application/x-msgpack`                                       |
| `Authorization` | `Bearer <token>` when a token is configured                   |
| Request timeout | 120 s (HTTP client), idle pool timeout 90 s                   |

Two exceptions:

- **Log chunks** are posted as raw bytes with
  `Content-Type: application/octet-stream` — no base64, no msgpack
  envelope. The Bearer is still attached.
- The **exec-result callback** is accepted by the control plane as
  either JSON or msgpack; nymph sends msgpack.

There is no envelope. There is no `v` / `id` / `ts` / `kind` wrapper
around the body — the body *is* the request struct. Protocol versioning
rides on `HelloRequest.versions.protocol`, whose current value is the
string `"1"`.

## Endpoints

Everything the daemon initiates:

| Path                                    | Purpose                                                                 |
| --------------------------------------- | ----------------------------------------------------------------------- |
| `/v1/daemon/hello`                      | First contact. Identity, capabilities, versions, optional public key.   |
| `/v1/daemon/poll`                       | Long-poll. Carries in-flight status; receives commands.                 |
| `/v1/daemon/ack`                        | Terminal outcome of one `run` invocation.                               |
| `/v1/daemon/keys`                       | [Key rotation](/nymph/key-rotation.md) — signed by the outgoing key.       |
| `/v1/daemon/exec/{exec_id}/result`      | Exec completion callback (exit code, stdout, stderr, duration).         |
| *(server-supplied `LogSink.stdout_url`)* | Streaming exec log chunks. In practice `/v1/daemon/exec/{exec_id}/chunk`. |
| `/v1/namespaces/{ns}/payloads/presign`  | Resolve a large arg payload into a download URL.                        |

There are no server → daemon endpoints. Everything the server wants
done arrives in a poll response.

> **Note:** The control plane serves a `/v1/daemon/event` route (it is what writes
> `invocation.progress` / `invocation.log` / `invocation.event` rows), and
> a `/v1/daemon/result-chunk` route for streamed results. The Rust daemon
> posts to neither today — a stale module docstring in `src/protocol.rs`
> claims otherwise.

## Hello

The first call after boot. It carries everything the control plane needs
to admit the daemon and materialise its Worker row.

### Request

| Field          | Type                    | Notes                                                                  |
| -------------- | ----------------------- | ---------------------------------------------------------------------- |
| `machine_id`   | string                  | Stable per host. `"auto"` in the config resolves to the hostname.      |
| `label`        | string                  | Human label. Empty falls back to the worker id.                        |
| `tags`         | string[]                | `key=value` strings, config tags merged with detected host tags.        |
| `lanes`        | string[]                | Queue memberships. Omitted from the wire entirely when empty.           |
| `capabilities` | `{ runners, max_invocations, workdir }` | What this host can actually run.                        |
| `versions`     | object                  | Self-reported binary + tooling versions (below).                        |
| `public_key`   | string \| absent        | Raw 32-byte Ed25519 key, base64url no padding. Present only when an identity is configured. |

`lanes` is the historical name and remains the wire field; the control
plane maps it onto `Worker.queues`. An empty/absent value means
default-queue membership.

`versions` is probed once at boot, best-effort — anything that fails to
resolve is `null`:

| Key        | Probe                                                       |
| ---------- | ----------------------------------------------------------- |
| `daemon`   | `CARGO_PKG_VERSION` of the running binary                    |
| `protocol` | the constant `"1"`                                           |
| `runsc`    | `runsc --version`                                            |
| `docker`   | `docker version --format {{.Server.Version}}`                |
| `python`   | `python3 --version`, else `python --version`                 |
| `kernel`   | `uname -sr`                                                  |
| `distro`   | `PRETTY_NAME=` from `/etc/os-release`                        |
| `arch`     | `uname -m`                                                   |
| `cpu`      | `/proc/cpuinfo` → `{ model, cores }`                         |
| `gpu`      | `nvidia-smi --query-gpu=name,driver_version` + the `CUDA Version:` header → `[{ name, driver, cuda }]` |

### Response

| Field                    | Type            | Notes                                                                  |
| ------------------------ | --------------- | ---------------------------------------------------------------------- |
| `worker_id`              | string \| null  | Canonical Mongo ObjectId. When null, the daemon keeps its provisional `{machine_id}-{ULID}`. |
| `session_token`          | string \| null  | A ULID. **Not persisted server-side** — vestigial for signed daemons.  |
| `session_ttl_s`          | number          | 86400.                                                                 |
| `config`                 | object          | Live config map. Empty (`{}`) today.                                   |
| `cursor`                 | string \| null  | Always null today.                                                     |
| `reject`                 | object \| null  | `{ code, message }`. Non-null aborts daemon startup.                   |
| `key_id`                 | string \| null  | Echoed when a keypair was enrolled on this hello.                      |
| `key_rotation_max_age_s` | number          | Advisory only — the daemon stores nothing and waits for the `rotate_key` directive. |
| `server_time_epoch_s`    | number \| null  | For clock-skew calibration.                                            |

### Rejection

A refusal comes back as **HTTP 200** with a `reject` object, not as a
4xx status. The exact codes:

`protocol_mismatch` · `enroll_token_required` · `invalid_enroll_token` ·
`enroll_token_consumed` · `enroll_token_expired` ·
`enroll_signature_required` · `key_id_mismatch` ·
`enroll_signature_invalid` · `enrollment_signature_required`

A malformed request (missing `machine_id`, `label`, `tags`,
`capabilities.runners`, `versions.daemon`, or `versions.protocol`) is a
`422` carrying `{ error, field }`.

## Poll

### Request

| Field                  | Type                    | Notes                                                       |
| ---------------------- | ----------------------- | ----------------------------------------------------------- |
| `worker_id`            | string                  | Canonical id when hello returned one.                       |
| `queue`                | string                  | From `--queue` / `DREAMLAKE_QUEUE`. Trailing `*` = prefix.  |
| `timeout_s`            | number                  | `[poll] wait_max_s`, default **20.0**. The server caps it at 20. |
| `cursor`               | null                    | Always sent as `null` — cursor tracking is not implemented. |
| `invocations`          | array                   | One `InvocationStatus` per locally-tracked run.             |
| `updated_capabilities` | object \| absent        | Sent on the first poll after a `setup` with `refresh_capabilities`, then cleared. |

`InvocationStatus` is
`{ invocation_id, state, started_at, exit_code, stdout_tail }`. In
practice the daemon only ever publishes `state: "running"` with
`exit_code: null` and an empty `stdout_tail` — stdout tailing is not
implemented.

### Response

```text
{ commands: [ { id, kind, body }, … ], target_nymph_version: string | null }
```

`body` is decoded as a raw msgpack value and then dispatched on `kind`.
A known `kind` whose body does not match, or an entirely unknown `kind`,
decodes to `Unknown` and is logged and skipped — it never wedges the
loop.

### Long-poll mechanics

Each iteration races three things: the cancellation token (SIGTERM /
SIGINT), an internal wake signal fired the instant any exec finishes, and
the poll itself. That wake signal is why back-to-back execs flow at
completion speed instead of waiting out `wait_max_s`.

On a poll error the daemon backs off with full jitter —
`backoff + rand(0,1) * min(backoff, 5.0)` — then doubles up to
`[poll] retry_max_s`. A successful poll resets it to `retry_min_s`.

Concurrency is bounded by a semaphore sized to `[runtime] max_invocations`
(`0` means unbounded). When no permit is free the daemon stops
long-polling entirely and waits for one.

## Command kinds

Exactly seven, evaluated in this order:

| Kind            | Body                                                                | What the daemon does                                                                                        |
| --------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `hibernate`     | `{ until_epoch_s }`                                                  | Returns from the poll loop immediately, sleeps until the deadline on its own clock, then re-enters from scratch. |
| `setup`         | `{ script, sudo?, refresh_capabilities?, name? }`                    | Pushes onto a serial FIFO drained by a dedicated task. Never awaited inline.                                 |
| `exec`          | `{ exec_id, command, env, stdin, workdir?, timeout_s?, log?, payload_inline?, payload_ref? }` | Spawns `bash -c <command>`, posts the result callback.                     |
| `ota_update`    | `{ version, url, sha256 }`                                           | Downloads, verifies, rename-installs over itself, and (by default) `exec()`s the new binary.                 |
| `reset_backoff` | *(none)*                                                             | Resets the poll backoff to `retry_min_s`.                                                                    |
| `rotate_key`    | *(dispatched by kind)*                                               | Generates a new keypair and calls `/v1/daemon/keys`. See [Key rotation](/nymph/key-rotation.md).                |
| `run`           | `{ invocation_id, function, run_config, args_blob, parent_invocation_id, root_invocation_id, script?, as_user? }` | Dispatches to a runner; acks the outcome.                |

> **Warning:** `ota_update`, `reset_backoff`, and `rotate_key` are snake_case. The
> other four are bare words. `reset_backoff` and `rotate_key` carry no
> typed body — they are dispatched purely on the `kind` string.

### `hibernate`

Handling a hibernate **returns from the inner poll loop**, so any
commands later in the same response are discarded. The daemon sleeps
`until_epoch_s − now` seconds (its own clock, cancellable by shutdown),
sets its run-state to `hibernating`, clears the idle tracker, and starts
a fresh poll loop.

### `setup`

Each setup runs through the `process` runner as a synthetic invocation:
`function.module = "_setup"`, `function.qualname` = the supplied `name`
or `"setup"`, and `script` = the body's script. `sudo: true` sets
`as_user = "root"`, which wraps the spawn in `runuser -u root -- …`.

On exit 0 with `refresh_capabilities: true`, host detection re-runs and
the new `Capabilities` snapshot ships on the next poll as
`updated_capabilities`. The control plane merges it into the Worker row
and flips `setting_up` → `active` once the queue has drained.

Setup scripts **must be idempotent** — probe before installing, use
`apt-get install -y`, prefer `systemctl enable --now`, and
read-modify-write configs rather than blind-append.

### `exec`

The command runs under `bash -c` — deliberately **not** `bash -lc`, so
there is no login shell and no `/etc/profile.d`. The supplied `env` map
is merged *on top of* the daemon's own environment; `stdin` is piped then
closed.

Exit-code sentinels on the result callback:

| Value      | Meaning                                                   |
| ---------- | --------------------------------------------------------- |
| *(actual)* | The child exited normally with this code.                 |
| `-1`       | Spawn failure, wait error, or cancellation at shutdown.   |
| `-2`       | Timeout. The child is killed with a 5 s settle grace.     |

Captured stdout and stderr are each capped at **1 MiB**; overflow sets
`error` to `"stdout truncated"` / `"stderr truncated"` /
`"stdout truncated; stderr truncated"`.

When `log` is present the daemon streams instead: reader tasks forward
8 KiB reads into a channel, and a shipper posts the accumulated buffer to
`log.stdout_url` whenever it crosses `flush_bytes` or `flush_ms` elapses.
A failed chunk post keeps the bytes buffered for the next flush and never
fails the exec. In streaming mode the final result callback ships
**empty** `stdout` / `stderr`.

`LogSink` is `{ stdout_url, stdin_url?, control_url?, flush_ms, flush_bytes }`;
`stdin_url` and `control_url` are declared but unused, reserved for
interactive sessions.

The result body is
`{ exit_code, stdout, stderr, duration_ms, error?, result_inline?, result_ref? }`.
`result_inline` and `result_ref` are always null today — the shell exec
path has no typed return value.

### `ota_update`

`sha256` must be exactly 64 ASCII hex characters. The daemon streams the
download to `.nymph-ota-{pid}-{ulid}.tmp` in the *same directory* as the
running executable so the install is an atomic `rename(2)`, hashes on the
fly, refuses on mismatch, `chmod`s `cur_perms | 0o111`, renames over
itself, and then `exec()`s the new binary with the original arguments.
That last step is gated on the `ota-self-replace` cargo feature, which is
**enabled by default**.

> **Warning:** With `[runtime] auto_update = true`, a `target_nymph_version` on the poll
> response that differs from the running version synthesises an OTA no more
> often than `auto_update_check_interval_s`. The URL it builds hard-codes
> **musl** triples (`https://lakeshore.dreamlake.ai/bin/nymph-{triple}-{version}`)
> while CI actually publishes **gnu** triples to the R2 release bucket. The
> source labels this path as scaffolded; prefer the explicit
> `lakeshore daemon update` path, where the control plane derives the URL.

### `run`

`run_config.runner` picks the runner (falling back to
`[runtime] default_runner`). See
[Daemons → Runners](/nymph/daemons.md#runners) for the six accepted kinds
and their contracts. An unknown kind does **not** fall back to the
default — it fails the invocation with `unknown runner kind: {other}`.

## Ack

A terminal `run` outcome is posted immediately rather than waiting for
the next poll:

| Field           | Type                                    |
| --------------- | --------------------------------------- |
| `invocation_id` | string                                  |
| `state`         | `"succeeded"` \| `"failed"` \| `"killed"` |
| `result_blob`   | bytes (empty unless succeeded)          |
| `error`         | `{ type, message, traceback? }` \| null |
| `worker_id`     | string (omitted when empty)             |

`error` is a structured object, never a bare string. On the wire the
inner field is `type` (the Rust struct spells it `type_name` and renames
via serde, because `type` is a keyword). A runner that fails with only a
message gets `type = "DaemonError"`; a cancelled invocation gets
`{ type: "Cancelled", message: "daemon shutdown grace expired" }`.

Ack failures are logged and dropped — they never propagate into the poll
loop.

## Payload store

When a `run` or `exec` body carries a `payload_ref` instead of inline
bytes, the daemon resolves it by posting to
`/v1/namespaces/{ns}/payloads/presign` with
`{ op: "get" | "put", key?, size?, jobId?, kind? }` (camelCase on the
wire) and downloading from the returned URL.

| Constant                  | Value              |
| ------------------------- | ------------------ |
| Inline threshold          | 256 KiB            |
| Max payload               | 100 MiB            |
| Presign fetch timeout     | 60 s               |

`PayloadRef` is `{ bucket, key, size, content_type, sha256? }` — the
daemon accepts both `contentType` and `content_type` on the wire. A
sha256 mismatch short-circuits the exec with `exit_code = -1` and no
child is spawned.

## Auth

Without an identity configured, requests carry the bootstrap bearer from
`[server] token_file` and **no** `X-LS-*` headers — byte-identical to the
legacy path.

With `[identity] enabled = true`, the daemon holds a per-host Ed25519
keypair, enrolls it with a one-shot `dle_` token on `/hello`, and signs
every subsequent request over a canonical string. The control plane's
signature hook guards exactly `/v1/daemon/poll`, `/ack`, and `/event`;
`/hello` and `/keys` verify their own signatures in-route; the exec
result and chunk callbacks are authenticated by the unguessable
`exec_id` in the URL.

The full contract — canonical string, headers, enrollment, rotation,
revocation, and the `LAKESHORE_REQUIRE_DAEMON_SIGNATURES` migration flag
— lives on [Daemon identity and key rotation](/nymph/key-rotation.md).

## Known gaps

- **Cursor tracking is not implemented.** `PollRequest.cursor` is
  hard-coded to `null`; there is no at-most-once replay guard on the
  daemon side. Setup and exec commands are expected to be idempotent for
  this reason.
- **No stdout tailing on poll.** `InvocationStatus.stdout_tail` is always
  empty; live output for a `run` is not on the wire.
- **No cancel signal.** Killing a *running* invocation from the control
  plane returns 409 — the wire protocol has no per-invocation cancel
  command. `SIGTERM` to the daemon is the escape hatch.

## Read next

- [Daemons](/nymph/daemons.md) — runners, workdir contracts, host detection.
- [Daemon lifecycle](/nymph/daemon-lifecycle.md) — the CLI verbs that emit
  these commands.
- [Daemon identity and key rotation](/nymph/key-rotation.md) — signing.
- [Function protocol](/api/function-protocol.md) — the invocation envelope a
  `run` command carries.