Daemon protocol

The nymph daemon talks to the control plane over outbound HTTPS only. It opens a long-poll, receives commands in the response, runs them, and posts results back. It never listens on a port and it never accepts an inbound connection, so a host behind NAT, a VPN, or a corporate firewall works with no extra plumbing.

This page is the wire-level reference. For the operator-facing commands that produce this traffic, see Daemon lifecycle.

Transport and framing

Every daemon → control-plane body is msgpack, encoded as a map with string keys (rmp_serde::to_vec_named) — not as positional arrays. Responses are decoded the same way.

Two exceptions:

• Log chunks are posted as raw bytes with Content-Type: application/octet-stream — no base64, no msgpack envelope. The Bearer is still attached.
• The exec-result callback is accepted by the control plane as either JSON or msgpack; nymph sends msgpack.

There is no envelope. There is no v / id / ts / kind wrapper around the body — the body is the request struct. Protocol versioning rides on HelloRequest.versions.protocol, whose current value is the string "1".

Endpoints

Everything the daemon initiates:

There are no server → daemon endpoints. Everything the server wants done arrives in a poll response.

Hello

The first call after boot. It carries everything the control plane needs to admit the daemon and materialise its Worker row.

Request

lanes is the historical name and remains the wire field; the control plane maps it onto Worker.queues. An empty/absent value means default-queue membership.

versions is probed once at boot, best-effort — anything that fails to resolve is null:

Response

Rejection

A refusal comes back as HTTP 200 with a reject object, not as a 4xx status. The exact codes:

protocol_mismatch · enroll_token_required · invalid_enroll_token · enroll_token_consumed · enroll_token_expired · enroll_signature_required · key_id_mismatch · enroll_signature_invalid · enrollment_signature_required

A malformed request (missing machine_id, label, tags, capabilities.runners, versions.daemon, or versions.protocol) is a 422 carrying { error, field }.

Poll

Request

InvocationStatus is { invocation_id, state, started_at, exit_code, stdout_tail }. In practice the daemon only ever publishes state: "running" with exit_code: null and an empty stdout_tail — stdout tailing is not implemented.

Response

{ commands: [ { id, kind, body }, … ], target_nymph_version: string | null }

body is decoded as a raw msgpack value and then dispatched on kind. A known kind whose body does not match, or an entirely unknown kind, decodes to Unknown and is logged and skipped — it never wedges the loop.

Long-poll mechanics

Each iteration races three things: the cancellation token (SIGTERM / SIGINT), an internal wake signal fired the instant any exec finishes, and the poll itself. That wake signal is why back-to-back execs flow at completion speed instead of waiting out wait_max_s.

On a poll error the daemon backs off with full jitter — backoff + rand(0,1) * min(backoff, 5.0) — then doubles up to [poll] retry_max_s. A successful poll resets it to retry_min_s.

Concurrency is bounded by a semaphore sized to [runtime] max_invocations (0 means unbounded). When no permit is free the daemon stops long-polling entirely and waits for one.

Command kinds

Exactly seven, evaluated in this order:

hibernate

Handling a hibernate returns from the inner poll loop, so any commands later in the same response are discarded. The daemon sleeps until_epoch_s − now seconds (its own clock, cancellable by shutdown), sets its run-state to hibernating, clears the idle tracker, and starts a fresh poll loop.

setup

Each setup runs through the process runner as a synthetic invocation: function.module = "_setup", function.qualname = the supplied name or "setup", and script = the body's script. sudo: true sets as_user = "root", which wraps the spawn in runuser -u root -- ….

On exit 0 with refresh_capabilities: true, host detection re-runs and the new Capabilities snapshot ships on the next poll as updated_capabilities. The control plane merges it into the Worker row and flips setting_up → active once the queue has drained.

Setup scripts must be idempotent — probe before installing, use apt-get install -y, prefer systemctl enable --now, and read-modify-write configs rather than blind-append.

exec

The command runs under bash -c — deliberately not bash -lc, so there is no login shell and no /etc/profile.d. The supplied env map is merged on top of the daemon's own environment; stdin is piped then closed.

Exit-code sentinels on the result callback:

Captured stdout and stderr are each capped at 1 MiB; overflow sets error to "stdout truncated" / "stderr truncated" / "stdout truncated; stderr truncated".

When log is present the daemon streams instead: reader tasks forward 8 KiB reads into a channel, and a shipper posts the accumulated buffer to log.stdout_url whenever it crosses flush_bytes or flush_ms elapses. A failed chunk post keeps the bytes buffered for the next flush and never fails the exec. In streaming mode the final result callback ships empty stdout / stderr.

LogSink is { stdout_url, stdin_url?, control_url?, flush_ms, flush_bytes }; stdin_url and control_url are declared but unused, reserved for interactive sessions.

The result body is { exit_code, stdout, stderr, duration_ms, error?, result_inline?, result_ref? }. result_inline and result_ref are always null today — the shell exec path has no typed return value.

ota_update

sha256 must be exactly 64 ASCII hex characters. The daemon streams the download to .nymph-ota-{pid}-{ulid}.tmp in the same directory as the running executable so the install is an atomic rename(2), hashes on the fly, refuses on mismatch, chmods cur_perms | 0o111, renames over itself, and then exec()s the new binary with the original arguments. That last step is gated on the ota-self-replace cargo feature, which is enabled by default.

run

run_config.runner picks the runner (falling back to [runtime] default_runner). See Daemons → Runners for the six accepted kinds and their contracts. An unknown kind does not fall back to the default — it fails the invocation with unknown runner kind: {other}.

Ack

A terminal run outcome is posted immediately rather than waiting for the next poll:

error is a structured object, never a bare string. On the wire the inner field is type (the Rust struct spells it type_name and renames via serde, because type is a keyword). A runner that fails with only a message gets type = "DaemonError"; a cancelled invocation gets { type: "Cancelled", message: "daemon shutdown grace expired" }.

Ack failures are logged and dropped — they never propagate into the poll loop.

Payload store

When a run or exec body carries a payload_ref instead of inline bytes, the daemon resolves it by posting to /v1/namespaces/{ns}/payloads/presign with { op: "get" | "put", key?, size?, jobId?, kind? } (camelCase on the wire) and downloading from the returned URL.

PayloadRef is { bucket, key, size, content_type, sha256? } — the daemon accepts both contentType and content_type on the wire. A sha256 mismatch short-circuits the exec with exit_code = -1 and no child is spawned.

Auth

Without an identity configured, requests carry the bootstrap bearer from [server] token_file and no X-LS-* headers — byte-identical to the legacy path.

With [identity] enabled = true, the daemon holds a per-host Ed25519 keypair, enrolls it with a one-shot dle_ token on /hello, and signs every subsequent request over a canonical string. The control plane's signature hook guards exactly /v1/daemon/poll, /ack, and /event; /hello and /keys verify their own signatures in-route; the exec result and chunk callbacks are authenticated by the unguessable exec_id in the URL.

The full contract — canonical string, headers, enrollment, rotation, revocation, and the LAKESHORE_REQUIRE_DAEMON_SIGNATURES migration flag — lives on Daemon identity and key rotation.

Known gaps

• Cursor tracking is not implemented. PollRequest.cursor is hard-coded to null; there is no at-most-once replay guard on the daemon side. Setup and exec commands are expected to be idempotent for this reason.
• No stdout tailing on poll. InvocationStatus.stdout_tail is always empty; live output for a run is not on the wire.
• No cancel signal. Killing a running invocation from the control plane returns 409 — the wire protocol has no per-invocation cancel command. SIGTERM to the daemon is the escape hatch.

Read next

• Daemons — runners, workdir contracts, host detection.
• Daemon lifecycle — the CLI verbs that emit these commands.
• Daemon identity and key rotation — signing.
• Function protocol — the invocation envelope a run command carries.