Daemon identity and key rotation

Each daemon can hold a per-host Ed25519 keypair. The private key never leaves the host; the daemon signs every control-plane request with it, and the control plane verifies against the public key it recorded at enrollment. This replaces the bearer-only handshake with a real host identity, per-request signatures with replay protection, scheduled and on-demand rotation, and immediate revocation.

The signing string is byte-identical on both sides, so it is documented here exactly.

Enroll tokens

Enrollment is gated by a one-shot enroll token, distinct from the namespace API tokens in Auth and secrets.

• Format dle_<base64url(32 bytes)>. The dle_ prefix is greppable for accidental-paste audits — note it differs from the CLI/SDK's dlk_.
• One-shot. Consumed atomically on first successful enrollment; two concurrent enrollments collapse to exactly one winner.
• TTL. Default 3600 s (1 hour); override per-mint with ttl_s, capped at 30 days.
• Namespace-scoped. A token only enrolls a daemon into the namespace it was minted for.
• Hash at rest. Only sha256(plaintext) (hex) and an 8-character display prefix are stored. The plaintext is returned once, at mint.

Labels match ^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$. The mint returns 201:

{
  "id": "…",
  "label": "bos14-rig",
  "prefix": "dle_AbCd",
  "plaintext": "dle_AbCd…",
  "expiresAt": "2026-06-26T13:34:56.000Z",
  "createdAt": "2026-06-26T12:34:56.000Z"
}

Daemon configuration

[identity]
enabled = true
key_file = "/etc/dreamlake/daemon.key"
enroll_token_file = "/etc/dreamlake/enroll-token"   # or: enroll_token = "dle_…"

The inline value wins when both are set. If enabled = true but the key cannot be loaded, the daemon fails closed and exits rather than silently degrading to bearer.

The key is persisted as the raw 32 secret bytes, base64url-nopad, one line, mode 0600 — written to a <name>.tmp sibling that is chmod'd before an atomic rename into place. load() accepts base64url or standard base64 and rejects anything that is not exactly 32 bytes. The Debug impl prints only the key id, never the secret.

Enrollment

On first boot the daemon generates its keypair, writes the private key, and derives its key id:

keyId = base64url_nopad(sha256(raw_32_byte_public_key))[:16]

It is a 16-character truncation of the base64url encoding — not of the hex digest, and not the full 43-character encoding.

The daemon then calls POST /v1/daemon/hello carrying the raw public key (base64url, no padding) in public_key, the enroll token as Authorization: Bearer dle_…, and the signature headers self-signed with the new key. Because the daemon does not yet know its control-plane worker id, the canonical string uses machine_id in the worker-id slot.

The control plane then:

1. Decodes the public key (must be exactly 32 bytes).
2. Validates the enroll token — exists, not consumed, not expired — but does not consume it yet.
3. Confirms X-LS-Key-Id matches the key id it derives from the supplied key, then verifies the self-signature.
4. Upserts the Worker, stores the key as active with expiresAt = now + key_rotation_max_age_s, and atomically consumes the enroll token.

Because the consume happens last, a bad signature never burns a token. If the consume loses a race, no key is stored and the response rejects with enroll_token_consumed.

Namespace precedence on hello: the enroll token's namespace wins, then any pre-existing Worker row matched by machineId, then default.

The successful hello response carries the canonical worker_id (which the daemon stamps in for all subsequent signing), the confirmed key_id, key_rotation_max_age_s, and server_time_epoch_s.

The signing scheme

Every signed request carries four headers and a detached Ed25519 signature over a canonical string — eight lines joined by a single \n:

lsd-v1
<METHOD uppercased>
<PATH, no query string>
<WORKER_ID>
<KEY_ID>
<TIMESTAMP epoch seconds, integer>
<NONCE, verbatim from the header>
<lowercase hex sha256(BODY bytes)>

lsd-v1 is the literal domain separator — it versions the scheme. Hashing the body into the string means one mechanism covers msgpack and JSON routes alike, and it is why the control plane re-registers its body parsers to stash the exact on-wire bytes.

The prefix is X-LS-, not X-Lakeshore-. Headers are looked up lowercase in the header bag.

Which routes are signed

Verification order

Each step is a distinct rejection, in this order:

1. Key lookup by (worker_id, key_id) — missing → 401 "unknown daemon key".
2. Status: revoked → 403; anything other than active or retiring → 403.
3. Past the hard expiresAt → 401 "daemon key expired".
4. |now − timestamp| > skew → 401 "timestamp outside skew window".
5. Ed25519 verify over the recomputed canonical string → 401 "signature verification failed".
6. Nonce check-and-record → 401 "replayed nonce".

The nonce step runs last on purpose, so an unauthenticated caller cannot burn cache entries.

The replay cache keys on (worker_id, nonce) with a TTL equal to the skew window. It uses Redis (SET NX PX on nonce:{worker}:{nonce}) when REDIS_URL is configured, and an in-memory map otherwise. The in-memory store is correct for a single instance; configure Redis before running more than one control-plane dyno, since only the Redis path's atomic SET NX prevents two dynos accepting the same nonce.

Header-presence policy

Rotation

Keys rotate on two triggers: scheduled (the active key reaches key_rotation_max_age_s, default 30 days, or passes expiresAt) and on-demand. The flow is zero-downtime via an overlap window (default 600 s).

1. On a poll, the control plane attaches a rotate_key command:

   jsonCopy

   {
     "id": "cmd-rotate_key-1750000000",
     "kind": "rotate_key",
     "body": {
       "reason": "max_age_exceeded",
       "deadline_epoch_s": 1750000600,
       "key_id": "<current key id>"
     }
   }

   reason is max_age_exceeded or key_expired; deadline_epoch_s is now + overlap.

2. The daemon generates a new keypair and calls POST /v1/daemon/keys with { worker_id, public_key }, signed by the old key using the canonical worker id. Rotating to the same key id is rejected 422 ("new key must differ from the current key").

3. On success the new key is stored active; the old key flips to retiring with retiringAt = now and revokedAt = now + overlap — a future timestamp acting as the overlap deadline. Both keys verify during the window, so in-flight old-key requests still succeed.

4. The response is { key_id, status, expires_at, overlap_s, server_time_epoch_s }. The daemon swaps its live signing key only after that confirmation, via an atomic swap behind a lock — safe while polls are in flight.

active ──rotate──▶ retiring ──overlap elapses──▶ revoked
   ▲                  │
   └── both verify during the overlap ──┘

An opportunistic reaper (run on each poll, idempotent) flips the retiring key to revoked once the deadline passes.

A failed rotation is best-effort: the daemon logs a warning, keeps its current key, and never wedges the poll loop. With no identity configured the directive is a logged no-op.

HelloResponse.key_rotation_max_age_s is purely advisory — the daemon stores nothing and acts only on the server-issued directive.

Inspection and revocation

# List a worker's keys (keyId, status, createdAt, retiringAt, revokedAt,
# expiresAt, publicKey)
curl $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"

# Immediate kill
curl -X POST \
  $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys/$KEY_ID/revoke \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"

Revocation sets status = "revoked" and revokedAt = now; every subsequent request signed with that key is rejected 403. The call is idempotent — a second revoke returns alreadyRevoked: true. A revoked daemon must re-enroll with a fresh enroll token.

Note that keyId is unique per (workerId, keyId), not globally.

Enforcing signatures

LAKESHORE_REQUIRE_DAEMON_SIGNATURES is the migration switch. It must be the literal string "true"; anything else, including unset, is off.

Roll out in order: (1) run a control plane that accepts both; (2) enable [identity] on the fleet; (3) confirm no daemon-bearer traffic in the logs; (4) flip the flag and restart.

Bearer fallback (migration only)

With the flag off and no signature headers present, the request is accepted in daemon-bearer mode and the bearer token itself is not validated — its presence is informational, a bridge for daemons that have not started signing. Likewise the session_token in the hello response is vestigial: minted, never persisted, on the way out. Treat both as legacy.

Tuning knobs

Operator runbook

Bring up a signed daemon. Mint a token, capture the plaintext (shown once), and hand it to the daemon:

curl -sX POST $CP/v1/namespaces/$NS/enroll-tokens \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"label": "bos14-rig", "ttl_s": 3600}'
# → { "plaintext": "dle_…", … }   save it now — never shown again

Write it to the host's enroll_token_file, set [identity] enabled = true, and start the daemon. It generates its keypair on first boot, enrolls, and the token is spent the moment it registers.

Force a rotation. Scheduled rotation is automatic. To pull it forward, lower LAKESHORE_DAEMON_KEY_MAX_AGE_S on the control plane — the next poll for an over-age daemon then carries the directive. No restart required.

Revoke a compromised host. Find the worker and key id (lakeshore daemon show <id>, or the keys endpoint above), then POST the revoke route. Re-enroll with a fresh token if the host should rejoin.

Read next

• Daemon protocol — the wire protocol these signed requests ride on.
• Auth and secrets — namespace API tokens (dlk_), distinct from daemon enroll tokens (dle_).
• Daemons — what the daemon does once it is trusted.