# Daemon identity and key rotation

Each daemon can hold a per-host **Ed25519** keypair. The private key
never leaves the host; the daemon signs every control-plane request with
it, and the control plane verifies against the public key it recorded at
enrollment. This replaces the bearer-only handshake with a real host
identity, per-request signatures with replay protection, scheduled and
on-demand rotation, and immediate revocation.

The signing string is byte-identical on both sides, so it is documented
here exactly.

> **Note:** The control plane accepts signatures today and the daemon can generate,
> sign, and rotate. Neither is enforced: nymph's `[identity] enabled`
> defaults to `false`, and the control plane defaults to
> signatures-optional. Enabling is a two-sided, per-fleet decision — see
> [Enforcing signatures](#enforcing-signatures).

## Enroll tokens

Enrollment is gated by a one-shot **enroll token**, distinct from the
namespace API tokens in [Auth and secrets](/api/auth-and-secrets.md).

- **Format** `dle_<base64url(32 bytes)>`. The `dle_` prefix is greppable
  for accidental-paste audits — note it differs from the CLI/SDK's
  `dlk_`.
- **One-shot.** Consumed atomically on first successful enrollment; two
  concurrent enrollments collapse to exactly one winner.
- **TTL.** Default **3600 s** (1 hour); override per-mint with `ttl_s`,
  capped at 30 days.
- **Namespace-scoped.** A token only enrolls a daemon into the namespace
  it was minted for.
- **Hash at rest.** Only `sha256(plaintext)` (hex) and an 8-character
  display prefix are stored. The plaintext is returned **once**, at mint.

| Verb     | Path                                   | Notes                                                                                                    |
| -------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `POST`   | `/v1/namespaces/:ns/enroll-tokens`     | Admin only. Body `{ label, ttl_s? }`. Returns the plaintext once.                                        |
| `GET`    | `/v1/namespaces/:ns/enroll-tokens`     | Lists `{id, label, prefix, createdAt, expiresAt, consumedAt, consumedByWorkerId, spent}` — never plaintext. |
| `DELETE` | `/v1/namespaces/:ns/enroll-tokens/:id` | Admin only. Burns the token (marks it consumed) while preserving the audit row.                           |

Labels match `^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$`. The mint returns `201`:

```json
{
  "id": "…",
  "label": "bos14-rig",
  "prefix": "dle_AbCd",
  "plaintext": "dle_AbCd…",
  "expiresAt": "2026-06-26T13:34:56.000Z",
  "createdAt": "2026-06-26T12:34:56.000Z"
}
```

## Daemon configuration

```toml file="udf-daemon.toml"
[identity]
enabled = true
key_file = "/etc/dreamlake/daemon.key"
enroll_token_file = "/etc/dreamlake/enroll-token"   # or: enroll_token = "dle_…"
```

| Key                 | Default                      | Meaning                                                        |
| ------------------- | ---------------------------- | -------------------------------------------------------------- |
| `enabled`           | `false`                      | Master switch. When off, requests carry **no** `X-LS-*` headers. |
| `key_file`          | `/etc/dreamlake/daemon.key`  | Private key. Created `0600` on first boot, reused thereafter.   |
| `enroll_token`      | *(unset)*                    | Inline `dle_…` plaintext.                                       |
| `enroll_token_file` | *(unset)*                    | File containing the token, trimmed. Keeps it out of the TOML.   |

The inline value wins when both are set. If `enabled = true` but the key
cannot be loaded, the daemon **fails closed** and exits rather than
silently degrading to bearer.

The key is persisted as the raw 32 secret bytes, base64url-nopad, one
line, mode `0600` — written to a `<name>.tmp` sibling that is `chmod`'d
before an atomic rename into place. `load()` accepts base64url or
standard base64 and rejects anything that is not exactly 32 bytes. The
`Debug` impl prints only the key id, never the secret.

## Enrollment

On first boot the daemon generates its keypair, writes the private key,
and derives its **key id**:

```text
keyId = base64url_nopad(sha256(raw_32_byte_public_key))[:16]
```

It is a 16-character truncation of the base64url encoding — not of the
hex digest, and not the full 43-character encoding.

The daemon then calls `POST /v1/daemon/hello` carrying the raw public key
(base64url, no padding) in `public_key`, the enroll token as
`Authorization: Bearer dle_…`, and the
[signature headers](#the-signing-scheme) self-signed with the new key.
Because the daemon does not yet know its control-plane worker id, the
canonical string uses `machine_id` in the worker-id slot.

The control plane then:

1. Decodes the public key (must be exactly 32 bytes).
2. Validates the enroll token — exists, not consumed, not expired — but
   does **not** consume it yet.
3. Confirms `X-LS-Key-Id` matches the key id it derives from the supplied
   key, then verifies the self-signature.
4. Upserts the Worker, stores the key as `active` with
   `expiresAt = now + key_rotation_max_age_s`, and **atomically** consumes
   the enroll token.

Because the consume happens last, a bad signature never burns a token. If
the consume loses a race, no key is stored and the response rejects with
`enroll_token_consumed`.

Namespace precedence on hello: the enroll token's namespace wins, then
any pre-existing Worker row matched by `machineId`, then `default`.

> **Warning:** A refused enrollment comes back as `200` with a
> `{ "reject": { "code", "message" } }` msgpack body — not a 401 or 403.
> Codes: `enroll_token_required`, `invalid_enroll_token`,
> `enroll_token_consumed`, `enroll_token_expired`,
> `enroll_signature_required`, `key_id_mismatch`,
> `enroll_signature_invalid`, `enrollment_signature_required`, and
> `protocol_mismatch`.

The successful hello response carries the canonical `worker_id` (which
the daemon stamps in for all subsequent signing), the confirmed `key_id`,
`key_rotation_max_age_s`, and `server_time_epoch_s`.

## The signing scheme

Every signed request carries four headers and a detached Ed25519
signature over a **canonical string** — eight lines joined by a single
`\n`:

```text file="canonical-signing-string"
lsd-v1

<PATH, no query string>

<NONCE, verbatim from the header>
<lowercase hex sha256(BODY bytes)>
```

`lsd-v1` is the literal domain separator — it versions the scheme.
Hashing the body into the string means one mechanism covers msgpack and
JSON routes alike, and it is why the control plane re-registers its body
parsers to stash the exact on-wire bytes.

| Header           | Encoding                                                  |
| ---------------- | --------------------------------------------------------- |
| `X-LS-Key-Id`    | The 16-character key id.                                  |
| `X-LS-Timestamp` | Integer epoch **seconds**, not milliseconds.              |
| `X-LS-Nonce`     | base64url (no padding) of 16 CSPRNG bytes.                |
| `X-LS-Signature` | base64url (no padding) of the 64-byte signature.          |

The prefix is `X-LS-`, not `X-Lakeshore-`. Headers are looked up
lowercase in the header bag.

### Which routes are signed

| Route                                    | How it is authenticated                                  |
| ---------------------------------------- | -------------------------------------------------------- |
| `/v1/daemon/poll`, `/ack`, `/event`      | The signature `preHandler` hook.                          |
| `/v1/daemon/hello`                       | Verifies its own self-signature in-route.                 |
| `/v1/daemon/keys`                        | Verifies in-route, **always** — no bearer fallback ever.  |
| `/v1/daemon/exec/:id/{result,chunk}`, `/v1/daemon/result-chunk` | Not signature-guarded — authenticated by the unguessable id in the URL. |

### Verification order

Each step is a distinct rejection, in this order:

1. Key lookup by `(worker_id, key_id)` — missing → `401` "unknown daemon key".
2. Status: `revoked` → `403`; anything other than `active` or `retiring`
   → `403`.
3. Past the hard `expiresAt` → `401` "daemon key expired".
4. `|now − timestamp| > skew` → `401` "timestamp outside skew window".
5. Ed25519 verify over the recomputed canonical string → `401`
   "signature verification failed".
6. Nonce check-and-record → `401` "replayed nonce".

The nonce step runs **last on purpose**, so an unauthenticated caller
cannot burn cache entries.

The replay cache keys on `(worker_id, nonce)` with a TTL equal to the
skew window. It uses Redis (`SET NX PX` on `nonce:{worker}:{nonce}`) when
`REDIS_URL` is configured, and an in-memory map otherwise. The in-memory
store is correct for a single instance; **configure Redis before running
more than one control-plane dyno**, since only the Redis path's atomic
`SET NX` prevents two dynos accepting the same nonce.

### Header-presence policy

| Headers present    | Behaviour                                                                    |
| ------------------ | ---------------------------------------------------------------------------- |
| All four           | Verify as above (`authMode = "daemon-sig"`).                                 |
| **None**           | Fall back to legacy bearer (`authMode = "daemon-bearer"`) — unless enforcement is on, in which case `401`. |
| Some but not all   | Hard `401` "incomplete signature headers". Never a silent bearer fallback.    |

## Rotation

Keys rotate on two triggers: **scheduled** (the active key reaches
`key_rotation_max_age_s`, default 30 days, or passes `expiresAt`) and
**on-demand**. The flow is zero-downtime via an **overlap window**
(default 600 s).

1. On a poll, the control plane attaches a `rotate_key` command:

   ```json
   {
     "id": "cmd-rotate_key-1750000000",
     "kind": "rotate_key",
     "body": {
       "reason": "max_age_exceeded",
       "deadline_epoch_s": 1750000600,
       "key_id": "<current key id>"
     }
   }
   ```

   `reason` is `max_age_exceeded` or `key_expired`; `deadline_epoch_s` is
   `now + overlap`.

2. The daemon generates a new keypair and calls `POST /v1/daemon/keys`
   with `{ worker_id, public_key }`, **signed by the old key** using the
   canonical worker id. Rotating to the same key id is rejected `422`
   ("new key must differ from the current key").

3. On success the new key is stored `active`; the old key flips to
   `retiring` with `retiringAt = now` and `revokedAt = now + overlap` — a
   *future* timestamp acting as the overlap deadline. Both keys verify
   during the window, so in-flight old-key requests still succeed.

4. The response is
   `{ key_id, status, expires_at, overlap_s, server_time_epoch_s }`. The
   daemon swaps its live signing key only after that confirmation, via an
   atomic swap behind a lock — safe while polls are in flight.

```text
active ──rotate──▶ retiring ──overlap elapses──▶ revoked
   ▲                  │
   └── both verify during the overlap ──┘
```

An opportunistic reaper (run on each poll, idempotent) flips the
`retiring` key to `revoked` once the deadline passes.

A failed rotation is best-effort: the daemon logs a warning, keeps its
current key, and never wedges the poll loop. With no identity configured
the directive is a logged no-op.

`HelloResponse.key_rotation_max_age_s` is purely advisory — the daemon
stores nothing and acts only on the server-issued directive.

## Inspection and revocation

```bash
# List a worker's keys (keyId, status, createdAt, retiringAt, revokedAt,
# expiresAt, publicKey)
curl $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"

# Immediate kill
curl -X POST \
  $CP/v1/namespaces/$NS/workers/$WORKER_ID/keys/$KEY_ID/revoke \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN"
```

Revocation sets `status = "revoked"` and `revokedAt = now`; every
subsequent request signed with that key is rejected `403`. The call is
idempotent — a second revoke returns `alreadyRevoked: true`. A revoked
daemon must **re-enroll** with a fresh enroll token.

Note that `keyId` is unique per `(workerId, keyId)`, not globally.

## Enforcing signatures

`LAKESHORE_REQUIRE_DAEMON_SIGNATURES` is the migration switch. It must be
the literal string `"true"`; anything else, including unset, is off.

| State             | `/hello`                                                         | `/poll`, `/ack`, `/event`                     | `/keys`                     |
| ----------------- | ---------------------------------------------------------------- | ---------------------------------------------- | --------------------------- |
| **Off** (default) | `public_key` optional; legacy bearer enrollment still accepted   | Signatures optional; bearer fallback accepted  | Always requires a signature |
| **On**            | Must provide `public_key` + enroll token; legacy path rejected   | Signatures **required**; bearer fallback `401` | Always requires a signature |

Roll out in order: (1) run a control plane that accepts both; (2) enable
`[identity]` on the fleet; (3) confirm no `daemon-bearer` traffic in the
logs; (4) flip the flag and restart.

### Bearer fallback (migration only)

With the flag off and no signature headers present, the request is
accepted in `daemon-bearer` mode and the bearer token itself is **not
validated** — its presence is informational, a bridge for daemons that
have not started signing. Likewise the `session_token` in the hello
response is vestigial: minted, never persisted, on the way out. Treat
both as legacy.

### Tuning knobs

| Env var                               | Default   | Meaning                                            |
| ------------------------------------- | --------- | -------------------------------------------------- |
| `LAKESHORE_DAEMON_SKEW_S`             | `300`     | Timestamp skew window, seconds.                    |
| `LAKESHORE_DAEMON_KEY_OVERLAP_S`      | `600`     | Retiring-key overlap window, seconds.              |
| `LAKESHORE_DAEMON_KEY_MAX_AGE_S`      | `2592000` | Max key age before a rotation directive (30 days). |
| `LAKESHORE_REQUIRE_DAEMON_SIGNATURES` | *(unset)* | `"true"` enforces signatures.                       |

## Operator runbook

**Bring up a signed daemon.** Mint a token, capture the plaintext (shown
once), and hand it to the daemon:

```bash
curl -sX POST $CP/v1/namespaces/$NS/enroll-tokens \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"label": "bos14-rig", "ttl_s": 3600}'
# → { "plaintext": "dle_…", … }   save it now — never shown again
```

Write it to the host's `enroll_token_file`, set `[identity] enabled =
true`, and start the daemon. It generates its keypair on first boot,
enrolls, and the token is spent the moment it registers.

**Force a rotation.** Scheduled rotation is automatic. To pull it
forward, lower `LAKESHORE_DAEMON_KEY_MAX_AGE_S` on the control plane —
the next poll for an over-age daemon then carries the directive. No
restart required.

**Revoke a compromised host.** Find the worker and key id
(`lakeshore daemon show <id>`, or the keys endpoint above), then POST the
revoke route. Re-enroll with a fresh token if the host should rejoin.

## Read next

- [Daemon protocol](/nymph/protocol.md) — the wire protocol these signed
  requests ride on.
- [Auth and secrets](/api/auth-and-secrets.md) — namespace API tokens
  (`dlk_`), distinct from daemon enroll tokens (`dle_`).
- [Daemons](/nymph/daemons.md) — what the daemon does once it is trusted.