# Auth and secrets

Two related concerns. **Auth** is how a caller proves itself to the
control plane. **Secrets** are server-stored credential material the
control plane decrypts in memory when it launches hosts on your behalf.

Both share the same dev ergonomic: in production the relevant env var is
required and the server refuses to boot without it; in dev, omitting it
degrades to an insecure fallback with a loud one-time warning.

## Credential types

Four distinct bearer credentials. Do not mix them up.

| Credential | Format | Who holds it | What it grants |
| --- | --- | --- | --- |
| Admin token | operator-chosen, from `LAKESHORE_ADMIN_TOKEN` on the server | the operator | Everything, in every namespace. Required to mint the other two token types. |
| Namespace token | `dlk_<base64url-32-bytes>` | CLI, SDK, dashboard | CRUD in its own namespace (plus org siblings). |
| Enroll token | `dle_<base64url-32-bytes>` | a daemon, once | One-shot daemon enrollment on `/v1/daemon/hello`. |
| Daemon signature | Ed25519 over `X-LS-*` headers | a daemon, ongoing | Poll / ack / event. Covered on [Key rotation](/nymph/key-rotation.md). |

The `dlk_` / `dle_` prefixes are greppable so an accidental paste is
auditable, and they make the two token types impossible to confuse in a
log.

## Where the auth hook fires

The `onRequest` auth hook matches exactly two path shapes:

- `/v1/admin/*` — admin token or open mode only. Anything else gets `403`.
- `/v1/namespaces/:ns/*` — the tiered check below.

**Every other route is outside the hook**, including all of
`/v1/daemon/*` and `/v1/producer/*`. Those either carry their own
in-route verification (daemon signatures) or are authenticated by an
unguessable id in the URL.

One explicit exemption inside the namespaced space:
`/v1/namespaces/:ns/workers/:id/exec/:exec_id/result` bypasses namespace
auth entirely — it is a daemon callback authenticated by the `exec_id`.

### The tiered check

1. **Open mode.** When `LAKESHORE_ADMIN_TOKEN` is unset, every request
   passes with `authMode = "open"`.
2. **Missing bearer** → `401 missing bearer token`.
3. **Admin.** Bearer equals `LAKESHORE_ADMIN_TOKEN` (constant-time
   compare) → `authMode = "admin"`, full access to any namespace.
4. **Namespace token.** Look up a `Token` row by `hash =
   sha256(bearer)`. Unknown → `401 invalid token`. Revoked → `403 token
   revoked`. Namespace mismatch → `403`, *unless* both namespaces share a
   non-null `orgId`. On success `authMode = "token"` and `lastUsedAt` is
   bumped asynchronously.

`authMode` is one of `admin`, `token`, `open`, `daemon-sig`,
`daemon-bearer` — note the hyphens. `whoami` only ever reports the first
three.

Only four routes require the admin tier beyond the hook itself: minting
a namespace token, minting an enroll token, deleting an enroll token, and
uploading a nymph release. `requireAdmin()` returns true for `authMode`
`admin` **or** `open`.

## Server env config

```bash file="lakeshore-controlplane/.env"
# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 24 | tr '+/' '-_' | tr -d '='
LAKESHORE_ADMIN_TOKEN=""

# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 32
SECRETS_KEY=""
```

- **Production** (`NODE_ENV=production`) — the server logs a fatal and
  exits when either is missing.
- **Dev** — leaving `LAKESHORE_ADMIN_TOKEN` blank enters **open mode**
  with one loud warn at startup. Leaving `SECRETS_KEY` blank falls back
  to a hard-coded `DEV_FALLBACK_KEY`, also with a warn.

> **Note:** The server formerly read `ADMIN_TOKEN` with no prefix. It is still
> accepted, with a one-time stderr deprecation warning. Migrate `.env`
> files and Heroku config vars to `LAKESHORE_ADMIN_TOKEN` before the
> back-compat goes away.

## Client-side env vars

These are easy to conflate — they are read by different programs and are
not interchangeable.

| Var | Read by | Effect |
| --- | --- | --- |
| `LAKESHORE_URL` | every CLI command except `auth`; the Python SDK | Names the control plane. **When set, the CLI sends no bearer at all** — it deliberately bypasses the saved auth file. |
| `LAKESHORE_SERVER` | `lakeshore auth login \| status` only; the `nymph` daemon | Server URL for the auth commands and for the daemon's `--server`. |
| `LAKESHORE_NAMESPACE` | CLI (with `LAKESHORE_URL`), Python SDK | Namespace slug. Defaults to `default`. |
| `LAKESHORE_CLIENT_TOKEN` | the Python SDK's default dispatch | Bearer for `HttpDispatch`. `lakeshore worker` passes the token to its Python child this way — never on argv. |
| `LAKESHORE_ADMIN_TOKEN` | the CP server (validation); the CLI's `admin` group and `auth login` | On the server host, and once in your shell when bootstrapping. |

> **Warning:** `lakeshore worker start` / `worker once` deliberately do not consult
> `auth.yml`. The plane a worker drains must come from `--url` or
> `LAKESHORE_URL`. Running `lakeshore auth login` is not sufficient setup
> for a worker.

## CLI auth commands

```bash
lakeshore auth login [--server <url>] [--namespace <slug>] [--token <plaintext>] [--name <label>]
lakeshore auth status [--server <url>] [--namespace <slug>]
lakeshore auth logout
```

Server URL precedence for these three commands, highest first:
`--server`, `LAKESHORE_SERVER`, the saved auth file, then
`http://localhost:8080`.

`login` resolves a token, validates it by calling
`GET <server>/v1/namespaces/<ns>/whoami` with a Bearer header, and saves
on success. `status` reloads the saved auth and re-validates. `logout`
deletes the file.

Credentials land at `$XDG_CONFIG_HOME/lakeshore/auth.yml` (defaulting to
`~/.config/lakeshore/auth.yml`), written `chmod 600`:

```yaml file="~/.config/lakeshore/auth.yml"
server: https://api.lakeshore.dreamlake.ai
namespace: default
token: dlk_xxxxxxxxxxxx…
```

### First-time bootstrap via the admin token

If `--token` is absent and `LAKESHORE_ADMIN_TOKEN` is set in your shell,
`login` mints a fresh namespace token through the admin path and saves
**that** — not the admin token — to `auth.yml`:

```bash
export LAKESHORE_ADMIN_TOKEN=...          # same value as the server's .env
lakeshore auth login --server http://localhost:8080 --namespace dev --name cli-$USER
lakeshore auth status                     # → authMode: token
```

The label actually stored is `<--name>-<timestamp>`; the suffix keeps a
re-run from colliding with the per-namespace unique-name constraint. With
neither `--token` nor `LAKESHORE_ADMIN_TOKEN`, `login` prompts on a TTY
and fails with exit 2 otherwise.

## `whoami`

`GET /v1/namespaces/:ns/whoami` returns the caller's view. Reaching `200`
is what `auth login` treats as validation.

```json
{
  "namespace": "default",
  "tokenPrefix": "dlk_xxxx",
  "tokenName": "cli-ge-2026-05-18T05-40-00",
  "lastUsedAt": "2026-05-18T05:40:00.000Z",
  "authMode": "token"
}
```

For an admin or open-mode caller there is no backing `Token` row, so the
route returns a synthetic envelope: `tokenPrefix: null` and `tokenName`
of `<admin>` or `<open>`.

## Token CRUD

| Verb | Path | Notes |
| --- | --- | --- |
| `POST` | `/v1/namespaces/:ns/tokens` | Admin only. `201`. Returns plaintext once. |
| `GET` | `/v1/namespaces/:ns/tokens` | Metadata only. |
| `DELETE` | `/v1/namespaces/:ns/tokens/:id` | Soft delete — sets `revokedAt`. |

Mint body is `{ "name": "cli-ge" }`. Labels match
`^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$` (underscores allowed, unlike the
lowercase-and-hyphen regex used for resource names) and are unique per
namespace. The mint response is `{id, name, prefix, plaintext,
createdAt}`.

Only `hash = sha256(plaintext)` (globally unique) and `prefix` (the first
8 characters of the plaintext, for display) are stored. Revocation is a
soft delete so audit listings keep the row; a revoked token returns
`403`.

> **Warning:** The mint response is the only time a token's plaintext leaves the server.
> There is no recovery path — lose it and the remedy is revoke plus mint a
> new one.

The CLI wraps the same routes under `lakeshore admin tokens create |
list | revoke` (alias `ls` for `list`), all of which hard-require
`LAKESHORE_ADMIN_TOKEN` in the environment.

## Organizations

Namespaces carry an optional `orgId`. Namespaces sharing a non-null
`orgId` are treated as siblings.

- **Cross-namespace access**: a token scoped to `alice` (org `acme`) may
  call `/v1/namespaces/bob/...` when `bob` is also in org `acme`. A
  namespace with no `orgId` is reachable only by its own tokens.
- **Org-scoped listing**: `GET /v1/namespaces/:ns/providers` and
  `GET /v1/namespaces/:ns/queues` aggregate rows across every namespace
  in the org. Provider rows carry a `namespace` field so you can tell
  them apart; queue summaries do the same when the org has more than one
  namespace. Provider names must be unique **within the org**, not just
  within the namespace.

There is no `User` or `Organization` model — `orgId` is a plain string on
the `Namespace` row, set at create time or patched later (admin only):

```bash
curl -X POST $CP/v1/admin/namespaces \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "alice", "orgId": "acme"}'

curl -X PATCH $CP/v1/admin/namespaces/alice \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"orgId": "acme"}'
```

Namespace slugs match `^[a-z0-9][a-z0-9-]{0,63}$`. Creating a duplicate
returns `409`.

## Enroll tokens

A one-shot credential a daemon presents as a Bearer on
`POST /v1/daemon/hello` while enrolling its Ed25519 public key.

| Verb | Path | Notes |
| --- | --- | --- |
| `POST` | `/v1/namespaces/:ns/enroll-tokens` | Admin only. Returns `dle_…` plaintext once. |
| `GET` | `/v1/namespaces/:ns/enroll-tokens` | Metadata only. |
| `DELETE` | `/v1/namespaces/:ns/enroll-tokens/:id` | Admin only. |

Stored as `sha256` hex plus an 8-character display prefix and an operator
label (same label regex as namespace tokens). Single-use via
`consumedAt`, time-limited via `expiresAt` — default TTL 3600 s, hard cap
30 days.

The consume is atomic and happens **after** signature verification, so a
daemon that presents a bad self-signature does not burn the token.
Enrollment failures come back as HTTP `200` with a
`{ "reject": { "code", "message" } }` msgpack body, not as a 4xx — codes
include `enroll_token_required`, `invalid_enroll_token`,
`enroll_token_consumed`, `enroll_token_expired`, `key_id_mismatch`, and
`enroll_signature_invalid`. Full flow: [Key rotation](/nymph/key-rotation.md).

## Secrets

Server-stored credentials, AES-256-GCM-encrypted at rest, referenced from
provider kwargs as `{ "$secret": "<name>" }`. **Plaintext only flows
in.** No `GET` route ever returns it.

### Kinds

`kind` tells the consumer how to read the plaintext. Exactly four values
are accepted:

| `kind` | Plaintext shape | Consumer |
| --- | --- | --- |
| `ssh_key` | PEM / OpenSSH private key | SSH and SLURM launchers |
| `aws_keypair` | `<access_key_id>:<secret_access_key>` — colon-delimited, split on the **first** `:` | EC2 launcher, storage provisioning |
| `gcp_sa_json` | Raw service-account JSON | GCE launcher |
| `opaque` | Anything — the consumer decides | Any |

> **Warning:** The `$secret` substitution used by every launcher (`shapeBySecretKind`
> in `launchers/ec2.ts`) reads `aws_keypair` plaintext as
> `<access_key_id>:<secret_access_key>` and throws
> `aws_keypair secret must be formatted as <access_key_id>:<secret_access_key>`
> on a value with no `:`. The storage-provisioning path is the one
> exception: it *also* accepts a JSON object whose keys are
> `accessKeyId` / `access_key_id` / `AccessKeyId` (and the matching
> secret-key spellings), falling back to the colon form. Store the colon
> form and both paths work.

Names match `^[a-z0-9][a-z0-9-]*$` and are unique per namespace — a
duplicate returns `409`. Plaintext is capped at **64 KiB**; a larger body
returns `413`. An unknown `kind` or a malformed name returns `400`.

The row stores `ciphertext`, a 12-byte `nonce`, a 16-byte `tag`, plus
`name` / `kind` / `description` / timestamps. `GET` list and show return
metadata only — never `ciphertext`, `nonce`, or `tag`.

### Encryption at rest

AES-256-GCM via Node's `crypto`. Each encryption generates a fresh
12-byte nonce; the 16-byte auth tag is stored alongside. Decrypt throws
on a tag mismatch — GCM's tamper detection — and callers treat the throw
as "corrupted blob or wrong key" and fail closed.

The master key comes from `SECRETS_KEY`, base64-encoded 32 bytes:

```bash
openssl rand -base64 32
```

A `SECRETS_KEY` that decodes to anything other than exactly 32 bytes
throws rather than silently truncating.

> **Warning:** Rotating `SECRETS_KEY` without re-encrypting orphans every existing row.
> A rotation today means: mint the new key, decrypt and re-encrypt every
> row with it, then switch the env var. Plan the downtime.

### CLI

```bash
lakeshore secrets add <name> --kind <ssh_key|aws_keypair|gcp_sa_json|opaque> \
    [--from-file <path>] [--description <text>]
lakeshore secrets list
lakeshore secrets show <name>
lakeshore secrets rotate <name> [--from-file <path>]
lakeshore secrets remove <name>
```

`--kind` is required on `add`. Plaintext comes from `--from-file <path>`
or from piped stdin when the flag is absent; on a TTY with neither, the
command exits 2 with `use --from-file <path> or pipe plaintext via
stdin`.

```bash
$ printf '%s:%s' "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" \
    | lakeshore secrets add aws-prod-keys --kind aws_keypair
ok  secret "aws-prod-keys" created  (aws_keypair)  2026-05-18T05:42:00.000Z
```

> **Warning:** Both the CLI and the server treat the plaintext as a UTF-8 string. PEM,
> OpenSSH, and JSON credentials round-trip losslessly; arbitrary binary
> does not. There is also no `--from-file -` convention — omit the flag and
> pipe instead.

### `$secret` references

A provider's kwargs may carry secret references inline:

```json
{
  "name": "aws-us-east",
  "launcher": "EC2",
  "kwargs": {
    "region": "us-east-1",
    "aws_credentials": { "$secret": "aws-prod-keys" }
  }
}
```

At create and update time the server walks the kwargs tree with
`findSecretRefs()` (depth-capped at 20; a malformed marker or a depth
overflow is a `400`) and verifies every referenced name exists in the
namespace. A typoed name fails with `422 {"error": "unknown secret",
"secret": "<name>"}` at provider-create time rather than silently at
launch time.

`lakeshore providers show` and `lakeshore modes show` mask references as
`"<secret: name>"` in their YAML preview, so a copy-pasted output cannot
leak a name into the wrong place.

> **Warning:** `readSecretPlain()` is internal-only. It decrypts in memory for the
> launch path and the result is never written to disk, returned over the
> API, or logged. Never call it from an HTTP handler.

## Operator setup

```bash
heroku config:set \
  SECRETS_KEY="$(openssl rand -base64 32)" \
  LAKESHORE_ADMIN_TOKEN="$(openssl rand -base64 24 | tr '+/' '-_' | tr -d '=')" \
  --app <your-app>
```

For a local stack, leave both blank in `lakeshore-controlplane/.env`. The
control plane boots in open mode with the dev fallback key, and the CLI,
SDK, and dashboard all skip auth. Secrets stored under the fallback key
survive restarts but give zero security — a dev database is not
load-bearing.

To exercise the real auth path locally, set both, restart, then bootstrap
with `lakeshore auth login` as shown above.

## See also

- [Configuration](/api/configuration.md) — where `$secret` references live inside `.dreamrc` provider entries.
- [Key rotation](/nymph/key-rotation.md) — Ed25519 daemon identity, enrollment, and rotation.
- [Token lifecycle](/cli/happy-paths/15-token-lifecycle.md) — the mint-login-inherit walkthrough.
- [Auth and setup](/cli/examples/auth.md) — the CLI cookbook.