DreamLake

Auth and secrets

Two related concerns. Auth is how a caller proves itself to the control plane. Secrets are server-stored credential material the control plane decrypts in memory when it launches hosts on your behalf.

Both share the same dev ergonomic: in production the relevant env var is required and the server refuses to boot without it; in dev, omitting it degrades to an insecure fallback with a loud one-time warning.

Credential types

Four distinct bearer credentials. Do not mix them up.

CredentialFormatWho holds itWhat it grants
Admin tokenoperator-chosen, from LAKESHORE_ADMIN_TOKEN on the serverthe operatorEverything, in every namespace. Required to mint the other two token types.
Namespace tokendlk_<base64url-32-bytes>CLI, SDK, dashboardCRUD in its own namespace (plus org siblings).
Enroll tokendle_<base64url-32-bytes>a daemon, onceOne-shot daemon enrollment on /v1/daemon/hello.
Daemon signatureEd25519 over X-LS-* headersa daemon, ongoingPoll / ack / event. Covered on Key rotation.

The dlk_ / dle_ prefixes are greppable so an accidental paste is auditable, and they make the two token types impossible to confuse in a log.

Where the auth hook fires

The onRequest auth hook matches exactly two path shapes:

  • /v1/admin/* — admin token or open mode only. Anything else gets 403.
  • /v1/namespaces/:ns/* — the tiered check below.

Every other route is outside the hook, including all of /v1/daemon/* and /v1/producer/*. Those either carry their own in-route verification (daemon signatures) or are authenticated by an unguessable id in the URL.

One explicit exemption inside the namespaced space: /v1/namespaces/:ns/workers/:id/exec/:exec_id/result bypasses namespace auth entirely — it is a daemon callback authenticated by the exec_id.

The tiered check

  1. Open mode. When LAKESHORE_ADMIN_TOKEN is unset, every request passes with authMode = "open".
  2. Missing bearer → 401 missing bearer token.
  3. Admin. Bearer equals LAKESHORE_ADMIN_TOKEN (constant-time compare) → authMode = "admin", full access to any namespace.
  4. Namespace token. Look up a Token row by hash = sha256(bearer). Unknown → 401 invalid token. Revoked → 403 token revoked. Namespace mismatch → 403, unless both namespaces share a non-null orgId. On success authMode = "token" and lastUsedAt is bumped asynchronously.

authMode is one of admin, token, open, daemon-sig, daemon-bearer — note the hyphens. whoami only ever reports the first three.

Only four routes require the admin tier beyond the hook itself: minting a namespace token, minting an enroll token, deleting an enroll token, and uploading a nymph release. requireAdmin() returns true for authMode admin or open.

Server env config

lakeshore-controlplane/.envbash
# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 24 | tr '+/' '-_' | tr -d '='
LAKESHORE_ADMIN_TOKEN=""

# Required in production — the server exits at boot without it.
# Generate with: openssl rand -base64 32
SECRETS_KEY=""
  • Production (NODE_ENV=production) — the server logs a fatal and exits when either is missing.
  • Dev — leaving LAKESHORE_ADMIN_TOKEN blank enters open mode with one loud warn at startup. Leaving SECRETS_KEY blank falls back to a hard-coded DEV_FALLBACK_KEY, also with a warn.
`ADMIN_TOKEN` is still read as a fallback

The server formerly read ADMIN_TOKEN with no prefix. It is still accepted, with a one-time stderr deprecation warning. Migrate .env files and Heroku config vars to LAKESHORE_ADMIN_TOKEN before the back-compat goes away.

Client-side env vars

These are easy to conflate — they are read by different programs and are not interchangeable.

VarRead byEffect
LAKESHORE_URLevery CLI command except auth; the Python SDKNames the control plane. When set, the CLI sends no bearer at all — it deliberately bypasses the saved auth file.
LAKESHORE_SERVERlakeshore auth login | status only; the nymph daemonServer URL for the auth commands and for the daemon's --server.
LAKESHORE_NAMESPACECLI (with LAKESHORE_URL), Python SDKNamespace slug. Defaults to default.
LAKESHORE_CLIENT_TOKENthe Python SDK's default dispatchBearer for HttpDispatch. lakeshore worker passes the token to its Python child this way — never on argv.
LAKESHORE_ADMIN_TOKENthe CP server (validation); the CLI's admin group and auth loginOn the server host, and once in your shell when bootstrapping.
`lakeshore worker` ignores saved credentials

lakeshore worker start / worker once deliberately do not consult auth.yml. The plane a worker drains must come from --url or LAKESHORE_URL. Running lakeshore auth login is not sufficient setup for a worker.

CLI auth commands

bash
lakeshore auth login [--server <url>] [--namespace <slug>] [--token <plaintext>] [--name <label>]
lakeshore auth status [--server <url>] [--namespace <slug>]
lakeshore auth logout

Server URL precedence for these three commands, highest first: --server, LAKESHORE_SERVER, the saved auth file, then http://localhost:8080.

login resolves a token, validates it by calling GET <server>/v1/namespaces/<ns>/whoami with a Bearer header, and saves on success. status reloads the saved auth and re-validates. logout deletes the file.

Credentials land at $XDG_CONFIG_HOME/lakeshore/auth.yml (defaulting to ~/.config/lakeshore/auth.yml), written chmod 600:

~/.config/lakeshore/auth.ymlyaml
server: https://api.lakeshore.dreamlake.ai
namespace: default
token: dlk_xxxxxxxxxxxx…

First-time bootstrap via the admin token

If --token is absent and LAKESHORE_ADMIN_TOKEN is set in your shell, login mints a fresh namespace token through the admin path and saves that — not the admin token — to auth.yml:

bash
export LAKESHORE_ADMIN_TOKEN=...          # same value as the server's .env
lakeshore auth login --server http://localhost:8080 --namespace dev --name cli-$USER
lakeshore auth status                     # → authMode: token

The label actually stored is <--name>-<timestamp>; the suffix keeps a re-run from colliding with the per-namespace unique-name constraint. With neither --token nor LAKESHORE_ADMIN_TOKEN, login prompts on a TTY and fails with exit 2 otherwise.

whoami

GET /v1/namespaces/:ns/whoami returns the caller's view. Reaching 200 is what auth login treats as validation.

json
{
  "namespace": "default",
  "tokenPrefix": "dlk_xxxx",
  "tokenName": "cli-ge-2026-05-18T05-40-00",
  "lastUsedAt": "2026-05-18T05:40:00.000Z",
  "authMode": "token"
}

For an admin or open-mode caller there is no backing Token row, so the route returns a synthetic envelope: tokenPrefix: null and tokenName of <admin> or <open>.

Token CRUD

VerbPathNotes
POST/v1/namespaces/:ns/tokensAdmin only. 201. Returns plaintext once.
GET/v1/namespaces/:ns/tokensMetadata only.
DELETE/v1/namespaces/:ns/tokens/:idSoft delete — sets revokedAt.

Mint body is { "name": "cli-ge" }. Labels match ^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$ (underscores allowed, unlike the lowercase-and-hyphen regex used for resource names) and are unique per namespace. The mint response is {id, name, prefix, plaintext, createdAt}.

Only hash = sha256(plaintext) (globally unique) and prefix (the first 8 characters of the plaintext, for display) are stored. Revocation is a soft delete so audit listings keep the row; a revoked token returns 403.

Plaintext is shown once

The mint response is the only time a token's plaintext leaves the server. There is no recovery path — lose it and the remedy is revoke plus mint a new one.

The CLI wraps the same routes under lakeshore admin tokens create | list | revoke (alias ls for list), all of which hard-require LAKESHORE_ADMIN_TOKEN in the environment.

Organizations

Namespaces carry an optional orgId. Namespaces sharing a non-null orgId are treated as siblings.

  • Cross-namespace access: a token scoped to alice (org acme) may call /v1/namespaces/bob/... when bob is also in org acme. A namespace with no orgId is reachable only by its own tokens.
  • Org-scoped listing: GET /v1/namespaces/:ns/providers and GET /v1/namespaces/:ns/queues aggregate rows across every namespace in the org. Provider rows carry a namespace field so you can tell them apart; queue summaries do the same when the org has more than one namespace. Provider names must be unique within the org, not just within the namespace.

There is no User or Organization model — orgId is a plain string on the Namespace row, set at create time or patched later (admin only):

bash
curl -X POST $CP/v1/admin/namespaces \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "alice", "orgId": "acme"}'

curl -X PATCH $CP/v1/admin/namespaces/alice \
  -H "Authorization: Bearer $LAKESHORE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"orgId": "acme"}'

Namespace slugs match ^[a-z0-9][a-z0-9-]{0,63}$. Creating a duplicate returns 409.

Enroll tokens

A one-shot credential a daemon presents as a Bearer on POST /v1/daemon/hello while enrolling its Ed25519 public key.

VerbPathNotes
POST/v1/namespaces/:ns/enroll-tokensAdmin only. Returns dle_… plaintext once.
GET/v1/namespaces/:ns/enroll-tokensMetadata only.
DELETE/v1/namespaces/:ns/enroll-tokens/:idAdmin only.

Stored as sha256 hex plus an 8-character display prefix and an operator label (same label regex as namespace tokens). Single-use via consumedAt, time-limited via expiresAt — default TTL 3600 s, hard cap 30 days.

The consume is atomic and happens after signature verification, so a daemon that presents a bad self-signature does not burn the token. Enrollment failures come back as HTTP 200 with a { "reject": { "code", "message" } } msgpack body, not as a 4xx — codes include enroll_token_required, invalid_enroll_token, enroll_token_consumed, enroll_token_expired, key_id_mismatch, and enroll_signature_invalid. Full flow: Key rotation.

Secrets

Server-stored credentials, AES-256-GCM-encrypted at rest, referenced from provider kwargs as { "$secret": "<name>" }. Plaintext only flows in. No GET route ever returns it.

Kinds

kind tells the consumer how to read the plaintext. Exactly four values are accepted:

kindPlaintext shapeConsumer
ssh_keyPEM / OpenSSH private keySSH and SLURM launchers
aws_keypair<access_key_id>:<secret_access_key> — colon-delimited, split on the first :EC2 launcher, storage provisioning
gcp_sa_jsonRaw service-account JSONGCE launcher
opaqueAnything — the consumer decidesAny
`aws_keypair` is colon-delimited, not JSON

The $secret substitution used by every launcher (shapeBySecretKind in launchers/ec2.ts) reads aws_keypair plaintext as <access_key_id>:<secret_access_key> and throws aws_keypair secret must be formatted as <access_key_id>:<secret_access_key> on a value with no :. The storage-provisioning path is the one exception: it also accepts a JSON object whose keys are accessKeyId / access_key_id / AccessKeyId (and the matching secret-key spellings), falling back to the colon form. Store the colon form and both paths work.

Names match ^[a-z0-9][a-z0-9-]*$ and are unique per namespace — a duplicate returns 409. Plaintext is capped at 64 KiB; a larger body returns 413. An unknown kind or a malformed name returns 400.

The row stores ciphertext, a 12-byte nonce, a 16-byte tag, plus name / kind / description / timestamps. GET list and show return metadata only — never ciphertext, nonce, or tag.

Encryption at rest

AES-256-GCM via Node's crypto. Each encryption generates a fresh 12-byte nonce; the 16-byte auth tag is stored alongside. Decrypt throws on a tag mismatch — GCM's tamper detection — and callers treat the throw as "corrupted blob or wrong key" and fail closed.

The master key comes from SECRETS_KEY, base64-encoded 32 bytes:

bash
openssl rand -base64 32

A SECRETS_KEY that decodes to anything other than exactly 32 bytes throws rather than silently truncating.

There is no key versioning

Rotating SECRETS_KEY without re-encrypting orphans every existing row. A rotation today means: mint the new key, decrypt and re-encrypt every row with it, then switch the env var. Plan the downtime.

CLI

bash
lakeshore secrets add <name> --kind <ssh_key|aws_keypair|gcp_sa_json|opaque> \
    [--from-file <path>] [--description <text>]
lakeshore secrets list
lakeshore secrets show <name>
lakeshore secrets rotate <name> [--from-file <path>]
lakeshore secrets remove <name>

--kind is required on add. Plaintext comes from --from-file <path> or from piped stdin when the flag is absent; on a TTY with neither, the command exits 2 with use --from-file <path> or pipe plaintext via stdin.

bash
$ printf '%s:%s' "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" \
    | lakeshore secrets add aws-prod-keys --kind aws_keypair
ok  secret "aws-prod-keys" created  (aws_keypair)  2026-05-18T05:42:00.000Z
Plaintext is read as UTF-8 text

Both the CLI and the server treat the plaintext as a UTF-8 string. PEM, OpenSSH, and JSON credentials round-trip losslessly; arbitrary binary does not. There is also no --from-file - convention — omit the flag and pipe instead.

$secret references

A provider's kwargs may carry secret references inline:

json
{
  "name": "aws-us-east",
  "launcher": "EC2",
  "kwargs": {
    "region": "us-east-1",
    "aws_credentials": { "$secret": "aws-prod-keys" }
  }
}

At create and update time the server walks the kwargs tree with findSecretRefs() (depth-capped at 20; a malformed marker or a depth overflow is a 400) and verifies every referenced name exists in the namespace. A typoed name fails with 422 {"error": "unknown secret", "secret": "<name>"} at provider-create time rather than silently at launch time.

lakeshore providers show and lakeshore modes show mask references as "<secret: name>" in their YAML preview, so a copy-pasted output cannot leak a name into the wrong place.

Plaintext stays inside the server process

readSecretPlain() is internal-only. It decrypts in memory for the launch path and the result is never written to disk, returned over the API, or logged. Never call it from an HTTP handler.

Operator setup

bash
heroku config:set \
  SECRETS_KEY="$(openssl rand -base64 32)" \
  LAKESHORE_ADMIN_TOKEN="$(openssl rand -base64 24 | tr '+/' '-_' | tr -d '=')" \
  --app <your-app>

For a local stack, leave both blank in lakeshore-controlplane/.env. The control plane boots in open mode with the dev fallback key, and the CLI, SDK, and dashboard all skip auth. Secrets stored under the fallback key survive restarts but give zero security — a dev database is not load-bearing.

To exercise the real auth path locally, set both, restart, then bootstrap with lakeshore auth login as shown above.

See also