Auth and secrets
Two related concerns. Auth is how a caller proves itself to the control plane. Secrets are server-stored credential material the control plane decrypts in memory when it launches hosts on your behalf.
Both share the same dev ergonomic: in production the relevant env var is required and the server refuses to boot without it; in dev, omitting it degrades to an insecure fallback with a loud one-time warning.
Credential types
Four distinct bearer credentials. Do not mix them up.
| Credential | Format | Who holds it | What it grants |
|---|---|---|---|
| Admin token | operator-chosen, from LAKESHORE_ADMIN_TOKEN on the server | the operator | Everything, in every namespace. Required to mint the other two token types. |
| Namespace token | dlk_<base64url-32-bytes> | CLI, SDK, dashboard | CRUD in its own namespace (plus org siblings). |
| Enroll token | dle_<base64url-32-bytes> | a daemon, once | One-shot daemon enrollment on /v1/daemon/hello. |
| Daemon signature | Ed25519 over X-LS-* headers | a daemon, ongoing | Poll / ack / event. Covered on Key rotation. |
The dlk_ / dle_ prefixes are greppable so an accidental paste is
auditable, and they make the two token types impossible to confuse in a
log.
Where the auth hook fires
The onRequest auth hook matches exactly two path shapes:
/v1/admin/*— admin token or open mode only. Anything else gets403./v1/namespaces/:ns/*— the tiered check below.
Every other route is outside the hook, including all of
/v1/daemon/* and /v1/producer/*. Those either carry their own
in-route verification (daemon signatures) or are authenticated by an
unguessable id in the URL.
One explicit exemption inside the namespaced space:
/v1/namespaces/:ns/workers/:id/exec/:exec_id/result bypasses namespace
auth entirely — it is a daemon callback authenticated by the exec_id.
The tiered check
- Open mode. When
LAKESHORE_ADMIN_TOKENis unset, every request passes withauthMode = "open". - Missing bearer →
401 missing bearer token. - Admin. Bearer equals
LAKESHORE_ADMIN_TOKEN(constant-time compare) →authMode = "admin", full access to any namespace. - Namespace token. Look up a
Tokenrow byhash = sha256(bearer). Unknown →401 invalid token. Revoked →403 token revoked. Namespace mismatch →403, unless both namespaces share a non-nullorgId. On successauthMode = "token"andlastUsedAtis bumped asynchronously.
authMode is one of admin, token, open, daemon-sig,
daemon-bearer — note the hyphens. whoami only ever reports the first
three.
Only four routes require the admin tier beyond the hook itself: minting
a namespace token, minting an enroll token, deleting an enroll token, and
uploading a nymph release. requireAdmin() returns true for authMode
admin or open.
Server env config
- Production (
NODE_ENV=production) — the server logs a fatal and exits when either is missing. - Dev — leaving
LAKESHORE_ADMIN_TOKENblank enters open mode with one loud warn at startup. LeavingSECRETS_KEYblank falls back to a hard-codedDEV_FALLBACK_KEY, also with a warn.
The server formerly read ADMIN_TOKEN with no prefix. It is still
accepted, with a one-time stderr deprecation warning. Migrate .env
files and Heroku config vars to LAKESHORE_ADMIN_TOKEN before the
back-compat goes away.
Client-side env vars
These are easy to conflate — they are read by different programs and are not interchangeable.
| Var | Read by | Effect |
|---|---|---|
LAKESHORE_URL | every CLI command except auth; the Python SDK | Names the control plane. When set, the CLI sends no bearer at all — it deliberately bypasses the saved auth file. |
LAKESHORE_SERVER | lakeshore auth login | status only; the nymph daemon | Server URL for the auth commands and for the daemon's --server. |
LAKESHORE_NAMESPACE | CLI (with LAKESHORE_URL), Python SDK | Namespace slug. Defaults to default. |
LAKESHORE_CLIENT_TOKEN | the Python SDK's default dispatch | Bearer for HttpDispatch. lakeshore worker passes the token to its Python child this way — never on argv. |
LAKESHORE_ADMIN_TOKEN | the CP server (validation); the CLI's admin group and auth login | On the server host, and once in your shell when bootstrapping. |
lakeshore worker start / worker once deliberately do not consult
auth.yml. The plane a worker drains must come from --url or
LAKESHORE_URL. Running lakeshore auth login is not sufficient setup
for a worker.
CLI auth commands
Server URL precedence for these three commands, highest first:
--server, LAKESHORE_SERVER, the saved auth file, then
http://localhost:8080.
login resolves a token, validates it by calling
GET <server>/v1/namespaces/<ns>/whoami with a Bearer header, and saves
on success. status reloads the saved auth and re-validates. logout
deletes the file.
Credentials land at $XDG_CONFIG_HOME/lakeshore/auth.yml (defaulting to
~/.config/lakeshore/auth.yml), written chmod 600:
First-time bootstrap via the admin token
If --token is absent and LAKESHORE_ADMIN_TOKEN is set in your shell,
login mints a fresh namespace token through the admin path and saves
that — not the admin token — to auth.yml:
The label actually stored is <--name>-<timestamp>; the suffix keeps a
re-run from colliding with the per-namespace unique-name constraint. With
neither --token nor LAKESHORE_ADMIN_TOKEN, login prompts on a TTY
and fails with exit 2 otherwise.
whoami
GET /v1/namespaces/:ns/whoami returns the caller's view. Reaching 200
is what auth login treats as validation.
For an admin or open-mode caller there is no backing Token row, so the
route returns a synthetic envelope: tokenPrefix: null and tokenName
of <admin> or <open>.
Token CRUD
| Verb | Path | Notes |
|---|---|---|
POST | /v1/namespaces/:ns/tokens | Admin only. 201. Returns plaintext once. |
GET | /v1/namespaces/:ns/tokens | Metadata only. |
DELETE | /v1/namespaces/:ns/tokens/:id | Soft delete — sets revokedAt. |
Mint body is { "name": "cli-ge" }. Labels match
^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$ (underscores allowed, unlike the
lowercase-and-hyphen regex used for resource names) and are unique per
namespace. The mint response is {id, name, prefix, plaintext, createdAt}.
Only hash = sha256(plaintext) (globally unique) and prefix (the first
8 characters of the plaintext, for display) are stored. Revocation is a
soft delete so audit listings keep the row; a revoked token returns
403.
The mint response is the only time a token's plaintext leaves the server. There is no recovery path — lose it and the remedy is revoke plus mint a new one.
The CLI wraps the same routes under lakeshore admin tokens create | list | revoke (alias ls for list), all of which hard-require
LAKESHORE_ADMIN_TOKEN in the environment.
Organizations
Namespaces carry an optional orgId. Namespaces sharing a non-null
orgId are treated as siblings.
- Cross-namespace access: a token scoped to
alice(orgacme) may call/v1/namespaces/bob/...whenbobis also in orgacme. A namespace with noorgIdis reachable only by its own tokens. - Org-scoped listing:
GET /v1/namespaces/:ns/providersandGET /v1/namespaces/:ns/queuesaggregate rows across every namespace in the org. Provider rows carry anamespacefield so you can tell them apart; queue summaries do the same when the org has more than one namespace. Provider names must be unique within the org, not just within the namespace.
There is no User or Organization model — orgId is a plain string on
the Namespace row, set at create time or patched later (admin only):
Namespace slugs match ^[a-z0-9][a-z0-9-]{0,63}$. Creating a duplicate
returns 409.
Enroll tokens
A one-shot credential a daemon presents as a Bearer on
POST /v1/daemon/hello while enrolling its Ed25519 public key.
| Verb | Path | Notes |
|---|---|---|
POST | /v1/namespaces/:ns/enroll-tokens | Admin only. Returns dle_… plaintext once. |
GET | /v1/namespaces/:ns/enroll-tokens | Metadata only. |
DELETE | /v1/namespaces/:ns/enroll-tokens/:id | Admin only. |
Stored as sha256 hex plus an 8-character display prefix and an operator
label (same label regex as namespace tokens). Single-use via
consumedAt, time-limited via expiresAt — default TTL 3600 s, hard cap
30 days.
The consume is atomic and happens after signature verification, so a
daemon that presents a bad self-signature does not burn the token.
Enrollment failures come back as HTTP 200 with a
{ "reject": { "code", "message" } } msgpack body, not as a 4xx — codes
include enroll_token_required, invalid_enroll_token,
enroll_token_consumed, enroll_token_expired, key_id_mismatch, and
enroll_signature_invalid. Full flow: Key rotation.
Secrets
Server-stored credentials, AES-256-GCM-encrypted at rest, referenced from
provider kwargs as { "$secret": "<name>" }. Plaintext only flows
in. No GET route ever returns it.
Kinds
kind tells the consumer how to read the plaintext. Exactly four values
are accepted:
kind | Plaintext shape | Consumer |
|---|---|---|
ssh_key | PEM / OpenSSH private key | SSH and SLURM launchers |
aws_keypair | <access_key_id>:<secret_access_key> — colon-delimited, split on the first : | EC2 launcher, storage provisioning |
gcp_sa_json | Raw service-account JSON | GCE launcher |
opaque | Anything — the consumer decides | Any |
The $secret substitution used by every launcher (shapeBySecretKind
in launchers/ec2.ts) reads aws_keypair plaintext as
<access_key_id>:<secret_access_key> and throws
aws_keypair secret must be formatted as <access_key_id>:<secret_access_key>
on a value with no :. The storage-provisioning path is the one
exception: it also accepts a JSON object whose keys are
accessKeyId / access_key_id / AccessKeyId (and the matching
secret-key spellings), falling back to the colon form. Store the colon
form and both paths work.
Names match ^[a-z0-9][a-z0-9-]*$ and are unique per namespace — a
duplicate returns 409. Plaintext is capped at 64 KiB; a larger body
returns 413. An unknown kind or a malformed name returns 400.
The row stores ciphertext, a 12-byte nonce, a 16-byte tag, plus
name / kind / description / timestamps. GET list and show return
metadata only — never ciphertext, nonce, or tag.
Encryption at rest
AES-256-GCM via Node's crypto. Each encryption generates a fresh
12-byte nonce; the 16-byte auth tag is stored alongside. Decrypt throws
on a tag mismatch — GCM's tamper detection — and callers treat the throw
as "corrupted blob or wrong key" and fail closed.
The master key comes from SECRETS_KEY, base64-encoded 32 bytes:
A SECRETS_KEY that decodes to anything other than exactly 32 bytes
throws rather than silently truncating.
Rotating SECRETS_KEY without re-encrypting orphans every existing row.
A rotation today means: mint the new key, decrypt and re-encrypt every
row with it, then switch the env var. Plan the downtime.
CLI
--kind is required on add. Plaintext comes from --from-file <path>
or from piped stdin when the flag is absent; on a TTY with neither, the
command exits 2 with use --from-file <path> or pipe plaintext via stdin.
Both the CLI and the server treat the plaintext as a UTF-8 string. PEM,
OpenSSH, and JSON credentials round-trip losslessly; arbitrary binary
does not. There is also no --from-file - convention — omit the flag and
pipe instead.
$secret references
A provider's kwargs may carry secret references inline:
At create and update time the server walks the kwargs tree with
findSecretRefs() (depth-capped at 20; a malformed marker or a depth
overflow is a 400) and verifies every referenced name exists in the
namespace. A typoed name fails with 422 {"error": "unknown secret", "secret": "<name>"} at provider-create time rather than silently at
launch time.
lakeshore providers show and lakeshore modes show mask references as
"<secret: name>" in their YAML preview, so a copy-pasted output cannot
leak a name into the wrong place.
readSecretPlain() is internal-only. It decrypts in memory for the
launch path and the result is never written to disk, returned over the
API, or logged. Never call it from an HTTP handler.
Operator setup
For a local stack, leave both blank in lakeshore-controlplane/.env. The
control plane boots in open mode with the dev fallback key, and the CLI,
SDK, and dashboard all skip auth. Secrets stored under the fallback key
survive restarts but give zero security — a dev database is not
load-bearing.
To exercise the real auth path locally, set both, restart, then bootstrap
with lakeshore auth login as shown above.
See also
- Configuration — where
$secretreferences live inside.dreamrcprovider entries. - Key rotation — Ed25519 daemon identity, enrollment, and rotation.
- Token lifecycle — the mint-login-inherit walkthrough.
- Auth and setup — the CLI cookbook.