Storages

A Storage is a named reference to an S3-compatible bucket (or a scoped prefix inside one) registered under a namespace. The control plane keeps the long-lived credentials encrypted in its Secret store and hands out presigned URLs or short-lived STS credentials on request. Your scripts see a storage name; they never see an access key.

Two kinds, and only two:

Note the hyphen in s3-prefix. Omitting endpoint means AWS S3; setting it targets Ceph, MinIO, R2, or Backblaze B2 (the client switches to path-style addressing when an endpoint is present).

Immutability

bucket, prefix, and endpoint are denormalized out of the config onto the row and are immutable after creation. Repointing a storage would silently break every reference to it, so you create a new entry instead. Only description, creds, and region are mutable.

Two unique indexes back this: (namespace, name) and (namespace, bucket, prefix, endpoint) — the second de-duplicates two names pointing at the same target.

Register a storage

lakeshore storage add my-data \
  --kind s3 \
  --bucket my-lab-datasets \
  --region us-east-1 \
  --creds aws-lab

--creds <name> is shorthand for --kwarg 'creds.$secret=<name>'. The named secret must already exist in the namespace:

printf '%s:%s' "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" \
  | lakeshore secrets add aws-lab --kind aws_keypair

The storage code path accepts an aws_keypair secret in either the JSON form ({"accessKeyId": …, "secretAccessKey": …}) or the colon-delimited <access_key_id>:<secret_access_key> form. Everywhere else — the EC2/GCE/SSH $secret substitution — only the colon form parses, so store the colon form and the same secret works for both.

The name must match ^[a-z0-9][a-z0-9-]*$. You may prefix it with a namespace — lakeshore storage add team-robotics/training-data … targets that namespace instead of your default. The same <namespace>/<name> form works on show, update, remove, presign, and credentials (but not list).

Provisioning a bucket

--provision has the control plane create the bucket for you. It is only supported for kind: s3, requires a creds.$secret pointing at an aws_keypair, and needs an explicit --bucket — the name is not derived from the namespace.

lakeshore storage add scratch \
  --kind s3 \
  --bucket dreamlake-geyang-scratch \
  --region us-east-1 \
  --creds aws-lab \
  --provision \
  --s3-option ACL=private

The call is idempotent: a HeadBucket runs first, and an existing bucket you own is adopted rather than recreated (the response reports bucketCreated: false). --s3-option key=value entries are forwarded verbatim as CreateBucket parameters.

The verbs

storage is singular, unlike every sibling group. There are seven subcommands:

Presigned URLs

# Download URL (default), 1 hour
lakeshore storage presign my-data runs/exp-01/model.pt

# Upload URL
lakeshore storage presign my-data runs/exp-01/model.pt --put --expires-in 900

--expires-in defaults to 3600 seconds and is clamped to 86400. The key you pass is relative to the storage's configured prefix; the server joins them and returns the full key alongside the URL.

Then move the bytes yourself:

URL=$(lakeshore storage presign my-data runs/exp-01/model.pt --put)
curl -X PUT --upload-file ./model.pt "$URL"

Temporary credentials

For anything that wants a real S3 client — boto3, the AWS CLI, rclone, torch.save straight to S3 — ask for STS session credentials instead:

# Shell exports
eval "$(lakeshore storage credentials my-data --env)"
aws s3 sync ./checkpoints/ s3://my-lab-datasets/runs/exp-01/

# Or JSON, for scripting
lakeshore storage credentials my-data --json --duration 7200

--duration defaults to 3600 seconds and accepts 900–129600. The control plane calls STS GetSessionToken with the storage's own credentials.

From Python

from dreamlake.lakeshore import Storage

store = Storage("my-data")          # server/namespace/token resolved from env or auth.yml

url = store.presign("runs/exp-01/model.pt")                 # presigned GET
url = store.presign("runs/exp-01/model.pt", operation="put")

data = store.get("runs/exp-01/config.json")                 # -> bytes
store.put("runs/exp-01/out.json", b'{"loss": 0.01}')
text = store.get_text("runs/exp-01/notes.md")

creds = store.credentials(duration_seconds=7200)
session = boto3.Session(**creds.to_boto3_kwargs())
env = creds.to_env()      # AWS_ACCESS_KEY_ID / …_SECRET_ACCESS_KEY / …_SESSION_TOKEN / AWS_REGION

Storage resolves the control plane the same way the rest of the SDK does: an explicit base_url=, then LAKESHORE_URL, then ~/.config/lakeshore/auth.yml, then http://localhost:8080. The token comes from LAKESHORE_CLIENT_TOKEN (or auth.yml).

Shared storage across namespaces

Register the storage under the team namespace, then reference it by <namespace>/<name>:

# Admin registers it under team-robotics
lakeshore storage add team-robotics/training-data \
  --kind s3 --bucket dreamlake-team-robotics-data \
  --region us-east-1 --creds aws-team

# Any member of the org mints a URL against it
lakeshore storage presign team-robotics/training-data v2/dataset.tar

Cross-namespace access requires the two namespaces to share an orgId — a token scoped to one namespace in an org authorises requests against its siblings.

HTTP

POST   /v1/namespaces/:ns/storages
GET    /v1/namespaces/:ns/storages
GET    /v1/namespaces/:ns/storages/:name
PATCH  /v1/namespaces/:ns/storages/:name
DELETE /v1/namespaces/:ns/storages/:name
POST   /v1/namespaces/:ns/storages/:name/presign      { key, operation?, expiresIn? }
POST   /v1/namespaces/:ns/storages/:name/credentials  { durationSeconds? }

Deletion is a soft delete: the row is renamed with a tombstone suffix and deletedAt is stamped, so the unique index stays intact.

See also

• Payload store — the other S3 path, owned by the control plane and used for job arguments and results.
• Mounts — the registry for filesystems the runner will attach into a job's workdir.
• Auth and secrets — how $secret references and aws_keypair secrets work.
• Storage, code, and mounts examples — runnable end-to-end commands.