Storage, code, and mounts

Part of the CLI examples cookbook. Auth setup lives on Auth + setup.

Storage

lakeshore storage registers S3-compatible object stores. Note the verb is singular — storage, not storages — even though every sibling group is plural.

Two kinds:

Both also accept region, endpoint (for MinIO / R2 / Ceph / Backblaze), and creds.$secret pointing at a registered aws_keypair secret.

storage add — register a backend

The common fields have direct flags. --creds <secret> is shorthand for --kwarg creds.$secret=<name>.

lakeshore storage add my-s3 \
  --kind s3 \
  --bucket my-bucket \
  --region us-east-1 \
  --creds aws-prod-keys

lakeshore storage add training-data \
  --kind s3-prefix \
  --bucket my-bucket \
  --prefix datasets/imagenet \
  --creds aws-prod-keys \
  --description "read-only imagenet source"

Non-AWS S3 needs --endpoint:

lakeshore storage add r2-cache \
  --kind s3 \
  --bucket cache \
  --endpoint https://<account>.r2.cloudflarestorage.com \
  --creds r2-keys

Longer configs can come from a file, and --kwarg sets arbitrary config fields with dotted keys nesting:

lakeshore storage add my-s3 --kind s3 --config-file /tmp/storage.yaml
lakeshore storage add my-s3 --kind s3 --kwarg bucket=my-bucket --kwarg 'creds.$secret=aws-prod-keys'

storage add --provision — create the bucket too

Requires creds to resolve to an aws_keypair secret. The controlplane calls CreateBucket on your behalf; it's idempotent if the bucket already exists under the same account. --s3-option is repeatable and is forwarded verbatim as a CreateBucket parameter.

lakeshore storage add new-bucket \
  --kind s3 \
  --bucket lakeshore-scratch-42 \
  --region us-east-1 \
  --creds aws-prod-keys \
  --provision \
  --s3-option ACL=private

storage list / show

lakeshore storage list
lakeshore storage list --json
lakeshore storage show my-s3
lakeshore storage show my-s3 --json

storage update — patch the config

--kwarg deep-merges (pass null to drop a key); --config-file replaces the config wholesale. --description updates the description.

lakeshore storage update my-s3 --kwarg region=us-west-2
lakeshore storage update my-s3 --description "prod artifacts"

storage remove — delete the entry

Deletes the record only. --purge also deletes the S3 bucket, which needs credentials in the config.

lakeshore storage remove my-s3
lakeshore storage remove my-s3 --purge

storage presign — a presigned URL

Two positionals: the storage name and the object key (relative to the storage prefix). Defaults to a GET; --put presigns an upload. --expires-in defaults to 3600 seconds and is capped at 86400.

lakeshore storage presign my-s3 path/to/artifact.tar.gz
lakeshore storage presign my-s3 path/to/artifact.tar.gz --expires-in 600
lakeshore storage presign my-s3 uploads/new.bin --put

Because the result is a plain URL, curl is the object-verb surface:

URL=$(lakeshore storage presign my-s3 hello.txt --put --expires-in 600)
curl -X PUT --upload-file ./hello.txt "$URL"
curl -s "$(lakeshore storage presign my-s3 hello.txt)"

storage credentials — temporary STS credentials

Short-lived credentials a local script or SDK can use against the bucket directly. --duration defaults to 3600 seconds and accepts 900–129600.

lakeshore storage credentials my-s3
lakeshore storage credentials my-s3 --duration 7200 --json
eval "$(lakeshore storage credentials my-s3 --env)"   # export into this shell

Code

lakeshore code archives the current git tree and uploads it to a storage entry, so a remote daemon can pull the exact snapshot.

code push — upload a snapshot

Deduped by (git remote, commit) — re-running on the same clean commit is a no-op that reports already archived. The default storage entry name is the literal code-staging.

lakeshore code push
lakeshore code push --storage my-s3

An uncommitted working tree is refused (exit 2) unless you pass --dirty, which stashes, archives, and pops.

lakeshore code push --dirty

If the storage entry doesn't exist yet, the failure message tells you how to create it:

lakeshore storage add code-staging --kind s3 \
  --kwarg bucket=<your-bucket> --kwarg 'creds.$secret=<aws-creds>'

code list — show pushed snapshots

lakeshore code list
lakeshore code list --remote https://github.com/you/your-repo.git
lakeshore code list --json

Mounts

A mount is a declaration the runner attaches into the job workdir. The CLI writes metadata; the daemon does the actual mounting.

--kind is required and must be one of:

nfs  samba  ftp  sftp  s3  s3fs  google_drive  dropbox  bind  configmap

Each kind has its own required config fields, validated server-side:

s3 is the userspace-client form (copy on read); s3fs is the FUSE driver that gives job code a real filesystem path.

mounts add

# NFS export
lakeshore mounts add datasets \
  --kind nfs \
  --kwarg server=10.0.0.10 \
  --kwarg path=/srv/datasets

# S3 via s3fs; creds.$secret references a registered secret by name
lakeshore mounts add s3-cache \
  --kind s3fs \
  --kwarg bucket=my-bucket \
  --kwarg prefix=cache/ \
  --kwarg 'creds.$secret=s3-creds'

# Host bind
lakeshore mounts add hf-cache \
  --kind bind \
  --kwarg hostPath=/home/ge/.cache/huggingface \
  --kwarg readOnly=true

# Kubernetes ConfigMap
lakeshore mounts add my-cm \
  --kind configmap \
  --kwarg kubeNamespace=default \
  --kwarg name=app-cm

--config-file supplies the same fields from YAML or JSON. --kind stays on the command line either way:

cat > /tmp/mount.yaml <<'YAML'
server: 10.0.0.10
path: /srv/shared
YAML
lakeshore mounts add shared --kind nfs --config-file /tmp/mount.yaml

Any $secret marker inside the config must name a secret that exists in the same namespace, or the write is rejected with a 422.

mounts list / show / update / remove

lakeshore mounts list
lakeshore mounts list --json
lakeshore mounts show datasets --json

lakeshore mounts update datasets --kwarg path=/srv/datasets-v2
lakeshore mounts remove datasets

Read next

• Storages · Mounts
• Payloads — how job inputs and outputs move.
• Secrets, modes, tunnels — registering the credentials these reference.