DreamLake
  1. Get Started
  2. CLI
  3. Python
  4. Nymph
  5. API
  6. Admin

CLI examples — Queues and jobs

Part of the CLI examples cookbook. Auth setup lives on Auth + setup.

Queues

A queue is the unified scheduling primitive: a named pipe of pending Invocations that daemons subscribe to by name. Workers register themselves into queues via Worker.queues: string[]; the empty array means the default queue (there is no sentinel name). See Queues and /dev/queues for the concept and design pages.

Queue verbs are irregular

It is lakeshore queues ls and lakeshore queues rm — not list and remove. Every other group in the CLI uses list / remove; this one doesn't.

The full verb set is add, ls, show, patch, archive, unarchive, drain, rm, events, stats.

queues add — create a queue

The bare minimum is a name (matching ^[a-z0-9][a-z0-9-]*$). --kind is one of fifo (default), priority, boltzmann, filo.

bash
lakeshore queues add training-h100
lakeshore queues add eval --kind priority --description "evaluation, urgent first"

queues add — with an elasticity declaration

--elasticity accepts fixed, fully-elastic, pool-with-threshold, or max-count, with --min, --max, and --threshold (0.0–1.0) supplying its parameters. --instance-type, --image-id, and --runner seed the daemon template used when the queue scales up. --max-depth plus --on-full (reject or block) bound the backlog.

bash
lakeshore queues add training-h100 \
  --kind priority \
  --provider my-gpu-provider \
  --elasticity pool-with-threshold \
  --min 2 --max 16 --threshold 0.7 \
  --instance-type p4d.24xlarge \
  --max-depth 1000 --on-full reject
Two spellings of the elasticity kinds

The --elasticity flag takes hyphenated values (fully-elastic, pool-with-threshold, max-count), while the elasticity.kind field in lakeshore.yaml accepts only the underscored spellings (fully_elastic, pool_with_threshold, max_count). They are not interchangeable.

queues ls — list queues

--state filters on active, draining, paused, or archived.

bash
lakeshore queues ls
lakeshore queues ls --state active
lakeshore queues ls --json | jq -r '.[].name'

The table shows NAME / KIND / STATE / MEMBERS / PROVIDER / ELASTICITY / DESCRIPTION. MEMBERS is the count of workers whose queues array contains this queue's name.

queues show — one queue and its members

bash
lakeshore queues show training-h100
lakeshore queues show training-h100 --json

queues patch — mutate a queue in place

Accepts --description, --provider, --min, --max, --threshold, --max-depth, --on-full, and --json. Note it does not accept --kind or --elasticity.

bash
lakeshore queues patch training-h100 --description "h100 prod"
lakeshore queues patch training-h100 --max 32 --threshold 0.6
lakeshore queues patch training-h100 --max-depth 2000 --on-full block
lakeshore queues patch training-h100 --provider ""    # clear the provider

queues drain / archive / unarchive

drain stops admitting new work but lets in-flight invocations finish. archive soft-deletes — the row survives but stops accepting work. unarchive re-activates a drained or archived queue. All three take a name and --json.

bash
lakeshore queues drain training-h100
lakeshore queues archive eval
lakeshore queues unarchive eval

queues rm — delete

Errors if any worker is still bound to the queue. --force cascades — it scrubs the queue name out of every Worker.queues array first.

bash
lakeshore queues rm eval
lakeshore queues rm eval --force

queues events — the elasticity audit trail

One row per controller decision. tick_noop rows are hidden unless you pass --verbose, which is the quickest way to check whether the controller is running at all. --limit defaults to 50, capped at 200.

bash
lakeshore queues events training-h100
lakeshore queues events training-h100 --verbose --limit 200
lakeshore queues events training-h100 --json

queues stats — current signals

A point-in-time view of what the controller is looking at: daemon count, utilization, pending invocations, and the last few non-noop scale actions.

bash
lakeshore queues stats training-h100
lakeshore queues stats training-h100 --json

Jobs

Invocations tracked by the control plane — the things @dls.udf calls queue up.

jobs list — recent invocations

--limit is 1–200 (default 50) and --offset defaults to 0.

bash
lakeshore jobs list
lakeshore jobs list --limit 20 --offset 20

jobs list — filters

--state is repeatable and accepts queued, running, succeeded, failed, killed, timeout. --worker and --queue take ids.

bash
lakeshore jobs list --state running --state queued
lakeshore jobs list --worker "$D"
lakeshore jobs list --json | jq '.[] | {id, state, worker}'

jobs show — one invocation

bash
JOB=$(lakeshore jobs list --json | jq -r '.[0].id // empty')
lakeshore jobs show "$JOB"

jobs kill — cancel one invocation

Queued jobs are marked killed immediately. Running jobs return 409 — to stop in-flight work, kill the process on the daemon (lakeshore daemon exec) or drop the daemon itself.

bash
lakeshore jobs kill "$JOB"

jobs bulk-cancel — cancel everything queued

Only touches rows in state queued; running work is untouched.

bash
lakeshore jobs bulk-cancel
lakeshore jobs bulk-cancel --queue training-h100
lakeshore jobs bulk-cancel --json

Read next