Queues

A Queue is the unified scheduling primitive in Lakeshore. It is a named pipe of pending work; daemons subscribe to it by name; the control plane hands each queued Invocation to whichever subscribed daemon claims it first.

There is exactly one primitive at the scheduling layer. Workers carry a queues: string[] array recording which queues they pull from, and the empty array means the default queue. Everything else — elasticity, admission, work-stealing — is a policy field on the queue row, not a second primitive.

       producer                control plane                  workers
       ────────                ─────────────                  ───────

   submit(invocation)
   queue=training-h100  ───▶   ┌──────────────┐
                               │   training-  │  ◀── poll ──  daemon-A (queues=["training-h100"])
                               │     h100     │
                               │              │  ◀── poll ──  daemon-B (queues=["training-h100","eval"])
                               └──────────────┘
                               ┌──────────────┐
                               │     eval     │  ◀── poll ──  daemon-B
                               └──────────────┘
                               ┌──────────────┐
                               │   default    │  ◀── poll ──  daemon-C (queues=[])
                               └──────────────┘

Queue names must match ^[a-z0-9][a-z0-9-]*$.

What is on a queue row

Membership

A daemon's queue membership lives on the worker row, not the queue row — Worker.queues: string[] is the source of truth. queues show walks the workers and reports who is pulling from a queue right now.

Default-queue membership is the empty array. A daemon launched without --queue lands in the default queue. There is no sentinel string written into Worker.queues by the system. (The membership lookup does also accept a literal "default" entry if you put one there, but the explicit add/remove routes reject default — the empty array is the contract.)

Lifecycle

lakeshore queues drain training-h100      # stop admitting; finish in-flight
lakeshore queues archive training-h100    # soft-delete, keep the row
lakeshore queues unarchive training-h100  # back to active (reverses both)
lakeshore queues rm training-h100         # hard delete

rm refuses with a 409 while the queue has members; --force cascades by scrubbing the name out of every Worker.queues array first.

The schema also declares a paused state, but no route can set it — active, draining, and archived are the three reachable values.

The default queue

You do not have to create it. The control plane materialises a default row on the first queues ls or queues show default in a namespace, with kind: "fifo" and the description "Default queue (auto-created). Cannot be archived or deleted." It is a reserved name: archive and rm both return 409.

lakeshore daemon launch --provider my-ec2 --name scratch --wait
lakeshore daemon show scratch     # queues: []  — i.e. the default queue

Organization visibility

When namespaces are grouped by a shared orgId, queues ls aggregates queues from every namespace in the org and stamps a namespace field on each row so you can tell who owns it. Two namespaces in one org may hold same-named queues; they are distinct rows scoped to their own namespace.

A token scoped to one namespace in the org also authorises requests against sibling namespaces — put the sibling slug in the URL path:

curl -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/bob/queues/inference

A worked example

Spin up two queues — one for CPU pre-processing, one for GPU training — and a few daemons for each.

# Create the queues. Both fixed for now.
lakeshore queues add cpu-prep      --kind fifo     --elasticity fixed
lakeshore queues add training-h100 --kind priority --elasticity fixed

# Launch daemons into each. --queue is repeatable.
lakeshore daemon launch --provider my-ec2 --queue cpu-prep      --count 4 --name prep  --wait
lakeshore daemon launch --provider my-gpu --queue training-h100 --count 2 --name train --wait

# What's where?
lakeshore queues ls
lakeshore queues ls --state active
lakeshore daemon list
lakeshore queues show training-h100    # members + full queue row
lakeshore queues stats training-h100   # daemon_count / utilization / pending

Edit membership on a running daemon — no relaunch needed. The daemon picks the change up on its next poll:

lakeshore daemon update train-00 --add-queue eval
lakeshore daemon update train-00 --remove-queue eval

Patch a queue in place:

lakeshore queues patch training-h100 --max-depth 1000 --on-full reject
lakeshore queues patch training-h100 --description "h100 production"

CLI surface

The queues group uses two irregular verbs — ls (not list) and rm (not remove). Everything else in the CLI uses the regular forms.

Note that patch cannot change --kind or --elasticity <kind>. To change the elasticity policy itself, PATCH the queue's elasticity object over HTTP, or recreate the queue.

Shared pools

Multiple queues can share one worker fleet by carrying the same pool name inside their elasticity object. Workers in the pool can claim work from any member queue, and the controller scales the pool as a single unit.

pool, match, steal_delay_s, and cost_weight have no CLI flags — set them by PATCHing the queue's elasticity object directly:

curl -X PATCH \
  -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"elasticity":{"kind":"fully_elastic","max":8,"pool":"mixed"}}' \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/$NS/queues/cpu-prep

See Elasticity — shared pools for the semantics.

HTTP

POST   /v1/namespaces/:ns/queues
GET    /v1/namespaces/:ns/queues[?state=…]
GET    /v1/namespaces/:ns/queues/:name
PATCH  /v1/namespaces/:ns/queues/:name
POST   /v1/namespaces/:ns/queues/:name/archive
POST   /v1/namespaces/:ns/queues/:name/unarchive
POST   /v1/namespaces/:ns/queues/:name/drain
DELETE /v1/namespaces/:ns/queues/:name[?force=true]
GET    /v1/namespaces/:ns/queues/:name/events[?verbose=true&limit=…]
GET    /v1/namespaces/:ns/queues/:name/stats
POST   /v1/namespaces/:ns/workers/:id/queues

An unknown namespace on any of these is a 404 — the queue routes do not auto-create namespaces the way the secrets / providers / modes routes do.

Read next

• Elasticity — the four scaling policies you can declare on a queue.
• Scaling rules — the rule book the controller follows on every tick.
• Jobs — inspecting the invocations queued here.
• Compose — declarative queues + daemons in lakeshore.yaml.
• Python SDK — the producer side; @udf(queue="…") lands on the queues you create here.
• /dev/queues — the developer-side design note.