# Queues

A **Queue** is the unified scheduling primitive in Lakeshore. It is a
named pipe of pending work; daemons subscribe to it by name; the control
plane hands each queued Invocation to whichever subscribed daemon claims
it first.

There is exactly one primitive at the scheduling layer. Workers carry a
`queues: string[]` array recording which queues they pull from, and the
**empty array means the default queue**. Everything else — elasticity,
admission, work-stealing — is a policy field on the queue row, not a
second primitive.

```
       producer                control plane                  workers
       ────────                ─────────────                  ───────

   submit(invocation)
   queue=training-h100  ───▶   ┌──────────────┐
                               │   training-  │  ◀── poll ──  daemon-A (queues=["training-h100"])
                               │     h100     │
                               │              │  ◀── poll ──  daemon-B (queues=["training-h100","eval"])
                               └──────────────┘
                               ┌──────────────┐
                               │     eval     │  ◀── poll ──  daemon-B
                               └──────────────┘
                               ┌──────────────┐
                               │   default    │  ◀── poll ──  daemon-C (queues=[])
                               └──────────────┘
```

Queue names must match `^[a-z0-9][a-z0-9-]*$`.

## What is on a queue row

| Field | What it holds |
| --- | --- |
| `kind` | Scheduling kind — `fifo` (default), `priority`, `boltzmann`, `filo`. |
| `kindParams` | Kind-specific knobs, e.g. `{ temperature: 0.5 }` or `{ priority_field: "p" }`. |
| `elasticity` | The autoscaling spec. See [Elasticity](/get-started/elasticity.md). |
| `admission` | `{ max_depth?, on_full, rate_limit?, deadline_cutoff_s? }` where `on_full` is `reject` or `block`. |
| `providerRef` | The Provider the elasticity controller launches new daemons under. |
| `daemonTemplate` | `{ instance_type?, image_id?, runner? }` used when the controller launches. |
| `state` | `active`, `draining`, or `archived`. |

> **Warning:** All four `kind` values are accepted and stored, but the dispatcher's
> claim always sorts `{ priority: -1, submittedAt: 1 }` regardless of the
> queue's kind. `boltzmann` and `filo` do not change dispatch order today,
> and `priority` is really "priority field on the invocation, FIFO within
> a tier" — it is not derived from `kindParams`. Likewise `admission` is
> validated and stored but **not enforced** at admit time.

## Membership

A daemon's queue membership lives on the **worker** row, not the queue
row — `Worker.queues: string[]` is the source of truth. `queues show`
walks the workers and reports who is pulling from a queue right now.

**Default-queue membership is the empty array.** A daemon launched
without `--queue` lands in the default queue. There is no sentinel
string written into `Worker.queues` by the system. (The membership
lookup does also accept a literal `"default"` entry if you put one
there, but the explicit add/remove routes reject `default` — the empty
array is the contract.)

## Lifecycle

```bash
lakeshore queues drain training-h100      # stop admitting; finish in-flight
lakeshore queues archive training-h100    # soft-delete, keep the row
lakeshore queues unarchive training-h100  # back to active (reverses both)
lakeshore queues rm training-h100         # hard delete
```

`rm` refuses with a 409 while the queue has members; `--force` cascades
by scrubbing the name out of every `Worker.queues` array first.

The schema also declares a `paused` state, but no route can set it —
`active`, `draining`, and `archived` are the three reachable values.

## The default queue

You do not have to create it. The control plane materialises a `default`
row on the first `queues ls` or `queues show default` in a namespace,
with `kind: "fifo"` and the description "Default queue (auto-created).
Cannot be archived or deleted." It is a reserved name: `archive` and
`rm` both return 409.

```bash
lakeshore daemon launch --provider my-ec2 --name scratch --wait
lakeshore daemon show scratch     # queues: []  — i.e. the default queue
```

## Organization visibility

When namespaces are grouped by a shared `orgId`, `queues ls` aggregates
queues from every namespace in the org and stamps a `namespace` field on
each row so you can tell who owns it. Two namespaces in one org may hold
same-named queues; they are distinct rows scoped to their own namespace.

A token scoped to one namespace in the org also authorises requests
against sibling namespaces — put the sibling slug in the URL path:

```bash
curl -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/bob/queues/inference
```

> **Note:** `--dreamrc <path>` is the CLI's only program-level option. The namespace
> the CLI targets comes from `LAKESHORE_NAMESPACE` (when `LAKESHORE_URL`
> is set) or from `~/.config/lakeshore/auth.yml`. To reach a sibling
> namespace, either re-login against it or call the HTTP route directly.

## A worked example

Spin up two queues — one for CPU pre-processing, one for GPU training —
and a few daemons for each.

```bash
# Create the queues. Both fixed for now.
lakeshore queues add cpu-prep      --kind fifo     --elasticity fixed
lakeshore queues add training-h100 --kind priority --elasticity fixed

# Launch daemons into each. --queue is repeatable.
lakeshore daemon launch --provider my-ec2 --queue cpu-prep      --count 4 --name prep  --wait
lakeshore daemon launch --provider my-gpu --queue training-h100 --count 2 --name train --wait

# What's where?
lakeshore queues ls
lakeshore queues ls --state active
lakeshore daemon list
lakeshore queues show training-h100    # members + full queue row
lakeshore queues stats training-h100   # daemon_count / utilization / pending
```

Edit membership on a running daemon — no relaunch needed. The daemon
picks the change up on its next poll:

```bash
lakeshore daemon update train-00 --add-queue eval
lakeshore daemon update train-00 --remove-queue eval
```

Patch a queue in place:

```bash
lakeshore queues patch training-h100 --max-depth 1000 --on-full reject
lakeshore queues patch training-h100 --description "h100 production"
```

## CLI surface

The `queues` group uses two irregular verbs — `ls` (not `list`) and `rm`
(not `remove`). Everything else in the CLI uses the regular forms.

| Command | Notable flags |
| --- | --- |
| `queues add <name>` | `--description`, `--kind`, `--provider`, `--elasticity`, `--min`, `--max`, `--threshold`, `--instance-type`, `--image-id`, `--runner`, `--max-depth`, `--on-full`, `--json` |
| `queues ls` | `--state <active\|draining\|paused\|archived>`, `--json` |
| `queues show <name>` | `--json` |
| `queues patch <name>` | `--description`, `--provider` (pass `""` to clear), `--min`, `--max`, `--threshold`, `--max-depth`, `--on-full`, `--json` |
| `queues archive` / `unarchive` / `drain` `<name>` | `--json` |
| `queues rm <name>` | `--force` |
| `queues events <name>` | `--verbose`, `--limit <n>` (default 50, max 200), `--json` |
| `queues stats <name>` | `--json` |

Note that `patch` cannot change `--kind` or `--elasticity <kind>`. To
change the elasticity policy itself, PATCH the queue's `elasticity`
object over HTTP, or recreate the queue.

## Shared pools

Multiple queues can share one worker fleet by carrying the same `pool`
name inside their `elasticity` object. Workers in the pool can claim
work from any member queue, and the controller scales the pool as a
single unit.

`pool`, `match`, `steal_delay_s`, and `cost_weight` have **no CLI
flags** — set them by PATCHing the queue's `elasticity` object directly:

```bash
curl -X PATCH \
  -H "Authorization: Bearer $LAKESHORE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"elasticity":{"kind":"fully_elastic","max":8,"pool":"mixed"}}' \
  https://api.lakeshore.dreamlake.ai/v1/namespaces/$NS/queues/cpu-prep
```

See [Elasticity — shared pools](/get-started/elasticity.md#shared-pools)
for the semantics.

## HTTP

```
POST   /v1/namespaces/:ns/queues
GET    /v1/namespaces/:ns/queues[?state=…]
GET    /v1/namespaces/:ns/queues/:name
PATCH  /v1/namespaces/:ns/queues/:name
POST   /v1/namespaces/:ns/queues/:name/archive
POST   /v1/namespaces/:ns/queues/:name/unarchive
POST   /v1/namespaces/:ns/queues/:name/drain
DELETE /v1/namespaces/:ns/queues/:name[?force=true]
GET    /v1/namespaces/:ns/queues/:name/events[?verbose=true&limit=…]
GET    /v1/namespaces/:ns/queues/:name/stats
POST   /v1/namespaces/:ns/workers/:id/queues
```

An unknown namespace on any of these is a 404 — the queue routes do not
auto-create namespaces the way the secrets / providers / modes routes do.

## Read next

- [Elasticity](/get-started/elasticity.md) — the four scaling policies you
  can declare on a queue.
- [Scaling rules](/get-started/scaling-rules.md) — the rule book the
  controller follows on every tick.
- [Jobs](/get-started/jobs.md) — inspecting the invocations queued here.
- [Compose](/get-started/compose.md) — declarative queues + daemons in
  `lakeshore.yaml`.
- [Python SDK](/python-sdk.md) — the producer side; `@udf(queue="…")` lands
  on the queues you create here.
- [`/dev/queues`](/dev/queues) — the developer-side design note.