# CLI examples — Queues and jobs

Part of the [CLI examples cookbook](/cli/examples.md). Auth setup lives
on [Auth + setup](/cli/examples/auth.md).

## Queues

A **queue** is the unified scheduling primitive: a named pipe of
pending Invocations that daemons subscribe to by name. Workers register
themselves into queues via `Worker.queues: string[]`; the empty array
means the **default queue** (there is no sentinel name). See
[Queues](/get-started/queues.md) and [`/dev/queues`](/dev/queues) for the
concept and design pages.

> **Warning:** It is `lakeshore queues ls` and `lakeshore queues rm` — **not** `list`
> and `remove`. Every other group in the CLI uses `list` / `remove`; this
> one doesn't.

The full verb set is `add`, `ls`, `show`, `patch`, `archive`,
`unarchive`, `drain`, `rm`, `events`, `stats`.

### `queues add` — create a queue

The bare minimum is a name (matching `^[a-z0-9][a-z0-9-]*$`).
`--kind` is one of `fifo` (default), `priority`, `boltzmann`, `filo`.

```bash
lakeshore queues add training-h100
lakeshore queues add eval --kind priority --description "evaluation, urgent first"
```

### `queues add` — with an elasticity declaration

`--elasticity` accepts `fixed`, `fully-elastic`, `pool-with-threshold`,
or `max-count`, with `--min`, `--max`, and `--threshold` (0.0–1.0)
supplying its parameters. `--instance-type`, `--image-id`, and
`--runner` seed the daemon template used when the queue scales up.
`--max-depth` plus `--on-full` (`reject` or `block`) bound the backlog.

```bash
lakeshore queues add training-h100 \
  --kind priority \
  --provider my-gpu-provider \
  --elasticity pool-with-threshold \
  --min 2 --max 16 --threshold 0.7 \
  --instance-type p4d.24xlarge \
  --max-depth 1000 --on-full reject
```

> **Warning:** The `--elasticity` flag takes **hyphenated** values
> (`fully-elastic`, `pool-with-threshold`, `max-count`), while the
> `elasticity.kind` field in `lakeshore.yaml` accepts only the
> **underscored** spellings (`fully_elastic`, `pool_with_threshold`,
> `max_count`). They are not interchangeable.

### `queues ls` — list queues

`--state` filters on `active`, `draining`, `paused`, or `archived`.

```bash
lakeshore queues ls
lakeshore queues ls --state active
lakeshore queues ls --json | jq -r '.[].name'
```

The table shows `NAME / KIND / STATE / MEMBERS / PROVIDER /
ELASTICITY / DESCRIPTION`. `MEMBERS` is the count of workers whose
`queues` array contains this queue's name.

### `queues show` — one queue and its members

```bash
lakeshore queues show training-h100
lakeshore queues show training-h100 --json
```

### `queues patch` — mutate a queue in place

Accepts `--description`, `--provider`, `--min`, `--max`,
`--threshold`, `--max-depth`, `--on-full`, and `--json`. Note it does
**not** accept `--kind` or `--elasticity`.

```bash
lakeshore queues patch training-h100 --description "h100 prod"
lakeshore queues patch training-h100 --max 32 --threshold 0.6
lakeshore queues patch training-h100 --max-depth 2000 --on-full block
lakeshore queues patch training-h100 --provider ""    # clear the provider
```

### `queues drain` / `archive` / `unarchive`

`drain` stops admitting new work but lets in-flight invocations finish.
`archive` soft-deletes — the row survives but stops accepting work.
`unarchive` re-activates a drained or archived queue. All three take a
name and `--json`.

```bash
lakeshore queues drain training-h100
lakeshore queues archive eval
lakeshore queues unarchive eval
```

### `queues rm` — delete

Errors if any worker is still bound to the queue. `--force` cascades —
it scrubs the queue name out of every `Worker.queues` array first.

```bash
lakeshore queues rm eval
lakeshore queues rm eval --force
```

### `queues events` — the elasticity audit trail

One row per controller decision. `tick_noop` rows are hidden unless you
pass `--verbose`, which is the quickest way to check whether the
controller is running at all. `--limit` defaults to 50, capped at 200.

```bash
lakeshore queues events training-h100
lakeshore queues events training-h100 --verbose --limit 200
lakeshore queues events training-h100 --json
```

### `queues stats` — current signals

A point-in-time view of what the controller is looking at: daemon
count, utilization, pending invocations, and the last few non-noop
scale actions.

```bash
lakeshore queues stats training-h100
lakeshore queues stats training-h100 --json
```

## Jobs

Invocations tracked by the control plane — the things `@dls.udf` calls
queue up.

### `jobs list` — recent invocations

`--limit` is 1–200 (default 50) and `--offset` defaults to 0.

```bash
lakeshore jobs list
lakeshore jobs list --limit 20 --offset 20
```

### `jobs list` — filters

`--state` is repeatable and accepts `queued`, `running`, `succeeded`,
`failed`, `killed`, `timeout`. `--worker` and `--queue` take ids.

```bash
lakeshore jobs list --state running --state queued
lakeshore jobs list --worker "$D"
lakeshore jobs list --json | jq '.[] | {id, state, worker}'
```

### `jobs show` — one invocation

```bash
JOB=$(lakeshore jobs list --json | jq -r '.[0].id // empty')
lakeshore jobs show "$JOB"
```

### `jobs kill` — cancel one invocation

Queued jobs are marked `killed` immediately. **Running jobs return
409** — to stop in-flight work, kill the process on the daemon
(`lakeshore daemon exec`) or drop the daemon itself.

```bash
lakeshore jobs kill "$JOB"
```

### `jobs bulk-cancel` — cancel everything queued

Only touches rows in state `queued`; running work is untouched.

```bash
lakeshore jobs bulk-cancel
lakeshore jobs bulk-cancel --queue training-h100
lakeshore jobs bulk-cancel --json
```

## Read next

- [Queues](/get-started/queues.md) — the concept.
- [Elasticity](/get-started/elasticity.md) · [Scaling rules](/get-started/scaling-rules.md)
- [Daemons + exec/run](/cli/examples/daemons-and-exec.md) — binding workers to queues.