Jobs

Every dispatched function call lands as an Invocation row in the control plane. lakeshore jobs … is the surface for seeing what is queued, what is running, and for cancelling what you no longer want.

Every verb takes --json, so the output pipes into jq or into other tooling; the default is a human-readable table.

The states

queued ──▶ running ──▶ succeeded
                   ├─▶ failed
                   ├─▶ killed
                   └─▶ timeout

succeeded, failed, killed, and timeout are the terminal set. The id is a ULID generated by the producer — the same string that crosses the wire, not a server-assigned ObjectId.

CLI

bash

lakeshore jobs list                                  # latest 50, all states
lakeshore jobs list --state queued                   # only queued
lakeshore jobs list --state queued --state running   # union
lakeshore jobs list --worker <worker-id> --limit 10
lakeshore jobs list --json | jq '.invocations[].id'

lakeshore jobs show <id>
lakeshore jobs show <id> --json

lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue training-h100

Filters on jobs list, all optional:

Flag	Meaning
`--state`	One of `queued`, `running`, `succeeded`, `failed`, `killed`, `timeout`. Repeatable; multiple values become an `in` filter.
`--worker`	Only invocations claimed by this worker id.
`--queue`	Only invocations on this queue, matched against `runConfig.queue`.
`--limit`	1–200, default 50.
`--offset`	Rows to skip, default 0.

Rows come back newest-submitted first.

Cancelling

The kill verb is deliberately narrow:

queued → the row flips to state=killed with finishedAt=now. Safe: no daemon has claimed it, and the dispatch claim filters on state="queued", so future polls skip it.
running → 409, with the claiming daemon's id in the response body. There is no daemon-side cancel signal in the wire protocol yet. The escape hatch is to stop the daemon: SIGTERM fires the per-invocation cancellation tokens the drain loop already plumbs through, and the in-flight invocations get acked as killed once shutdown.grace_s expires.
Already terminal → 409 with the row's current state.

jobs bulk-cancel applies the same narrow rule in bulk: it flips every state="queued" row to killed and returns { "cancelled": <n> }. Running invocations are untouched. Scope it with --queue <name>.

Per-invocation cancel for running work is on the follow-up list.

HTTP

The CLI is a thin wrapper over JSON endpoints — call them directly when the CLI shape does not fit:

GET    /v1/namespaces/:ns/invocations[?state=…&worker=…&queue=…&limit=…&offset=…]
GET    /v1/namespaces/:ns/invocations/:id
DELETE /v1/namespaces/:ns/invocations/:id
POST   /v1/namespaces/:ns/invocations/bulk-cancel[?queue=…]

The list endpoint returns { invocations: [...], limit, offset, count }. show returns one invocation. DELETE returns the updated row on success, or a 4xx body explaining why the kill was refused. An unknown namespace is a 404 on all four.

Operators with an admin token also get a long-polling firehose:

GET /v1/admin/invocations[?cursor=…&timeout_s=…&limit=…]
GET /v1/admin/invocations/:id

timeout_s caps at 60 and limit defaults to 100 (max 500).

The row shape

Field	Meaning
`id`	Producer-generated ULID.
`state`	One of the six states above.
`workerId`	Claiming worker, `null` until claimed.
`queueName`	Read out of `runConfig.queue`. There is no `queueId` column — Phase A.5 removed the FK.
`functionId`	The `Function` row this invocation calls.
`priority`, `attempt`	Dispatch priority and retry count.
`runConfig`	The full submitted run config, including `queue`.
`progress`, `error`	Whatever the daemon last reported.
`submittedAt` / `startedAt` / `finishedAt`	ISO timestamps; the last two are nullable.
`parentInvocationId`, `rootInvocationId`, `correlationId`	Pipeline lineage. `rootInvocationId` equals the row's own id when it is the root.

jobs list renders a subset — id, state, worker, queue, attempt, and relative ages for submitted/started. jobs show prints the whole record, and --json gives it verbatim on both.

Streaming a result

Results that arrive as a journal rather than one blob are read from the producer surface, not from jobs:

GET /v1/producer/invocations/:id/result/stream?since=<seq>

It replays the durable ResultChunk rows past since, then follows the live Redis journal. The response is raw chunked binary with content-type application/x-msgpack-stream — self-delimiting msgpack frames with no SSE wrapper and no delimiter between them. From Python, use SyncQueue.stream(inv) / Queue.stream(inv) instead of parsing it by hand.

Tab-completion

Once shell completion is installed (see Completion):

lakeshore jobs <TAB>                  → list show kill
lakeshore jobs list --state <TAB>     → queued running succeeded failed killed timeout
lakeshore jobs show <TAB>             → live invocation ids (30 s cache, 50 max)

bulk-cancel is a real subcommand but is missing from the hand-maintained completion tree, so it does not tab-expand — type it in full.