# Jobs

Every dispatched function call lands as an **Invocation** row in the
control plane. `lakeshore jobs …` is the surface for seeing what is
queued, what is running, and for cancelling what you no longer want.

Every verb takes `--json`, so the output pipes into `jq` or into other
tooling; the default is a human-readable table.

## The states

```
queued ──▶ running ──▶ succeeded
                   ├─▶ failed
                   ├─▶ killed
                   └─▶ timeout
```

`succeeded`, `failed`, `killed`, and `timeout` are the terminal set.
The id is a **ULID generated by the producer** — the same string that
crosses the wire, not a server-assigned ObjectId.

## CLI

```bash
lakeshore jobs list                                  # latest 50, all states
lakeshore jobs list --state queued                   # only queued
lakeshore jobs list --state queued --state running   # union
lakeshore jobs list --worker <worker-id> --limit 10
lakeshore jobs list --json | jq '.invocations[].id'

lakeshore jobs show <id>
lakeshore jobs show <id> --json

lakeshore jobs kill <id>
lakeshore jobs bulk-cancel --queue training-h100
```

Filters on `jobs list`, all optional:

| Flag | Meaning |
| --- | --- |
| `--state` | One of `queued`, `running`, `succeeded`, `failed`, `killed`, `timeout`. Repeatable; multiple values become an `in` filter. |
| `--worker` | Only invocations claimed by this worker id. |
| `--queue` | Only invocations on this queue, matched against `runConfig.queue`. |
| `--limit` | 1–200, default 50. |
| `--offset` | Rows to skip, default 0. |

Rows come back newest-submitted first.

## Cancelling

The kill verb is deliberately narrow:

- **`queued`** → the row flips to `state=killed` with `finishedAt=now`.
  Safe: no daemon has claimed it, and the dispatch claim filters on
  `state="queued"`, so future polls skip it.
- **`running`** → **409**, with the claiming daemon's id in the response
  body. There is no daemon-side cancel signal in the wire protocol yet.
  The escape hatch is to stop the daemon: `SIGTERM` fires the
  per-invocation cancellation tokens the drain loop already plumbs
  through, and the in-flight invocations get acked as `killed` once
  `shutdown.grace_s` expires.
- **Already terminal** → 409 with the row's current state.

`jobs bulk-cancel` applies the same narrow rule in bulk: it flips every
`state="queued"` row to `killed` and returns `{ "cancelled": <n> }`.
Running invocations are untouched. Scope it with `--queue <name>`.

Per-invocation cancel for running work is on the follow-up list.

## HTTP

The CLI is a thin wrapper over JSON endpoints — call them directly when
the CLI shape does not fit:

```
GET    /v1/namespaces/:ns/invocations[?state=…&worker=…&queue=…&limit=…&offset=…]
GET    /v1/namespaces/:ns/invocations/:id
DELETE /v1/namespaces/:ns/invocations/:id
POST   /v1/namespaces/:ns/invocations/bulk-cancel[?queue=…]
```

The list endpoint returns `{ invocations: [...], limit, offset, count }`.
`show` returns one invocation. `DELETE` returns the updated row on
success, or a 4xx body explaining why the kill was refused. An unknown
namespace is a 404 on all four.

Operators with an admin token also get a long-polling firehose:

```
GET /v1/admin/invocations[?cursor=…&timeout_s=…&limit=…]
GET /v1/admin/invocations/:id
```

`timeout_s` caps at 60 and `limit` defaults to 100 (max 500).

## The row shape

| Field | Meaning |
| --- | --- |
| `id` | Producer-generated ULID. |
| `state` | One of the six states above. |
| `workerId` | Claiming worker, `null` until claimed. |
| `queueName` | Read out of `runConfig.queue`. There is no `queueId` column — Phase A.5 removed the FK. |
| `functionId` | The `Function` row this invocation calls. |
| `priority`, `attempt` | Dispatch priority and retry count. |
| `runConfig` | The full submitted run config, including `queue`. |
| `progress`, `error` | Whatever the daemon last reported. |
| `submittedAt` / `startedAt` / `finishedAt` | ISO timestamps; the last two are nullable. |
| `parentInvocationId`, `rootInvocationId`, `correlationId` | Pipeline lineage. `rootInvocationId` equals the row's own id when it is the root. |

`jobs list` renders a subset — id, state, worker, queue, attempt, and
relative ages for submitted/started. `jobs show` prints the whole record,
and `--json` gives it verbatim on both.

## Streaming a result

Results that arrive as a journal rather than one blob are read from the
producer surface, not from `jobs`:

```
GET /v1/producer/invocations/:id/result/stream?since=<seq>
```

It replays the durable `ResultChunk` rows past `since`, then follows the
live Redis journal. The response is raw chunked binary with content-type
`application/x-msgpack-stream` — self-delimiting msgpack frames with no
SSE wrapper and no delimiter between them. From Python, use
`SyncQueue.stream(inv)` / `Queue.stream(inv)` instead of parsing it by
hand.

## Tab-completion

Once shell completion is installed (see [Completion](/cli/completion.md)):

```text
lakeshore jobs                   → list show kill
lakeshore jobs list --state      → queued running succeeded failed killed timeout
lakeshore jobs show              → live invocation ids (30 s cache, 50 max)
```

`bulk-cancel` is a real subcommand but is missing from the hand-maintained
completion tree, so it does not tab-expand — type it in full.

## Read next

- [Queues](/get-started/queues.md) — where these invocations are waiting.
- [Invocation ids](/python-sdk/invocations.md) — the Python side of the same
  handle.
- [Architecture](/get-started/architecture.md) — the Invocation model in
  context.