Status: plan of record for the native rewrite (2026-07-16). Visual checks on the
dreamlake frontend are **explicitly out of scope** here (manual + design work);
this plan covers everything automatable.

## 1. The shape of the problem

The control plane serves **113 routes**; seven consumers sit on them with three
different body encodings. The rewrite replaces the msgpack dispatch wire and
the SDK surface — so the blast radius is exactly: *who parses which bytes*.

```
                 ┌────────────── JSON (field-name-sensitive) ──────────────┐
   TS CLI ───────┤                                                          │
   lakeshore-web ┤   resource CRUD routes (~90): providers/queues/storages/ │
   dreamlake-ai ─┤   workers/modes/secrets/... — UNCHANGED by the rewrite   │
    (via proxy)  └──────────────────────────────────────────────────────────┘
                 ┌────────────── msgpack (wire-rewrite blast zone) ─────────┐
   Python SDK ───┤   producer/submit|await, daemon/poll|ack|event,          │
   nymph (Rust) ─┤   payloads/presign + (new) result journal routes         │
                 └──────────────────────────────────────────────────────────┘
```

## 2. Consumer → surface matrix

| Consumer | Lang / encoding | Surface | CP routes touched | Rewrite risk |
|---|---|---|---|---|
| **Python SDK + daemon** (`dreamlake-lakeshore`) | msgpack on `_post`, JSON on GETs | `@dls.udf`, `Queue`, `gather`, `Storage`, daemon loop | `producer/submit\|await`, `producer/queues`, `daemon/poll\|ack\|event`, ns `queues` CRUD, `storages/:name/presign\|credentials`, `code-repos*`, `workers` | **Total** — this is the thing being rewritten |
| **nymph** (Rust daemon) | msgpack (`rmp_serde`); `/chunk` raw octet-stream | worker runtime: hello/poll/ack/exec-result/presign | `daemon/hello\|poll\|ack`, `daemon/exec/:id/result\|chunk`, `payloads/presign` | **Highest external risk** — field-name/type-sensitive msgpack structs; must stay compatible or be updated in lockstep (Phase 3) |
| **TS CLI + Node SDK** (`lakeshore`) | JSON only | ~20 command groups (`providers`, `queues`, `daemon`, `jobs`, `storage`, `exec`, `compose`, `admin`, …) | resource CRUD + `workers/*` + `invocations` list/show/kill + `exec` + `nymph-releases` | Low — JSON field renames only |
| **lakeshore-web** *(removed — the dashboard was dropped from the workspace; row kept for historical context)* | JSON, strongly typed | dashboard reads + worker ops | `admin/invocations\|pipelines\|providers`, ns `queues\|storages\|workers` (+hibernate/patch/delete) | n/a — no longer a consumer |
| **dreamlake-server** | JSON; proxy is verbatim *except it re-serializes mutating bodies as JSON* | `Lakeshore` connection CRUD + `proxy/*` passthrough; probe = `GET ns queues` / `healthz` | none with knowledge (passthrough) | Low; **caveat: proxy cannot carry msgpack request bodies** — fine while its only clients speak JSON |
| **dreamlake-ai** | JSON via proxy | lakeshore UI: invocations/queues/workers/modes/storages/mounts/secrets/tunnels/providers panels | ~30 ns routes + `admin/providers` through `proxy/` | Low — field renames only |
| **lakeshore-examples** | via Python SDK | `@dls.udf`, `Queue`, `gather`, `as_completed`, `Storage`, thunks | (through SDK) | Total — must be migrated with the SDK |

**Orphan routes** (no consumer found; candidates to drop or intentionally reserve):
`GET /readyz`, `PATCH admin/namespaces/:slug`, daemon key-rotation (`daemon/keys`,
`workers/:id/keys*`), `enroll-tokens*` (server-internal), the namespaced
`workers/:id/exec/:eid/result` alias, `jobs/:id/result` (both verbs),
`exec/:id/log[/stream]`.

## 3. What the rewrite changes vs preserves

**Preserved (no consumer action):** all JSON resource-CRUD routes; the
dreamlake proxy model (until Phase 4); `healthz`; exec/log paths.

**Changed:** the msgpack dispatch wire (`Envelope` replaces the ad-hoc
submit/poll/ack body shapes; unified `wire.pack` result path), SDK public
surface (`Queue`/`SyncQueue`, no `QueueFuture`, string-key data returns),
**new** journal routes (result-chunk append + cursor-read stream). During
Phase 2/3 the Python daemon and CP move in lockstep; **nymph poll/ack
compatibility is a named gate** — either the CP keeps the legacy poll/ack body
shape for nymph's queues, or nymph's `protocol.rs` structs update in the same
PR (decision at Phase 3 start, golden fixtures either way).

## 4. Test plan (automatable only)

### A. Unit — SDK (runs on every commit, no network)
- `wire.py`: codec round-trips (ndarray/tensor/image/tuple/nesting), unknown-type
  raise, forward-compat ext codes, Envelope versioning, Frame reassembly across
  arbitrary chunk splits. ✅ landed (`test_wire.py`, 19 tests)
- `context.py`: local read/write/scope, shape-mirroring, key escaping,
  return-your-keys contract (all violation classes), **local-simple invariant**
  (fresh-interpreter import check: no httpx/msgpack on local paths). ✅ landed
  (`test_context.py`, 17 tests)
- Phase 2 additions: `@udf` dispatch matrix (sync/async/gen/asyncgen ×
  local/remote), `Queue`/`SyncQueue` protocol core against a mock transport,
  idempotency-key semantics, `check_returned_keys` for async-gen accumulation.

### B. Wire-contract conformance suite (the centerpiece)
One parametrized pytest suite, run against **every dispatch-plane backend**:
1. the minimal Python CP (in-process; every commit), and
2. the Node CP (full-stack harness via `LAKESHORE_CP_DIR`; pre-merge + CI).

Coverage — each item is a named test, not an aspiration:
- submit → claim → ack lifecycle; nack + retry; attempt increments
- **atomic claim**: N concurrent claimers, one job → exactly one winner
- **fencing**: a stale worker (job re-dispatched after unstale) cannot ack
- **idempotent submit**: same key twice → same invocation id, no dup job
- ordering: priority desc / FILO / Boltzmann-sampling distribution (χ² over
  many claims), all implemented as sorts/samples over the primary store
- **counts == contents** (zaku-desync regression): counts derived from the same
  rows that claim; assert after every lifecycle transition, incl. crash-mid-claim
- journal: N frames in-order, `seq` cursor reconnect resumes without loss/dup,
  explicit `END` vs dropped-connection distinguishable, `ERROR` reconstructs the
  exception, CHUNK runs reassemble, MAXLEN/TTL trim degrades to CP re-fetch
- streaming UDF: async-gen yields → one frame per yield → consumer iterates
  live; partial-key-list manifests concatenate; contract violation surfaces

### C. Integration / e2e
- Existing self-provisioning full-stack e2e (Mongo + Node CP + Python daemon)
  updated to the new wire; runs pre-merge (`E2E_REQUIRE_FULLSTACK=1` in CI).
- CP's own suite stays green — baseline **599 pass / 0 fail** recorded 2026-07-16.
- **nymph compat**: golden msgpack fixtures for poll/ack request/response bodies,
  checked into both repos; a fixture diff = a deliberate, reviewed wire change.

### D. Cross-consumer contract snapshots (catches field renames)
- Golden JSON snapshots for every response body the TS CLI, lakeshore-web, and
  dreamlake-ai parse (queues/workers/storages/invocations/providers lists +
  details). Generated against the Node CP, replayed against the minimal CP.
- dreamlake-server proxy suite stays green — baseline **108/109** (1 failure
  pre-existing on main: `pipelines.test.ts` PATCH-artifacts, documented
  2026-07-16, unrelated to lakeshore).

### E. Consumer pipelines (this repo = the real customer)
- Local golden run: stub-UDF `generate_scene` + `annotate_subtasks` under
  `dls.scope`, asserting the exact `source/ · process/ · outputs/{i:04d}/` tree
  and manifest keys. Runs with zero network (the local fast path IS the test).
- One remote smoke per release: `splats_to_mesh`-shaped UDF through the minimal
  CP end-to-end (submit → daemon → storage side channel → keys back).

### F. Examples as tests
- `lakeshore-examples/test_runner.py`: the no-CP examples (02, 05) run in CI as
  plain scripts; queue/gather examples (08, 09, 10) run against the minimal CP.
  Examples are migrated with the SDK in Phase 2 and become its acceptance tests.

### G. Explicitly out of scope (manual / needs design)
- Visual checks on the dreamlake frontend (lakeshore panels, provider UI).
- TUI rendering (`lakeshore tui`).
- Cloud-provider launch paths (EC2/GCE/kube) — covered by existing CP tests only.

## 5. Gates per phase

| Phase | Merge gate |
|---|---|
| 0 wire ✅ | unit A green (landed: 61 pass) |
| 1 seam ✅ | unit A green + local-simple invariant (landed) |
| 2 @udf/queues + minimal CP | A + B(minimal CP) + E(local golden) + F(no-CP examples); CP suite untouched-green |
| 3 streaming + Node CP journal | B(both backends) + C incl. nymph fixtures + D snapshots |
| 4 dreamlake collections | D snapshots re-baselined + dreamlake-server suite green (108→109: fix the pre-existing failure while in there) |