Branch feat/native-wire-udf · rewrite of the lakeshore Python SDK from
first principles. msgpack (the library) and the Storage broker survive;
everything above them is new. No zaku code — its lessons are encoded as
conformance tests.
Progress
| Phase | What | State |
|---|---|---|
| 0 | wire.py — ExtType codec, versioned Envelope, Frame + reader | ✅ |
| 1 | context.py — dls.run seam, string-key returns, key contract, lazy root | ✅ |
| 2 | dispatch.py minimal plane ✅ · functions.py native @udf ✅ · Queue/SyncQueue ✅ · httpx transport ✅ · worker loop ✅ · legacy deletion ✅ | ✅ |
| 3 | Real streaming (Node CP Redis-Streams journal) · e2e + examples rework | ⬜ |
| 4 | dreamlake collections merge + dispatch-plane split | ⬜ |
Tests: 113 passed (SDK unit; legacy tests deleted with the legacy surface) · control plane baseline 599 ✅ · dreamlake-server 108/109 (1 pre-existing failure, documented).
The locked design, in one breath
Zero new value types — data UDFs return string keys; file bytes never
ride the wire (worker↔storage side channel via dls.run.read/write); the
function's own shape picks execution (sync/async/generator/async-gen) and
transfer timing (return = upload-then-return, async-gen = stream partial
manifests); no Future — the durable handle is the invocation id;
Queue async-first with SyncQueue sibling; local tier imports nothing
distributed; every dispatch transition is one guarded atomic UPDATE
(fencing, idempotent submit, counts==contents by construction).
Documents
- Native API + type system design — the synthesized
design, dual-ergonomics audit, and concept-by-concept zaku comparison.
(Historical note: its
Refrecommendation was superseded by the zero-types / string-keys decision recorded here and in the API pages.) - Usage surface & test plan — 113 CP routes × 7 consumers × 3 encodings, and the automatable test strategy.
New modules (API)
- [
wire] — codec, envelope, frames - [
context] — thedls.runI/O seam - [
dispatch] / [http_dispatch] — the dispatch planes (API page) - [
functions] — the native@udf(API page) - [
queues] — Queue / SyncQueue / Topic (API page) - [
worker] — the claim-and-execute loop
The user-facing Python SDK section now documents this native surface.