DreamLake

Branch feat/native-wire-udf · rewrite of the lakeshore Python SDK from first principles. msgpack (the library) and the Storage broker survive; everything above them is new. No zaku code — its lessons are encoded as conformance tests.

Progress

PhaseWhatState
0wire.py — ExtType codec, versioned Envelope, Frame + reader✅
1context.py — dls.run seam, string-key returns, key contract, lazy root✅
2dispatch.py minimal plane ✅ · functions.py native @udf ✅ · Queue/SyncQueue ✅ · httpx transport ✅ · worker loop ✅ · legacy deletion ✅✅
3Real streaming (Node CP Redis-Streams journal) · e2e + examples rework⬜
4dreamlake collections merge + dispatch-plane split⬜

Tests: 113 passed (SDK unit; legacy tests deleted with the legacy surface) · control plane baseline 599 ✅ · dreamlake-server 108/109 (1 pre-existing failure, documented).

The locked design, in one breath

Zero new value types — data UDFs return string keys; file bytes never ride the wire (worker↔storage side channel via dls.run.read/write); the function's own shape picks execution (sync/async/generator/async-gen) and transfer timing (return = upload-then-return, async-gen = stream partial manifests); no Future — the durable handle is the invocation id; Queue async-first with SyncQueue sibling; local tier imports nothing distributed; every dispatch transition is one guarded atomic UPDATE (fencing, idempotent submit, counts==contents by construction).

Documents

  • Native API + type system design — the synthesized design, dual-ergonomics audit, and concept-by-concept zaku comparison. (Historical note: its Ref recommendation was superseded by the zero-types / string-keys decision recorded here and in the API pages.)
  • Usage surface & test plan — 113 CP routes × 7 consumers × 3 encodings, and the automatable test strategy.

New modules (API)

  • [wire] — codec, envelope, frames
  • [context] — the dls.run I/O seam
  • [dispatch] / [http_dispatch] — the dispatch planes (API page)
  • [functions] — the native @udf (API page)
  • [queues] — Queue / SyncQueue / Topic (API page)
  • [worker] — the claim-and-execute loop

The user-facing Python SDK section now documents this native surface.