TS CLI migration — Python CLI inventory and plan

TL;DR. All user-facing CLIs move to the TypeScript lakeshore CLI (npm @dreamlake/lakeshore), runnable under bun and node. The Python worker loop stays Python — it executes Python UDFs in-process — so the boundary is: TS owns argv/env/supervision, Python owns execution. Tranche 1 shipped lakeshore worker start|once, which spawns and supervises python -m dreamlake.lakeshore.daemon with full flag/env parity, plus bun runtime support for the whole CLI. The lakeshore-daemon console script was retired (0.3.6). Tranche 2 ported the dreamlake data-plane CLI into the same TypeScript CLI as the lakeshore dreamlake group — everything except artifact push, which stays Python behind a TS stub.

Status. Tranche 1 implemented (2026-07). Tranche 2 implemented (2026-07). Slug. ts-cli-migration Session. claude.ai/code

1. Inventory — every Python CLI entrypoint

1.1 lakeshore-daemon — the native worker (dreamlake-lakeshore)

[project.scripts] lakeshore-daemon = "dreamlake.lakeshore.daemon.__main__:main" (historical — the console script was removed from [project.scripts] in 0.3.6; only python -m dreamlake.lakeshore.daemon remains) in the dreamlake-lakeshore repo (the Python SDK). Declared once with params-proto; every flag doubles as an env var (flags win):

Consumers.

• tests/e2e/harness.py (the SDK's full-stack fixture) — spawns a fleet of sys.executable -m dreamlake.lakeshore.daemon --queue … --url … --namespace … --root … processes, env LAKESHORE_URL/NAMESPACE/ROOT set, tokens popped, terminated with SIGTERM → 8s → SIGKILL. Module spawn, not the console script — unaffected by console-script deprecation.
• tests/unit/test_code_mount.py — --help surface check + a --once drain of the local plane, both via python -m.
• This docs site (python-sdk/dispatch, happy-paths/05, 06) — user-facing lakeshore-daemon invocations. Update in tranche 1.5. Done (2026-07-24): the docs-page half of tranche 1.5 shipped — all reader-facing invocations now use lakeshore worker start|once.
• The TS CLI's lakeshore daemon start --local — spawned the module with a stale pre-native flag surface (--server/--tag); the new lakeshore worker group supersedes it for local workers.

Disposition: thin-TS-wrapper-spawning-python (shipped). The loop cannot leave Python (UDFs execute in-process). The CLI shell around it moved to lakeshore worker start|once; the console script printed a stderr pointer for one release and was removed from [project.scripts] in 0.3.6. python -m dreamlake.lakeshore.daemon stays supported indefinitely — it is the contract the TS CLI spawns.

1.2 Legacy surface B — lakeshore-udf-worker, udf-daemon

The lakeshore-py checkout inside this workspace is an older main of the same dreamlake-lakeshore repo and still shows the pre-native scripts: lakeshore-daemon (flags --server/--queue/--runner/ --worker_id/--log_level), lakeshore-udf-worker, and the udf-daemon alias. Upstream deleted these in the native rewrite.

Consumers (all pinned to the old flags): mprocs.yaml daemon panes (--server http://localhost:8080 --queue '*', autostart off), scripts/slurm-daemon.sbatch + docs/public/slurm-daemon.sbatch, docs/public/install-daemon.sh, example docstrings.

Disposition: stays deleted (superseded). No port. Refresh the lakeshore-py checkout, then repoint mprocs/sbatch at lakeshore worker start (tranche 1.5).

1.3 Not a CLI, but load-bearing: dreamlake.lakeshore.daemon.entry

The Rust nymph runners (process/docker/slurm/kube.rs) hard-code python3 -m dreamlake.lakeshore.daemon.entry as the workdir-contract child. Constraint on every tranche: the dreamlake.lakeshore.daemon module path must never move.

1.4 dreamlake — the data-plane CLI (dreamlake-py)

[project.scripts] dreamlake = "dreamlake.cli:main". Hand-rolled argv dispatch (no framework at the top level); subcommands mix argparse, params-proto, and ad-hoc parsers. Shared env: DREAMLAKE_REMOTE, DREAMLAKE_API_KEY, DREAMLAKE_BSS_URL (+ DREAMLAKE_BSS_TOKEN, DREAMLAKE_USER, DREAMLAKE_PROJECT, S3_BUCKET, AWS_REGION, DREAMLAKE_ARTIFACTS_BUCKET/REGION, DREAMLAKE_WEB_URL).

Consumers: documentation only (design docs, CLAUDE.md, UI copy in dreamlake-ai). Nothing spawns it programmatically — migration risk is docs drift, and several docs already show stale flags.

1.5 Everything else

A workspace-wide sweep of [project.scripts] found no other Python CLIs in the lakeshore/dreamlake stack (zaku, vuer, ml-dash have their own CLIs and are out of scope for this effort).

2. Design

2.1 Where the TS commands live

Extend the existing lakeshore CLI (repo dreamlake-ai/lakeshore, npm @dreamlake/lakeshore) — no new package. The new worker group sits beside daemon: daemon manages fleet daemons (nymph) through the control plane; worker runs the native Python queue worker on the current machine. The dreamlake data-plane CLI (tranche 2) landed as a scoped lakeshore dreamlake group inside the same CLI rather than a separate package: it talks to different servers (dreamlake-server + BSS) and keeps a separate credential store (~/.dreamlake), so one dreamlake prefix keeps the two product surfaces from colliding (lakeshore auth vs lakeshore dreamlake login, code push vs dreamlake upload) while staying extractable into its own package later — src/cli/dreamlake/ has no imports from the control-plane modules beyond helpers.ts.

2.2 Bun runtime plan

• Dual-runtime, bun-first. bin/lakeshore.js detects process.versions.bun and imports the TS source directly (bun executes TypeScript natively); node keeps the tsx loader; production installs keep using compiled dist/. One bin, three modes.
• Scripts: pnpm cli:bun (run from source under bun), pnpm test:bun (suite under bun), pnpm test:e2e:worker (bun-driven worker e2e).
• bun test runs the existing node:test suite; the remaining failures are bun's node:test shim gaps (oven-sh/bun#5090 — subtests), not CLI bugs. pnpm test (node) stays the merge gate until bun's shim closes the gap. One genuine runtime divergence was found and fixed while getting there: bun's os.homedir() ignores runtime HOME mutations, so all home-relative paths now route through a userHome() helper with node's $HOME-first semantics.
• Compiled binaries (bun build --compile) are the end state for daemon boxes (no node/bun install needed) — deferred until the ink TUI's dynamic imports are validated under the bundler; not required for any current consumer.

2.3 How TS spawns and supervises the Python worker

lakeshore worker start|once → python -u -m dreamlake.lakeshore.daemon:

• Parity rule: flags the user passes are forwarded verbatim; flags they omit stay off the child argv so the module's LAKESHORE_* env fallbacks remain authoritative. LAKESHORE_QUEUE=gpu lakeshore worker start ≡ lakeshore worker start --queue gpu.
• Env: child inherits the full environment plus --env K=V extras. --token travels as LAKESHORE_CLIENT_TOKEN, never argv — it must not leak into ps on shared boxes.
• Supervision: child runs in its own process group with inherited stdio; a tty Ctrl-C reaches only the CLI, which forwards SIGINT/SIGTERM/SIGHUP to the group exactly once (no double-delivery against the worker's KeyboardInterrupt handling). A second signal or the --kill-timeout (15s) escalates to SIGKILL. Exit code is the child's, or 128+N on signal death.
• Interpreter: --python / LAKESHORE_PYTHON / python3; --module overrides the target module (tests, forks). --dry-run prints the exact child command.
• Deliberately not consulted: saved lakeshore auth login credentials. A worker executes code; which plane it drains must be explicit (flag or env) — identical to the Python entrypoint, and what the e2e harness assumes.

2.4 Deprecation path for the Python entrypoints

1. Now (shipped): lakeshore-daemon (console script only) prints a stderr pointer to lakeshore worker start|once; python -m dreamlake.lakeshore.daemon is detected via argv[0] and stays silent — harness and TS CLI spawns see no noise.
2. Tranche 1.5: repoint docs pages, mprocs.yaml, and the sbatch / install scripts at lakeshore worker; refresh the stale lakeshore-py checkout. Docs pages done (2026-07-24); the mprocs / sbatch / install-script repoints remain open.
3. Tranche 2 (shipped): the thin dreamlake subcommands are ported; the dreamlake console script prints the same style of argv[0]-gated pointer (silent for programmatic imports, and silent for artifact push, which stays Python).
4. Endgame (shipped, 0.3.6): lakeshore-daemon dropped from [project.scripts]. The module entrypoint is permanent API (TS CLI
   • nymph entry spawn it).

3. Tranche 1 — shipped

• lakeshore worker start|once (src/cli/worker/), full parity + supervision as specified above.
• Bun runtime support: bin shim, cli:bun / test:bun / test:e2e:worker, userHome() divergence fix.
• scripts/worker-e2e.ts — round-trips real jobs against the SDK's minimal local plane under bun: python submits thunk(40 + 2), worker once drains it, python reads 42 back; a second leg proves worker start drains while running and that SIGTERM forwarding reaps the python worker (CLI exits 143, no orphan). Skips (exit 0) when no SDK python is importable.
• 15 hermetic unit tests (injected spawn/kill): argv parity, token-off-argv, env-fallback discipline, exit-code mapping, forward-once, SIGKILL escalation, handler cleanup.
• lakeshore-daemon deprecation notice + pure-helper tests in the dreamlake-lakeshore repo.

Verified: pnpm test 533 pass / 0 fail (node), worker suite green under bun test, e2e both legs PASS against the SDK venv.

4. Tranche 2 — shipped

The dreamlake data-plane CLI moved into the TS CLI as the lakeshore dreamlake group (dreamlake-ai/lakeshore#17); the Python console script prints an argv[0]-gated pointer (fortyfive-labs/dreamlake#10). Nothing was deleted on the Python side.

4.1 Command mapping

Shared surface: every leaf takes --remote / --bss-url / --token / --debug (env DREAMLAKE_REMOTE / DREAMLAKE_BSS_URL / DREAMLAKE_API_KEY; --debug = localhost servers + a locally minted dev JWT, same as the Python --debug).

4.2 Deliberate surface changes

• Token storage: ~/.dreamlake/auth.yml (chmod 600) instead of the OS keyring — headless-friendly, same policy as lakeshore auth. The device secret in ~/.dreamlake/config.json is shared with the Python CLI, so both look like the same device to vuer-auth.
• Renames: list bindr|dataset|episode → plural nouns; create bindr --episode <glob> → --episodes <glob> (--episode stays the target-scope flag everywhere else); video list --json-output → --json; login's --url → --remote.
• Additions: download --output, --json on every list.
• Dropped: the interactive [n]ext/[p]rev pager (all pages print; script with --json | jq), the ASCII QR code on login.
• Route modernization: asset listing, download, and vectorize's video resolution now use the live GET /nodes / GET /nodes/lookup / GET /nodes/:id/download surface — the Python CLI still queried GET /assets/video[…], which dreamlake-server removed. Download is now kind-agnostic (the Python one only handled video). video's stale http://localhost:4000 default BSS URL was fixed to the shared DREAMLAKE_BSS_URL default (:10234).
• Tranche 3 collapsed into 2: vectorize is pure HTTP orchestration, so it shipped now instead of later.

4.3 Bit-rot found in the Python CLI (why parity ≠ port)

Recorded here so nobody "fixes" the TS port back to the old behavior:

• dreamlake/cli/_target.py is committed broken — a space→project sed rename left ParsedTarget.nameproject, undefined space variables, and constructor kwargs that don't exist. Every command that parses a target crashes with NameError at runtime. The docstring contract (project[@namespace][:episode][//path]) is what the TS target.ts implements (with tests).
• download.py binds its flag to DownloadConfig.sess but the CLI maps --episode → key episode, so the scope never lands — a second, independent breakage of the same command.
• list/download/vectorize target GET /assets/video[…], which no longer exists server-side.
• Server-side, GET /nodes/:id/descendants declares nodes in its response schema but sends descendants — fastify serialization strips the array, so the route effectively returns only total. The TS vectorize avoids it (resolves bindr members via GET /nodes/:id → episode name → GET /nodes?episode=…).

4.4 Also in the tranche-2 PR: daemon start --local removed

The pre-worker lakeshore daemon start --local spawned python -m dreamlake.lakeshore.daemon --server … --tag … — flags the native daemon no longer accepts — so the child died at flag parse while a success LocalDaemonRecord was still written. It is gone; lakeshore worker start|once is the supported local path, and the daemon status/daemon list hints plus shell completion now say so. The nymph fleet surface (daemon launch/install/kill/…) is untouched.

4.5 Verification

• 84 hermetic tests (stubbed fetch, sandboxed HOME) across 10 new suites; pnpm test 612 pass / 1 skip / 0 fail; the new suites are green under bun test.
• Opt-in live smoke pnpm test:e2e:dreamlake (scripts/dreamlake-e2e.ts): self-skips without a server on :10334; verified 3/3 legs PASS against a local dev server (profile, missing-project bindr listing, GET /nodes query surface).
• dreamlake-py: pytest test/ 185 passed / 52 skipped (the pre-existing env-gated live-server tests), including 8 new notice tests.