# DreamLake

> Lakeshore runs your Python functions on remote GPUs — decorate a function, call it, and Lakeshore provisions the host, ships your code, executes it, and streams the result back.

## Happy paths

- [Overview](https://lakeshore.dreamlake.ai/admin/happy-paths.md): Happy paths for whole-stack bring-up, cluster ops, and observability.
- [Overview](https://lakeshore.dreamlake.ai/cli/happy-paths.md): Happy paths for the CLI surface — auth, providers, storage, daemon exec, tokens.
- [Overview](https://lakeshore.dreamlake.ai/python-sdk/happy-paths.md): Happy paths for the Python SDK surface — @udf decorator, queues, fan out, code mounts, storage access.
- [01 · Hello, local Python](https://lakeshore.dreamlake.ai/python-sdk/happy-paths/01-hello-local-python.md): Bare @udf runs in-process — no control plane, no daemon, no network.
- [02 · Hello, hosted CP](https://lakeshore.dreamlake.ai/cli/happy-paths/02-hello-hosted-cp.md): Authenticate the CLI against the hosted control plane and confirm your namespace round-trips.
- [03 · Hello, local stack](https://lakeshore.dreamlake.ai/admin/happy-paths/03-hello-local-stack.md): Local control plane + Mongo + nymph, all bootstrapped through mprocs.
- [04 · Smoke-exec cascade](https://lakeshore.dreamlake.ai/cli/happy-paths/04-smoke-exec-cascade.md): The canonical end-to-end exec test — stdout, stderr, exit code, env, stdin, workdir, timeout, long-poll, python. Bails on first failure.
- [05 · Hello, queue-bound UDF](https://lakeshore.dreamlake.ai/python-sdk/happy-paths/05-hello-queue-udf.md): Queue-bound @udf — submit, get an invocation id, fetch the result.
- [06 · Fan out and collect](https://lakeshore.dreamlake.ai/python-sdk/happy-paths/06-fan-out-gather.md): N parallel invocations collected by id, in submission order.
- [07 · SSH provider smoke](https://lakeshore.dreamlake.ai/cli/happy-paths/07-ssh-provider-smoke.md): Run a small script on a real SSH-reachable host via the provider abstraction — no daemon.
- [08 · Bootstrap daemon on a remote host](https://lakeshore.dreamlake.ai/admin/happy-paths/08-bootstrap-daemon-remote.md): Install nymph on a bare SSH-reachable host and have it register with the control plane.
- [09 · Storage round-trip](https://lakeshore.dreamlake.ai/cli/happy-paths/09-storage-round-trip.md): Register a named S3 storage, then upload, download, and list objects through presigned URLs and STS credentials.
- [10 · Code mount](https://lakeshore.dreamlake.ai/python-sdk/happy-paths/10-code-mount.md): The submitting process archives its git tree to S3; the worker fetches and extracts it before importing your module.
- [11 · Storage access from a UDF](https://lakeshore.dreamlake.ai/python-sdk/happy-paths/11-storage-from-udf.md): A UDF reads and writes a registered storage entry through the Storage handle — the control plane vends the presigned URL, no baked-in credentials.
- [12 · Compose a cluster declaratively](https://lakeshore.dreamlake.ai/admin/happy-paths/12-compose-cluster.md): Declare queues, daemon groups, and instance counts in one lakeshore.yaml. Bring it up, run a job, tear it down.
- [13 · Cloud provider launch](https://lakeshore.dreamlake.ai/admin/happy-paths/13-cloud-provider-launch.md): Provision a fresh worker on EC2 (or GCE / Kube) end to end, from secret resolution through daemon registration.
- [14 · Dashboard observability](https://lakeshore.dreamlake.ai/admin/happy-paths/14-dashboard-observability.md): Submit an invocation from the CLI or SDK and watch it appear and update in the web dashboard.
- [15 · Token lifecycle](https://lakeshore.dreamlake.ai/cli/happy-paths/15-token-lifecycle.md): Admin mints a per-namespace token; the CLI logs in with it; a worker inherits it via LAKESHORE_CLIENT_TOKEN.

## CLI Examples

- [Overview](https://lakeshore.dreamlake.ai/cli/examples.md): Worked CLI examples grouped by command family. Pick the page that matches what you're doing.
- [Auth and setup](https://lakeshore.dreamlake.ai/cli/examples/auth.md): One-time shell setup plus the auth login / status / logout cookbook. Start here before the rest of the cookbook.
- [Providers and discover](https://lakeshore.dreamlake.ai/cli/examples/providers-and-discover.md): Register cloud, SSH, SLURM, and Kube targets, plus the local discover helpers that surface unregistered profiles and hosts.
- [Queues and jobs](https://lakeshore.dreamlake.ai/cli/examples/queues-and-jobs.md): Named work queues, their elasticity declarations, and inspecting invocations via the jobs subcommands.
- [Daemons, exec, and run](https://lakeshore.dreamlake.ai/cli/examples/daemons-and-exec.md): Bring up worker daemons via providers, SSH, or compose; manage the fleet; and shell one-off commands or Python scripts at them.
- [Storage, code, and mounts](https://lakeshore.dreamlake.ai/cli/examples/storage-code-mounts.md): Worked CLI examples for S3 storage entries, git code archives, and mount declarations.
- [Secrets, modes, and tunnels](https://lakeshore.dreamlake.ai/cli/examples/secrets-modes-tunnels-config-nymph.md): Worked CLI examples for the long-tail management commands — secrets, modes, tunnels, nymph binaries, config diagnostics, and completion.

## CLI

- [Installation](https://lakeshore.dreamlake.ai/cli/installation.md): Install the lakeshore CLI, authenticate against a control plane, and understand which env vars override the saved login.
- [lakeshore CLI](https://lakeshore.dreamlake.ai/cli.md): What the `lakeshore` CLI can do — the full top-level command surface, plus short worked examples for registering a provider, bringing a daemon up, and running a one-off command with stdout streamed back.
- [TUI dashboard](https://lakeshore.dreamlake.ai/cli/tui.md): lakeshore tui — an interactive terminal dashboard for Workers, Queues, Invocations, and Storage, with inline per-item action hints (kill, terminate, hibernate, drain, archive, …) wired straight to the control plane.
- [Shell completion](https://lakeshore.dreamlake.ai/cli/completion.md): Tab-completion for `lakeshore` — install it in bash, zsh, or fish, and understand the static subcommand tree, the dynamic name lookups, and the 30-second cache behind them.
- [Release notes](https://lakeshore.dreamlake.ai/cli/release-notes.md): Per-version changelog for the `@dreamlake/lakeshore` CLI.

## Happy Paths

- [Overview](https://lakeshore.dreamlake.ai/dev/happy-paths.md): The canonical end-to-end success flows the project commits to. Each one defines a slice of working surface — together they're the implementation roadmap and the test plan.

## Concepts

- [Architecture](https://lakeshore.dreamlake.ai/get-started/architecture.md): The three layers, the object model the control plane actually stores, and how an invocation flows from your laptop to a remote worker daemon.
- [Queues](https://lakeshore.dreamlake.ai/get-started/queues.md): The unified scheduling primitive — named work queues that daemons subscribe to by name. The queue row, membership rules, the CLI workflow, and where elasticity attaches.
- [Jobs](https://lakeshore.dreamlake.ai/get-started/jobs.md): List, inspect, and cancel the invocations the control plane is tracking — the `lakeshore jobs` verbs, the HTTP routes behind them, and the current limits on killing running work.
- [Functions](https://lakeshore.dreamlake.ai/get-started/functions.md): What a Lakeshore Function is — the two tiers of `@udf`, how a callable crosses the wire, and why a pending job is an id rather than a Future.
- [Mounts](https://lakeshore.dreamlake.ai/get-started/mounts.md): A Mount is a stored shared-filesystem entry (NFS, SMB, FTP/SFTP, S3, s3fs, Drive, Dropbox, bind, configmap) the runner will attach into a job's workdir. Today the registry is live; runner-side activation is a follow-up.
- [Storages](https://lakeshore.dreamlake.ai/get-started/storages.md): Named S3-compatible backends registered per namespace. The control plane holds the credentials and vends presigned URLs or short-lived STS credentials, so your code never handles a raw AWS key.
- [Compose](https://lakeshore.dreamlake.ai/get-started/compose.md): Declarative cluster spec — a lakeshore.yaml at the project root plus top-level `up` / `down` / `ps` / `status` / `logs`. The docker-compose pattern for fleets of daemons.
- [Payload store](https://lakeshore.dreamlake.ai/get-started/payloads.md): The two-tier argument and result cache — small payloads inline on the job row, large payloads in S3 via client-direct presigned URLs the control plane mints.
- [Elasticity](https://lakeshore.dreamlake.ai/get-started/elasticity.md): The four elasticity policies a queue can carry, when to pick each, how shared pools and work-stealing behave, and which knobs have CLI flags.
- [Scaling Rules](https://lakeshore.dreamlake.ai/get-started/scaling-rules.md): The complete rule book the elasticity controller follows on every tick — signals, per-policy arithmetic, scale-down ordering, pools, and work-stealing.
- [Tunnels](https://lakeshore.dreamlake.ai/get-started/tunnels.md): A Tunnel is a stored network passage (WireGuard only) the launcher will bring up before reaching a Provider. The registry is live; `wg-quick up` at launch time is a follow-up.

## Tutorials

- [Tutorials](https://lakeshore.dreamlake.ai/get-started/tutorials.md): Hands-on walkthroughs — a first `@udf` round trip, fan-out with asyncio.gather, declarative cluster bring-up, and an end-to-end operator smoke test.
- [Hello UDF](https://lakeshore.dreamlake.ai/get-started/tutorials/hello-udf.md): Five minutes from install to a queue round-trip — first with zero infrastructure, then against a control plane.
- [Fan out](https://lakeshore.dreamlake.ai/get-started/tutorials/fan-out.md): Fan out N invocations with a list comprehension and join them with stdlib asyncio.gather — no Future type, no scheduling DSL.
- [Compose a cluster](https://lakeshore.dreamlake.ai/get-started/tutorials/compose.md): Declare a cluster in `lakeshore.yaml` and bring it up with `lakeshore up`.
- [End-to-end smoke](https://lakeshore.dreamlake.ai/get-started/tutorials/end-to-end.md): A guided tour of every operator surface — providers, daemons, queues, jobs, exec, storage, secrets, and completion. Run it before tagging a release, or as a hands-on tour of the system.

## Getting started

- [Introduction](https://lakeshore.dreamlake.ai/index.md): Lakeshore runs your Python functions on remote machines — decorate, call, get the result back.
- [Quick start](https://lakeshore.dreamlake.ai/get-started/quick-start.md): From `pip install` to a queue round-trip in a few minutes — first with zero infrastructure, then against a real control plane.
- [Status](https://lakeshore.dreamlake.ai/get-started/status.md): Live test-suite and example status across the Lakeshore repos.

## Nymph

- [Installation](https://lakeshore.dreamlake.ai/nymph/installation.md): Install the nymph daemon binary on a host — install.sh, the supported target triples, the udf-daemon.toml the daemon reads at boot, and the CLI flags and env vars it accepts.
- [Daemon protocol](https://lakeshore.dreamlake.ai/nymph/protocol.md): The wire contract between the nymph daemon and the control plane — msgpack over outbound HTTPS, the hello handshake, the long-poll command channel, the seven command kinds, and the ack / exec-result callbacks.
- [Daemon identity and key rotation](https://lakeshore.dreamlake.ai/nymph/key-rotation.md): Per-host Ed25519 daemon identity — enroll tokens, the request-signing scheme (canonical string plus X-LS-* headers), scheduled rotation with an overlap window, immediate revocation, and the LAKESHORE_REQUIRE_DAEMON_SIGNATURES migration flag.
- [Telemetry and introspection](https://lakeshore.dreamlake.ai/nymph/telemetry.md): Structured logging (human or JSON), an in-memory recent-events ring buffer, and the loopback-only introspection HTTP server (GET /status, /events, /healthz) that the nymph tui monitor consumes.
- [nymph tui monitor](https://lakeshore.dreamlake.ai/nymph/status-tui.md): nymph tui — a read-only live status monitor for a single local daemon. Connects to the daemon's loopback introspection endpoint and renders a status header plus a scrolling, level-filterable event log.
- [Daemons](https://lakeshore.dreamlake.ai/nymph/daemons.md): What the nymph daemon actually does on a host — boot sequence, the concurrent poll loop, idle policy, the six runners and their workdir contracts, host-context detection, and setup commands.
- [Daemon lifecycle](https://lakeshore.dreamlake.ai/nymph/daemon-lifecycle.md): The operator surface for nymph daemons — install, launch, list, exec, setup, hibernate, reset, update, kill, and cleanup, with the real flags each command accepts.
- [Bootstrap a daemon on a remote host](https://lakeshore.dreamlake.ai/nymph/install-remote-daemon.md): `lakeshore daemon install <ssh-alias>` SSHes into a host you can already reach, installs the nymph binary under $HOME, and starts it detached. For hosts the control plane cannot reach — login nodes, on-prem boxes, VMs behind a VPN.

## Python SDK

- [Python SDK](https://lakeshore.dreamlake.ai/python-sdk.md): What you can launch from Python — the native `@udf` decorator, string-key data returns, queues, and invocation ids as durable handles.
- [@udf decorator](https://lakeshore.dreamlake.ai/python-sdk/udf.md): One decorator, four body kinds — local by default, queue-bound for remote, string-key data returns through the dls.run seam.
- [Queue API](https://lakeshore.dreamlake.ai/python-sdk/queue.md): SyncQueue and Queue — submit / result / call / stream, the worker verbs, and Topic pub/sub over the frame journal.
- [Invocation ids](https://lakeshore.dreamlake.ai/python-sdk/invocations.md): No Futures — the durable handle is the invocation id. Reconnect-by-id, fan-out + collect, and live streaming via q.stream.
- [Dispatch planes](https://lakeshore.dreamlake.ai/python-sdk/dispatch.md): The minimal local dispatch plane (Dispatch), HttpDispatch against the Node control plane, and the run_worker loop.

## Architecture

- [The execution matrix](https://lakeshore.dreamlake.ai/python-sdk/architecture/execution-matrix.md): Form × level — every way Python says "run this", against what you're running. The matrix that framed the design space.
- [Dataflow primitives](https://lakeshore.dreamlake.ai/python-sdk/architecture/primitives.md): The declarative front-end draft — @udf(kind=), batch, stack, to_dataset, requeue, and a result algebra that traces into a palette-colored graph.
- [F1 — sync blocking](https://lakeshore.dreamlake.ai/python-sdk/architecture/f1-sync.md): The plain call — local tier, blocking results, and where blocking is exactly right.
- [F2 — deferred ids](https://lakeshore.dreamlake.ai/python-sdk/architecture/f2-ids.md): submit() returns a durable string id — persist it, poll it, reconnect from any process. The row that replaced Futures.
- [F3 — generators](https://lakeshore.dreamlake.ai/python-sdk/architecture/f3-generators.md): Generator bodies stream as journal frames; the consumer's pull is the backpressure, the cursor makes every stream reconnectable.
- [F4 — async](https://lakeshore.dreamlake.ai/python-sdk/architecture/f4-async.md): Async bodies await like any coroutine; dls.Queue is the async sibling of SyncQueue with the same producer verbs.
- [F5 — async generators](https://lakeshore.dreamlake.ai/python-sdk/architecture/f5-async-generators.md): async-gen bodies pull with async for; the streaming row of the async column, and where the async stream surface still thins out.
- [F6 — scopes](https://lakeshore.dreamlake.ai/python-sdk/architecture/f6-scopes.md): dls.scope sets the ambient key prefix; dls.run is the I/O seam — string keys in, string keys out, file bytes never on the wire.
- [F7 — the run object (gap)](https://lakeshore.dreamlake.ai/python-sdk/architecture/f7-run-object.md): The empty row — no pipeline run object exists. What it would hold, what already exists on each side, and a surface sketch.

## Reference

- [Function protocol](https://lakeshore.dreamlake.ai/api/function-protocol.md): The on-the-wire shape of a remote invocation — the msgpack value codec, the Envelope, the three HTTP hops, the ack, and the frame journal.
- [Auth and secrets](https://lakeshore.dreamlake.ai/api/auth-and-secrets.md): The four bearer credentials the control plane recognizes, how the CLI and SDK pick one up, and the AES-256-GCM secret store that provider kwargs reference.
- [Configuration](https://lakeshore.dreamlake.ai/api/configuration.md): The four config files Lakeshore reads — `.dreamrc` (modes and providers), `udf-daemon.toml` (nymph bootstrap), `.lakeshore` (project defaults for daemon launch), and `lakeshore.yaml` (compose).
- [LLM-Readable Docs](https://lakeshore.dreamlake.ai/api/llm-readable.md): Every page is available as clean markdown, plus an llms.txt index, a full-corpus dump, and an importable agent skill.

## Python SDK / Native Rewrite

- [Native Rewrite](https://lakeshore.dreamlake.ai/python-sdk/rewrite.md): Branch feat/native-wire-udf · rewrite of the lakeshore Python SDK from
- [Native API Design](https://lakeshore.dreamlake.ai/python-sdk/rewrite/design.md): Synthesis of the three proposals into one buildable design. Where the three converged (ExtType codec, one wire path, Redis-Streams journal, renamed queue verbs, gather-over-futures) I take the convergent answer. Where they split — the data-UDF file story — I resolve it with the dual-ergonomics filter as the tie-breaker, and say exactly why.
- [Surface Map & Test Plan](https://lakeshore.dreamlake.ai/python-sdk/rewrite/test-plan.md): Status: plan of record for the native rewrite (2026-07-16). Visual checks on the