Quick start

Run a Python function through a queue and get the result back. The first half needs nothing but Python; the second half points the same code at a control plane.

To host your own control plane, follow Setting Up Lakeshore Service.

Before you begin

Python 3.11+ — python --version
Nothing else, for Part 1.

The SDK works with no server at all

A bare @udf runs in-process, and queue-bound UDFs fall back to a local SQLite dispatch plane under ~/.lakeshore whenever LAKESHORE_URL is unset. Steps 1–4 need nothing installed but Python; a control plane only enters the picture at Step 5.

Step 1 — Install the SDK

bash

pip install dreamlake-lakeshore

python

import dreamlake.lakeshore as dls

Imports are lazy: a purely local run never pulls in httpx or msgpack.

Step 2 — A local function

A bare @udf is a plain in-process call. No queue, no worker, no network:

local.pypython

import dreamlake.lakeshore as dls

@dls.udf
def add(a: int, b: int) -> int:
    return a + b

print(add(2, 3))    # 5

Step 3 — A queue round-trip, with zero infrastructure

Bind the function to a queue and it submits instead of running. The handle you get back is the invocation id — a plain string, not a Future.

hello_udf.pypython

import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

def main():
    dp = dls.Dispatch(":memory:")             # the minimal dispatch plane
    q = dls.SyncQueue("hello", dispatch=dp)

    inv = add.submit(2, 3, _dispatch=dp)      # -> "01JZ…"
    print("invocation:", inv)

    dls.run_worker(q, once=True)              # claim + execute, in-process
    print("result:", q.result(inv, timeout=10))

if __name__ == "__main__":
    main()

bash

python hello_udf.py

invocation: 01JZ…
result: 5

dls.Dispatch is a single SQLite store — no control plane, no daemon, no network. Swap it for a real plane later and none of the decorated code changes.

Step 4 — Fan out

There is no gather() helper, because there is no Future type. A list comprehension is the spawn and a loop is the barrier:

fan_out.pypython

import asyncio, threading
import dreamlake.lakeshore as dls

@dls.udf(queue="fan-out-demo")
def square(x: int) -> int:
    return x * x

async def fan_out(n: int, dp) -> list[int]:
    q = dls.Queue("fan-out-demo", dispatch=dp)          # the async surface
    ids = [square.submit(i, _dispatch=dp) for i in range(n)]
    return await asyncio.gather(*(q.result(i, timeout=30) for i in ids))

def main():
    dp = dls.Dispatch(":memory:")
    done = threading.Event()
    threading.Thread(
        target=dls.run_worker,
        args=(dls.SyncQueue("fan-out-demo", dispatch=dp),),
        kwargs={"stop": done.is_set},
        daemon=True,
    ).start()

    print(asyncio.run(fan_out(10, dp)))
    done.set()

if __name__ == "__main__":
    main()

asyncio.gather over the awaitable Queue.result is the join. Results come back in submission order.

Step 5 — Point it at a control plane

Two things change: where the dispatch plane lives, and who runs the worker.

Authenticate

bash

lakeshore auth login --server https://api.lakeshore.dreamlake.ai --namespace <your-namespace>

That writes ~/.config/lakeshore/auth.yml (mode 600) with server, namespace, and token. In a notebook, use environment variables instead:

python

import os
os.environ["LAKESHORE_URL"] = "https://api.lakeshore.dreamlake.ai"
os.environ["LAKESHORE_NAMESPACE"] = "<your-namespace>"
os.environ["LAKESHORE_CLIENT_TOKEN"] = "<your-token>"

With LAKESHORE_URL set, the SDK's default dispatch plane becomes HttpDispatch against that control plane, and your code needs no _dispatch= argument at all.

Run a worker

bash

lakeshore worker start --queue hello --url https://api.lakeshore.dreamlake.ai --token <token>

nymph does not run @udf bodies

lakeshore daemon … manages the Rust fleet daemon, which supervises shell commands and invocations from the control plane. Queue-bound Python UDFs need lakeshore worker start, which spawns python -m dreamlake.lakeshore.daemon. The worker deliberately ignores saved auth login credentials — pass --url / --token, or set LAKESHORE_URL and LAKESHORE_CLIENT_TOKEN.

Submit

python

import dreamlake.lakeshore as dls

@dls.udf(queue="hello")
def add(a: int, b: int) -> int:
    return a + b

inv = add.submit(2, 3)                 # uses LAKESHORE_URL / auth.yml
q = dls.SyncQueue("hello")
print(q.result(inv, timeout=60.0))     # 5

Step 6 — The CLI for fleet management

The SDK submits work. The CLI manages the fleet:

bash

lakeshore queues ls           # queues in your namespace
lakeshore daemon list         # registered daemons
lakeshore jobs list           # invocations the control plane is tracking

See CLI installation to install and log in.

Where to next

I want to…	Page
Understand `@udf` in depth	`@udf` decorator
Work through it as a tutorial	Hello UDF
Understand queues	Queues
Move data in and out	Storages
Declare a cluster in YAML	Compose
Set up cloud providers	Providers
Auto-scale workers	Elasticity
Full CLI reference	CLI