F6 — scopes

Row F6 of the execution matrix: scoped contexts. The other rows are about invocation shape — this row is about the ambient context every call runs inside. dls.scope sets a key prefix; dls.run resolves prefix-relative string keys to local paths. A UDF body never sees a bucket, a credential, or an absolute path — string keys in, string keys out, file bytes never on the msgpack wire.

Everything on this page is the local tier — bare @dls.udf, no queue, no worker. Scopes work with zero infrastructure and touch no network.

L0 — the contract

A data UDF does I/O through dls.run.read(key) and dls.run.write(key) and returns the string keys it wrote. Keys are prefix-relative POSIX strings; the ambient context maps them to real locations. In local mode a key resolves to root / prefix / key — write hands out that path with parents created, read resolves it and raises if it does not exist.

The root defaults to the current working directory; the outermost dls.scope(..., root=...) of a run re-anchors it.

import tempfile
from pathlib import Path

import dreamlake.lakeshore as dls

root = Path(tempfile.mkdtemp())

@dls.udf
def normalize(raw: str) -> str:
    text = dls.run.read(raw).read_text()
    out = dls.run.write("process/clean.txt")
    out.write_text(text.strip().lower())
    return "process/clean.txt"          # the keys you wrote ARE the result

with dls.scope("scenes/0007", root=root):
    dls.run.write("source/raw.txt").write_text("  Hello Lakeshore  ")
    key = normalize("source/raw.txt")

assert key == "process/clean.txt"
assert (root / "scenes/0007/process/clean.txt").read_text() == "hello lakeshore"

normalize names nothing outside its scope — the scene number lives in the ambient prefix, the machine location lives in the root.

The return-your-keys contract is enforced after every call: each key handed out by dls.run.write must appear in the returned manifest. The accepted manifest shapes are str, list[str], tuple[str, ...], and dict[str, str] (named outputs, where the values are the keys). Return the wrong thing and the error states the fix:

@dls.udf
def sloppy(raw: str) -> str:
    dls.run.write("process/mesh.ply").write_text("x")
    return "done"                       # a string, but not the key

KeyError_: UDF wrote key(s) it did not return: ['process/mesh.ply'];
include them in the returned manifest (or don't write them)

Return nothing at all and the message spells out the accepted shapes:

KeyError_: UDF wrote 1 file(s) via dls.run.write (process/mesh.ply) but
did not return their keys; return the key string(s) — str, list[str],
or dict[str, str] — so the outputs are wired into the graph

KeyError_ is a subclass of ValueError, not of the builtin KeyError.

The check is one-directional: a returned string that was never written is allowed, because it may be plain data or an upstream key passed through by a selector, and the two are indistinguishable by inspection. A silent write is not allowed — lineage and downstream wiring key off the returned manifest, so an unreturned key is an invisible artifact. (A typo'd return still gets caught, from the other side: the correctly written key goes unreturned and fails the check.)

L1 — a ledger per call

Every @dls.udf call runs in a fresh child context: same root, prefix, storage, and ids as the ambient scope, but its own write ledger. The unit of the return-your-keys check is one call, so sequential calls under one scope never cross-contaminate.

@dls.udf
def stamp(name: str) -> str:
    dls.run.write(f"out/{name}.txt").write_text(name)
    return f"out/{name}.txt"

with dls.scope("batch", root=root) as ctx:
    keys = [stamp(n) for n in ["a", "b", "c"]]

assert keys == ["out/a.txt", "out/b.txt", "out/c.txt"]
assert ctx.writes == {}                 # each call kept its own ledger

If the ledger were shared, the second call would fail its check for not returning the first call's key. It is not — stamp("b") answers only for out/b.txt. The scope's own context stays clean; it accumulates nothing from the calls inside it.

Generator bodies follow the same contract stretched over time: each yield is a partial manifest, and together the yields are THE manifest, checked once when the body finishes.

@dls.udf
def shards(n: int):
    for i in range(n):
        dls.run.write(f"shards/{i:02d}.txt").write_text(str(i))
        yield f"shards/{i:02d}.txt"     # partial manifests concatenate

with dls.scope("gen", root=root):
    keys = list(shards(3))

assert keys == ["shards/00.txt", "shards/01.txt", "shards/02.txt"]

Yields that are not key-shaped concatenate to nothing, so a generator that streams plain data has no manifest. That is fine as long as it also wrote no files — the check only fires when the ledger is non-empty. A body that both writes files and yields plain data fails, with the same "did not return their keys" message.

L2 — scope per scene

Nesting joins prefixes. A pipeline scopes the run once at the top and each item once inside the loop; the stage chain in the middle is written entirely in relative keys.

import json

@dls.udf
def simulate(params: str) -> str:
    seed = json.loads(dls.run.read(params).read_text())["seed"]
    frames = dls.run.write("process/frames.txt")
    frames.write_text("\n".join(f"frame-{seed}-{t}" for t in range(3)))
    return "process/frames.txt"

@dls.udf
def encode(frames: str) -> str:
    n = len(dls.run.read(frames).read_text().splitlines())
    clip = dls.run.write("outputs/clip.txt")
    clip.write_text(f"clip[{n} frames]")
    return "outputs/clip.txt"

with dls.scope("dataset-v2", root=root):
    for i in range(3):
        with dls.scope(f"scenes/{i:04d}"):
            dls.run.write("source/params.json").write_text(json.dumps({"seed": i}))
            frames = simulate("source/params.json")
            clip = encode(frames)

assert (root / "dataset-v2/scenes/0002/outputs/clip.txt").read_text() == "clip[3 frames]"

Inside the loop, dls.run.prefix is dataset-v2/scenes/0002 — the inner scope joined onto the outer. Neither UDF nor the stage chain mentions a scene number or a directory. Move the run by changing root=, re-home it in a dataset by changing the outer scope string, and every key still reads the same.

For a stage that takes or produces several keys at once, dls.run.read_all and dls.run.write_all mirror the shape you hand them: a str maps to a Path, a list to a list, a dict to a dict.

L3 — scope per branch

When the graph's width is discovered at run time, open a scope per branch. Each branch gets its own namespace, and keys inside stay short and identical across branches.

A branch often needs an input from its parent scope. Keys may climb with ../ — resolution joins prefix and key, so ../ climbs the scope, and the result is checked against the run root: a key may climb out of its prefix, never out of the run. Absolute keys are rejected outright.

import posixpath

@dls.udf
def detect(frame: str) -> list[str]:
    labels = dls.run.read(frame).read_text().split()
    keys = []
    for j, label in enumerate(labels):
        dls.run.write(f"crops/{j:02d}.txt").write_text(label)
        keys.append(f"crops/{j:02d}.txt")
    return keys

@dls.udf
def refine(crop: str) -> str:
    label = dls.run.read(crop).read_text()
    dls.run.write("refined.txt").write_text(label.upper())
    return "refined.txt"

manifest = []
with dls.scope("scan-17", root=root):
    dls.run.write("source/frame.txt").write_text("cat dog heron")
    crops = detect("source/frame.txt")      # width decided by the data
    for j, crop in enumerate(crops):
        with dls.scope(f"objects/{j:02d}"):
            key = refine(f"../../{crop}")   # climb to the parent scope
            manifest.append(posixpath.join(dls.run.prefix, key))

assert manifest == [
    "scan-17/objects/00/refined.txt",
    "scan-17/objects/01/refined.txt",
    "scan-17/objects/02/refined.txt",
]
assert (root / manifest[2]).read_text() == "HERON"

Three labels in the frame, three branch scopes — the structure of the key space mirrors the structure the data dictated. refine is the same function in every branch; only the ambient prefix differs. The manifest is scope-qualified by joining dls.run.prefix onto each returned key — plain strings, ready for a journal line.

Climbing past the root is refused:

KeyError_: key '../../outside' escapes the run root from prefix
"scan-17"; ../ may climb to a parent scope, not out of the run

Why an ambient context

The alternative is threading a path or a context argument through every call, at which point UDF signatures grow a parameter that is not data and every caller becomes responsible for plumbing. With scopes, the pattern rows (F1 through F5) stay pure: a sync call is a call, a generator is a generator, and where the bytes land is decided entirely by the with blocks around them. The same body runs under a temp directory in a test, under scenes/0042 in a pipeline, and under a bound storage on a worker — unchanged.