11 · Storage access from a UDF

A worker moves bytes through a named storage entry without holding any AWS credentials. The control plane mints a presigned URL (or STS temporary credentials) at call time; the code only ever names the storage and the key.

Exercises

dls.Storage(name) — the handle for a storage entry registered on the control plane.
presign(key, operation="get"|"put", expires_in=...) → POST /v1/namespaces/:ns/storages/:name/presign.
credentials(duration_seconds=...) → POST /v1/namespaces/:ns/storages/:name/credentials, returning a TemporaryCredentials with to_boto3_kwargs() and to_env().
The convenience pairs get / put and get_text / put_text, which presign and then do a plain HTTP transfer.
Credential rotation: nothing is embedded in the code or the envelope, so rotation is transparent to running workloads.

Requires

Everything in 09 · Storage round-trip — a registered storage entry with a known object in it.
Everything in 05 · Hello, queue-bound UDF against a control plane, plus 10 · Code mount if the worker does not already have your module.
LAKESHORE_URL + LAKESHORE_CLIENT_TOKEN in the worker's environment (or saved lakeshore auth login credentials in ~/.config/lakeshore/auth.yml, which the Storage side channel does read).

Verifies

A worker reads and writes data it holds no static credentials for, by name, through the control plane.

Run

python

# storage_udf.py
import dreamlake.lakeshore as dls

@dls.udf(queue="default")
def peek(storage: str, key: str) -> dict:
    with dls.Storage(storage) as store:
        blob = store.get(key)
        store.put_text(f"{key}.size", str(len(blob)))
    return {"key": key, "bytes": len(blob)}

inv = peek.submit("my-bucket", "test/hello.txt")
q = dls.SyncQueue("default")
print(q.result(inv, timeout=60))

The bundled example, lakeshore-examples/_legacy/12-storage-access/access.py, walks the same three access modes (presign, credentials, put) from a local script rather than from inside a UDF:

bash

uv run access.py --storage my-bucket --mode presign
uv run access.py --storage my-bucket --mode credentials
uv run access.py --storage my-bucket --mode put

Expected output

{'key': 'test/hello.txt', 'bytes': N} — with no AWS variables in the worker's environment.

If it fails

Symptom	Likely cause
`403 Forbidden` on the presigned URL	The URL expired before the transfer, or the client and S3 clocks are skewed. Raise `expires_in` (`get` presigns for only 300 s by default; the cap is 86400).
`404` from the presign route	No storage entry by that name in this namespace. `lakeshore storage list`.
`NoSuchKey`	The object is not in the bucket. Upload it first.
The worker cannot reach the control plane	`Storage` resolves its server from `LAKESHORE_URL` / `DREAMLAKE_SERVER` / `~/.config/lakeshore/auth.yml`, then falls back to `http://localhost:8080` with no token.
A `boto3` import error	The presigned path is a plain HTTP transfer and needs no boto3. Only `credentials().to_boto3_kwargs()` implies boto3, and you supply it.

Storage is not wired into dls.run yet

dls.run.read / dls.run.write resolve keys on the worker's local filesystem under its --root. Nothing binds a Storage to the run context today, so a data UDF that needs object storage must call dls.Storage(...) explicitly, as above.

Status

Manual.

→ 12 · Compose a cluster declaratively