# The execution matrix

Python gives you an unusually large number of ways to express "run this" —
sync calls, futures, generators, coroutines, async generators, context
managers, run objects — and the pre-rewrite API had accreted most of them.
This page is the matrix that framed the question before any opinion: **two
axes**, the *form* (the Python execution pattern) and the *level* (what you
are running).

The question as originally posed: *generators, async generators, sync,
async, pipeline run object — what are the different ways we have, for
running the basics, the systems, and the system-level imperative dynamic
graph?*

## Axis 1 — the form

The full vocabulary Python offers, and how you get values out of each:

| Form | Invoke | Get results | Driver | Barrier? |
| --- | --- | --- | --- | --- |
| F1 sync blocking | `y = f(x)` | return value | caller | — |
| F2 deferred handle | `h = f.submit(x)` | resolve `h` later | caller polls | deferred |
| F3 sync generator | `for x in g()` | lazy pull, one at a time | **consumer** (backpressure) | streaming |
| F3′ gen with `.send()` | `x = yield y` | two-way (trampoline / scheduler) | scheduler | streaming |
| F4 async coroutine | `await f(x)` | awaited value | event loop | deferred |
| F5 async generator | `async for x in g()` | lazy pull plus await | consumer plus loop | streaming |
| F6 context manager | `with X() as h:` / `async with` | scoped lifecycle | — | — |
| F7 run/handle object | `r = pipe.submit(...)` | poll status, stream events, await | caller tracks | either |

Each form F1–F7 has its own page in the **Patterns** section, expanded into
runnable examples against the shipped SDK.

## Axis 2 — the level

- **L0 basic** — one UDF, one invocation.
- **L1 systems** — N calls across queues and workers; the scheduling
  substrate.
- **L2 static pipeline** — a fixed multi-stage DAG, knowable at author time
  (`examples/02_pipeline.py`).
- **L3 dynamic imperative graph** — structure discovered *while running*:
  data-dependent fan-out width, conditional branches, rework loops,
  recursion.

## The original grid — drawn against the pre-rewrite API

> **Warning:** Every symbol in the grid below (`apply`, `apply_spawn`, `apply_async`,
>   `gather`, `map`, `as_completed`, `Future`, `QueueFuture`, `Pool`,
>   `rpc_future`, `@pipeline`) was deleted in the native rewrite. The grid is
>   preserved because the *shape* of the analysis — where the mass sat, where
>   the holes were — is what drove the design. For what to write today, skip
>   to the re-grounded grid below.

| | L0 basic | L1 systems / fan-out | L2 static pipeline | L3 dynamic graph |
| --- | --- | --- | --- | --- |
| **F1 sync blocking** | `apply(fn, …)`, `fut.get()`, `@udf` local | `Queue.map(fn, xs)`, `gather([…])` | gather → gather (barriers) | `while` loop plus blocking calls plus requeue (serial) |
| **F2 sync plus future** | `apply_spawn` → `Future`; `@udf(queue=q)` → `QueueFuture` | `[fn(i) for i …]` → `list[Future]`; `as_completed` | a stage-1 Future passed into a stage-2 call (implicit edge) | spawn futures off resolved values; `parent_invocation_id` builds the tree |
| **F3 sync generator** | `for chunk in q.rpc_future(…)` stream | `as_completed` yields by completion; draft `batch(src, n)` | draft `@pipeline` (a generator that yields `to_dataset(…)`) | `.send()`-driven dynamic pull → **GAP** |
| **F4 async coroutine** | `await apply_async`, `await fut` | `await gather_async`, `map_async` | an `await` chain of stages | `asyncio.create_task` dynamic spawn |
| **F5 async generator** | streaming one call via `async for` → **GAP** | `as_completed_async(futs)` | async-gen stages → **GAP** | `async for` over a growing stream → **GAP** |
| **F6 context manager** | `with Queue(…) as q:` | `Pool(size=4)` | — | — |
| **F7 run object** | — | — | **GAP** (`@pipeline` returned the function unchanged) | **GAP** — no run you can poll, stream, or await |

### What the grid showed

- **The legacy API lived almost entirely in F1, F2, and F4** — blocking,
  futures, async. That was roughly fifteen symbols spent on one region:
  `apply` / `spawn` / `call` in sync and async spellings, `gather` / `map` /
  `as_completed` in sync and async spellings, and two Future classes.
- **The dataflow-primitives draft lived in F3** — `batch`, `@pipeline`, all
  generator-shaped. A *different form*, which is why it read so differently
  from the client.
- **The whole F7 row was empty.** `@pipeline` was a stub that returned the
  function unchanged. The draft's `Graph` is an author-time structure, not a
  live handle, and callee-side run telemetry is not a caller-side tracker.
- **Column L3 was only reachable imperatively.** It was real — the
  `parent_invocation_id` tree captured it at runtime — but never a
  first-class object you could hold, stream, or visualize.

Two cross-cutting tensions fall straight out of the grid: **barrier versus
streaming** (F1/F2 gather against F3/F5 generators), and **static versus
dynamic** (L2 knowable at author time against L3 discovered at run time).
The empty F7 row is exactly where those two reconcile — a run object is
*the* thing that can be both streamed and dynamic.

## Re-grounded: the shipped grid (0.3.7)

The rewrite deleted the legacy surface and collapsed most of the form axis
into the function body itself:

| Form | Shipped spelling | Status |
| --- | --- | --- |
| F1 sync | `f(x)` — local tier, or submit-and-wait when a queue is bound | shipped |
| F2 deferred | `inv = f.submit(x)` then `q.result(inv)` | shipped (**ids, not Futures**) |
| F3 generator | generator body; `q.stream(inv, since=cursor)` | shipped |
| F4 async | `async def` body; `dls.Queue`, the async sibling | shipped |
| F5 async-gen | `async def` generator body; `async for … in q.stream(…)` | shipped |
| F6 scope | `with dls.scope("scenes/0007"):` | shipped |
| F7 run handle | *none — there is no pipeline run object* | **GAP — the open seam** |

Three of the original F5 gaps closed because the async generator became a
first-class body kind, detected by `inspect.isasyncgenfunction` at
decoration. Two cells remain open rather than shipped:

- **F3 × L3, the two-way `.send()` trampoline.** The journal is one-way —
  workers append frames, consumers pull them — so there is no way to send a
  value *into* a paused remote body. Locally, plain Python generators do it
  fine.
- **F5 × L3, merging a growing set of streams.** No fan-in verb exists; you
  hand-roll it with a `TaskGroup` and an `asyncio.Queue`. See
  [F5](/python-sdk/architecture/f5-async-generators.md) for the working, clunky version.

The SDK's own matrix marks both of those cells "explore" rather than GAP —
they are reachable by hand today. Only the F7 row is marked GAP outright.
See [F7 — the run object](/python-sdk/architecture/f7-run-object.md).

The canonical, runnable version of this matrix lives in the SDK repo at
`docs/execution-patterns.md`, with explorations `examples/03_body_kinds.py`
through `examples/06_dynamic_graph.py`
([PR #18](https://github.com/dreamlake-ai/dreamlake-lakeshore/pull/18)).