Living dev note. Iterate freely.

Status

• Phase A shipped queue model + namespaced CRUD + Worker.queues (parallel to the legacy lane primitive).
• Phase A.5 (current) collapsed three competing primitives — Lane, the legacy FK-based Queue (modeId/policy/minHosts/maxHosts), and the new array-membership Queue — into one. The Lane model is gone; Worker.queueId is gone; Invocation.queueId is gone. The scheduling primitive is named Queue and workers reference it by name via Worker.queues: String[].
• Phase B (next): S3 payload store + elasticity controller backed by Queue.daemonTemplate.

Default-queue membership

The empty array. A worker that doesn't carry an explicit queues field — or carries [] — is in the default queue. There's no sentinel string in the schema. lakeshore queues add default … materialises the row at first list / show so a cold namespace still sees a predictable fallback target, but the membership relation is defined by absence, not by presence of "default" in the array.

Queue Types

We have a few different queues:

1. Queue: regular job queue, FIFO
2. Elastic Queues: a job queue connected with providers, that can create new physical hosts
3. Priority Queues: a job queue that returns the highest priority jobs first.
4. Boltzmann Queue: a priority queue that uses boltzmann sampling at a temperature to determine which job gets dispatched first.
5. FILO: first in last out.

Elasticity

1. Fixed Pool
2. Fully Elastic Queue: a new job == new machine (daemon? )
3. Elastic Pool with utilization threshold
4. Max daemon / runs count

Provider / Daemon types

1. aws: creates new physical machines. each with one daemon, can launch many job invocations.
2. slurm: one daemon that can create new job invocations as slurm jobs.
3. kube: one daemon that can create new job invocations as new docker instances.
4. ssh: one daemon can run many docker / gvisor runs.

Examples

The Queue will be launching and dispatching jobs to each daemon. However, now all queues need to do so, a queue can also be a simple, non-provider connected priority queue.

┌─────────────────────────────┬─────────────────┬──────────┬───────────────┐
│                             │  Queue type     │ Provider │  Elasticity   │
├─────────────────────────────┼─────────────────┼──────────┼───────────────┤
│ 1. Training                 │  Queue (FIFO)   │   aws    │ Fully Elastic │
├─────────────────────────────┼─────────────────┼──────────┼───────────────┤
│ 2. Pre-processing Pipeline  │ Boltzmann Queue │   aws    │     Fixed     │
├─────────────────────────────┼─────────────────┼──────────┼───────────────┤
│ 3. Population RL Training   │ Boltzmann Queue │   aws    │    Elastic    │
├─────────────────────────────┼─────────────────┼──────────┼───────────────┤
│ 4. Job inbox (BYO worker)   │  Queue (FIFO)   │    —     │       —       │
└─────────────────────────────┴─────────────────┴──────────┴───────────────┘

Batch Mode:

Packing multiple jobs in the same docker and SLURM run makes the system a lot more efficient.