Telemetry and introspection

The nymph daemon emits structured logs and — when you enable it — exposes a loopback-only HTTP server that surfaces its current status and a buffer of recent log events. nymph tui is the primary consumer; curl works just as well.

Nothing here leaves the host. The server is unauthenticated because it refuses to bind anywhere but loopback.

Structured logging

Configured under [log] in udf-daemon.toml:

udf-daemon.tomltoml

[log]
level = "info"          # error | warn | info | debug | trace
format = "human"        # human | json
event_buffer = 1024     # recent-events ring capacity; 0 disables it

Key	Type	Default	Meaning
`level`	string	`"info"`	Minimum level to emit.
`format`	string	`"human"`	`human` (pretty) or `json` (one object per line).
`event_buffer`	integer	`1024`	Capacity of the ring buffer `/events` serves. `0` disables it — the layer still installs but drops everything.

Environment overrides

Env var	Overrides	Notes
`RUST_LOG`	`log.level`	Full `tracing` `EnvFilter` syntax, e.g. `RUST_LOG=nymph::drain=debug,hyper=warn`. Highest precedence.
`LAKESHORE_LOG_FORMAT`	`log.format`	`human` or `json`, case-insensitive. Highest precedence.

An unrecognised format value falls back to human output and prints nymph: unknown log.format <value>; falling back to "human" to stderr.

With format = "json" the daemon installs tracing_subscriber's JSON layer, so each line is one object carrying timestamp, level, target, and a fields object (the log message itself lands inside fields as message). Point a log shipper at stdout.

The same events are mirrored into the ring buffer the introspection server reads, using the slightly different RecordedEvent shape described below.

The introspection server

Off by default. Enable it under [introspect]:

udf-daemon.tomltoml

[introspect]
enabled = true          # default false
bind = "127.0.0.1"      # loopback only
port = 9876             # default; 0 = OS-assigned ephemeral

Key	Type	Default	Meaning
`enabled`	boolean	`false`	Master switch. TOML is the only way to turn this on.
`bind`	string	`"127.0.0.1"`	Interface to bind. A non-loopback address is refused.
`port`	integer	`9876`	TCP port. `0` lets the OS pick — useful in tests.

The default endpoint is http://127.0.0.1:9876.

LAKESHORE_INTROSPECT does not work

A comment in nymph/src/config.rs advertises a LAKESHORE_INTROSPECT environment variable. Nothing in the codebase reads it. Set [introspect] enabled = true in the TOML — there is no env-var shortcut.

Two behaviours worth knowing:

Startup order. The server binds before /hello, so a daemon that cannot reach the control plane is still inspectable.
Bind failure is not fatal. A refused or unavailable address logs a warning and the daemon continues without introspection. A non-loopback bind (e.g. 0.0.0.0) is an error at that point — introspection is skipped, the daemon runs on.

Loopback-only by design

The server has no authentication. That is safe only because it will not bind a routable interface. Do not put a reverse proxy in front of it.

The implementation is a hand-rolled HTTP/1.1 server on a tokio::net::TcpListener — no keep-alive, explicit Content-Length, Connection: close. Every response is JSON.

Request	Response
`GET /status`	`200` — the `StatusSnapshot` object.
`GET /events`	`200` — `{ count, capacity, events[] }`.
`GET /` or `/healthz`	`200` — `{"ok":true}`.
Any other path	`404` — `{"error":"not_found","message":"no such route"}`.
Any non-`GET` method	`405` — `{"error":"method_not_allowed","message":"only GET is supported"}`.

`GET /status`

status.jsonjson

{
  "version": "0.1.4",
  "worker_id": "65f…",
  "queue": "default",
  "run_state": "polling",
  "backoff_s": 0.0,
  "last_poll_ok_ts": 1750000000.5,
  "uptime_s": 42.7,
  "inflight": { "count": 1, "last_exec_id": "01J…" }
}

Field	Type	Meaning
`version`	string	Daemon binary version (`CARGO_PKG_VERSION`).
`worker_id`	string \| null	Worker id in use — server-issued when available, else the local `{machine_id}-{ulid}`. Null before hello.
`queue`	string \| null	Queue this daemon subscribes to.
`run_state`	string	`starting` \| `polling` \| `backoff` \| `hibernating` \| `draining`.
`backoff_s`	number	Current poll backoff. `0.0` in steady state; grows on repeated failures.
`last_poll_ok_ts`	number \| null	Unix seconds of the last successful poll.
`uptime_s`	number	Seconds since daemon boot.
`inflight`	object	`{ count, last_exec_id }`. `count` covers invocations and execs; `last_exec_id` is a best-effort hint, not a list.

These field names are the nymph tui contract — treat them as stable.

`GET /events`

The ring buffer, oldest → newest:

events.jsonjson

{
  "count": 2,
  "capacity": 1024,
  "events": [
    { "ts": 1750000000.1, "level": "INFO", "target": "nymph::drain",
      "message": "first event", "fields": {} },
    { "ts": 1750000000.2, "level": "WARN", "target": "nymph::drain",
      "message": "poll failed", "fields": { "reason": "connection timeout" } }
  ]
}

Query param	Default	Meaning
`limit=N`	`200`	Return at most the N most-recent events after filtering.
`level=L`	(no filter)	Only events at or above severity L: `error`, `warn`/`warning`, `info`, `debug`, `trace` (case-insensitive). An unrecognised value skips the filter rather than erroring.

Level ranks are ERROR=1, WARN=2, INFO=3, DEBUG=4, TRACE=5, and the filter keeps everything with a rank at or below the requested one.

Each entry is a RecordedEvent: ts (unix seconds, float, sub-second resolution), level (uppercased), target (the tracing target / module path), message (empty when the event carried only structured fields), and fields (a string → string map — every value is stringified, ordered for stable output). count is the number returned after filtering; capacity is log.event_buffer.

The buffer is a VecDeque behind a mutex; the oldest entry is evicted once it is full.