# Telemetry and introspection

The nymph daemon emits structured logs and — when you enable it — exposes
a **loopback-only** HTTP server that surfaces its current status and a
buffer of recent log events. [`nymph tui`](/nymph/status-tui.md) is the
primary consumer; `curl` works just as well.

Nothing here leaves the host. The server is unauthenticated **because**
it refuses to bind anywhere but loopback.

## Structured logging

Configured under `[log]` in `udf-daemon.toml`:

```toml file="udf-daemon.toml"
[log]
level = "info"          # error | warn | info | debug | trace
format = "human"        # human | json
event_buffer = 1024     # recent-events ring capacity; 0 disables it
```

| Key            | Type    | Default   | Meaning                                                                                     |
| -------------- | ------- | --------- | --------------------------------------------------------------------------------------------- |
| `level`        | string  | `"info"`  | Minimum level to emit.                                                                       |
| `format`       | string  | `"human"` | `human` (pretty) or `json` (one object per line).                                            |
| `event_buffer` | integer | `1024`    | Capacity of the ring buffer `/events` serves. `0` disables it — the layer still installs but drops everything. |

### Environment overrides

| Env var                | Overrides    | Notes                                                                                                  |
| ---------------------- | ------------ | -------------------------------------------------------------------------------------------------------- |
| `RUST_LOG`             | `log.level`  | Full `tracing` `EnvFilter` syntax, e.g. `RUST_LOG=nymph::drain=debug,hyper=warn`. Highest precedence.  |
| `LAKESHORE_LOG_FORMAT` | `log.format` | `human` or `json`, case-insensitive. Highest precedence.                                                |

An unrecognised `format` value falls back to human output and prints
`nymph: unknown log.format <value>; falling back to "human"` to stderr.

With `format = "json"` the daemon installs `tracing_subscriber`'s JSON
layer, so each line is one object carrying `timestamp`, `level`,
`target`, and a `fields` object (the log message itself lands inside
`fields` as `message`). Point a log shipper at stdout.

The same events are mirrored into the ring buffer the introspection
server reads, using the slightly different `RecordedEvent` shape
described [below](#get-events).

## The introspection server

Off by default. Enable it under `[introspect]`:

```toml file="udf-daemon.toml"
[introspect]
enabled = true          # default false
bind = "127.0.0.1"      # loopback only
port = 9876             # default; 0 = OS-assigned ephemeral
```

| Key       | Type    | Default       | Meaning                                                                     |
| --------- | ------- | ------------- | ----------------------------------------------------------------------------- |
| `enabled` | boolean | `false`       | Master switch. TOML is the **only** way to turn this on.                      |
| `bind`    | string  | `"127.0.0.1"` | Interface to bind. A non-loopback address is refused.                         |
| `port`    | integer | `9876`        | TCP port. `0` lets the OS pick — useful in tests.                             |

The default endpoint is `http://127.0.0.1:9876`.

> **Warning:** A comment in `nymph/src/config.rs` advertises a `LAKESHORE_INTROSPECT`
> environment variable. Nothing in the codebase reads it. Set
> `[introspect] enabled = true` in the TOML — there is no env-var shortcut.

Two behaviours worth knowing:

- **Startup order.** The server binds *before* `/hello`, so a daemon that
  cannot reach the control plane is still inspectable.
- **Bind failure is not fatal.** A refused or unavailable address logs a
  warning and the daemon continues without introspection. A non-loopback
  `bind` (e.g. `0.0.0.0`) is an error at that point — introspection is
  skipped, the daemon runs on.

> **Warning:** The server has no authentication. That is safe only because it will not
> bind a routable interface. Do not put a reverse proxy in front of it.

The implementation is a hand-rolled HTTP/1.1 server on a
`tokio::net::TcpListener` — no keep-alive, explicit `Content-Length`,
`Connection: close`. Every response is JSON.

| Request                | Response                                            |
| ---------------------- | ---------------------------------------------------- |
| `GET /status`          | `200` — the `StatusSnapshot` object.                 |
| `GET /events`          | `200` — `{ count, capacity, events[] }`.             |
| `GET /` or `/healthz`  | `200` — `{"ok":true}`.                               |
| Any other path         | `404` — `{"error":"not_found","message":"no such route"}`. |
| Any non-`GET` method   | `405` — `{"error":"method_not_allowed","message":"only GET is supported"}`. |

### `GET /status`

```json file="status.json"
{
  "version": "0.1.4",
  "worker_id": "65f…",
  "queue": "default",
  "run_state": "polling",
  "backoff_s": 0.0,
  "last_poll_ok_ts": 1750000000.5,
  "uptime_s": 42.7,
  "inflight": { "count": 1, "last_exec_id": "01J…" }
}
```

| Field             | Type           | Meaning                                                                                       |
| ----------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| `version`         | string         | Daemon binary version (`CARGO_PKG_VERSION`).                                                   |
| `worker_id`       | string \| null | Worker id in use — server-issued when available, else the local `{machine_id}-{ulid}`. Null before hello. |
| `queue`           | string \| null | Queue this daemon subscribes to.                                                               |
| `run_state`       | string         | `starting` \| `polling` \| `backoff` \| `hibernating` \| `draining`.                           |
| `backoff_s`       | number         | Current poll backoff. `0.0` in steady state; grows on repeated failures.                       |
| `last_poll_ok_ts` | number \| null | Unix seconds of the last **successful** poll.                                                  |
| `uptime_s`        | number         | Seconds since daemon boot.                                                                     |
| `inflight`        | object         | `{ count, last_exec_id }`. `count` covers invocations **and** execs; `last_exec_id` is a best-effort hint, not a list. |

These field names are the `nymph tui` contract — treat them as stable.

### `GET /events`

The ring buffer, oldest → newest:

```json file="events.json"
{
  "count": 2,
  "capacity": 1024,
  "events": [
    { "ts": 1750000000.1, "level": "INFO", "target": "nymph::drain",
      "message": "first event", "fields": {} },
    { "ts": 1750000000.2, "level": "WARN", "target": "nymph::drain",
      "message": "poll failed", "fields": { "reason": "connection timeout" } }
  ]
}
```

| Query param | Default       | Meaning                                                                                                   |
| ----------- | ------------- | ----------------------------------------------------------------------------------------------------------- |
| `limit=N`   | `200`         | Return at most the N most-recent events after filtering.                                                   |
| `level=L`   | *(no filter)* | Only events at or above severity L: `error`, `warn`/`warning`, `info`, `debug`, `trace` (case-insensitive). An unrecognised value skips the filter rather than erroring. |

Level ranks are `ERROR=1`, `WARN=2`, `INFO=3`, `DEBUG=4`, `TRACE=5`, and
the filter keeps everything with a rank at or below the requested one.

Each entry is a `RecordedEvent`: `ts` (unix seconds, float, sub-second
resolution), `level` (uppercased), `target` (the tracing target / module
path), `message` (empty when the event carried only structured fields),
and `fields` (a string → string map — every value is stringified, ordered
for stable output). `count` is the number returned *after* filtering;
`capacity` is `log.event_buffer`.

The buffer is a `VecDeque` behind a mutex; the oldest entry is evicted
once it is full.

## Read next

- [`nymph tui`](/nymph/status-tui.md) — the live monitor that renders
  `/status` and `/events`.
- [Daemon configuration](/api/configuration.md) — the full
  `udf-daemon.toml` schema.
- [Daemon protocol](/nymph/protocol.md) — the control-plane channel, which
  is entirely separate from this local server.