Health monitoring

Two health surfaces, for two audiences.

The local check

ndr-health

Exits 0 only when everything it asserts is true:

• the API is live and ready,
• ClickHouse is up with the complete schema,
• the transport’s runtime identity and service account are the expected ones,
• Grafana is up,
• every configured listener is actually listening — each syslog port, each flow socket, each local socket,
• the privacy policy files match their recorded hashes,
• Zeek is running,
• the timers exist.

Wire it into whatever monitors your other machines. A non-zero exit is the signal; the output says which assertion failed.

The check is deliberately read-only about its listeners: it inspects the socket table for a listening process rather than opening a connection to it. An earlier version connected to the syslog ports to prove they were open, and the source-restriction on those ports then correctly refused the loopback connection — producing a pair of alarming journal lines every minute while the actual data flow was perfectly healthy. Checking by observation rather than by probing avoids manufacturing that kind of symptom.

The tenant health API

GET /api/v1/health, which is what the system view renders. It returns 200 for ok and degraded, and 503 for unavailable — the status code is part of the contract, so an uptime check gets a useful answer without parsing the body.

See the system view for what each field means.

The loopback /healthz

Since 0.36 the API also answers GET /healthz on its loopback bind — unauthenticated, but only for a peer on 127.0.0.1 or ::1 without a proxy header, rate limited, and carrying nothing but the overall status, the version and every component as id, kind and status (no messages, no host paths, none of the objects). It exists for the updater’s health gate and for a curl on the box; it is internal and not promised to any other client. The shape is on the endpoints.

The analysis component

Since 0.33 the report carries an analysis component line (kind: "analysis") and an analysis object describing the analysis job: lag_seconds is the age of the job’s checkpoint — how far the published model results are behind now, null before the first drained window; checkpoint_at; window_open, window_lower and window_upper for a window being paged through; budget and slice_seconds, the bounds it runs under; last_run_at, last_run_flows, last_run_rows, last_run_seconds and last_run_drained for the last pass; and last_error, why the last pass failed, or null.

Since 0.34 the slice is adaptive and remembered, and the object says so (additive fields): slice_seconds is now the width the job currently works at — halved after ClickHouse refused a window for memory or cut it at the query timeout, grown back slowly — and slice_configured_seconds the cap (NDR_ANALYZE_SLICE) it never exceeds; slice_limit_at, slice_limit_class (memory_limit or timeout) and slice_limit_seconds are the last query bound the job ran into and the width it tried, all null while it never has. A failed pass carries last_error_class (the same two values, null for any other failure) and last_run_slice_seconds, the width it ran under; last_error itself then reads memory_limit at slice 1h0m0s: … or timeout at slice 30m0s: …, followed by ClickHouse’s words.

Since 0.35 the object also says how the job runs and how fast it catches up (additive fields): mode is loop while the catch-up service ndr-analyze-loop@<tenant> has written a heartbeat in the last five minutes and timer otherwise; cycles, paused_for_load (the loop is waiting for the one-minute load average to fall under the ceiling) and load (the average it last read) come from that heartbeat and are null/false/null under the timer; load_ceiling is the configured threshold (NDR_ANALYZE_LOAD_CEILING, three quarters of the CPUs by default). backlog_rate_hours_per_hour is the throughput — the widths of the windows drained by the passes of the last hour of wall clock, per hour of wall clock: 1.0 keeps up exactly, more catches up, and the gaps between timer runs count against it; null until the run log is a minute old. eta_seconds is the time until the job is live at that rate, null while it is live, while the rate is not measurable, or while the rate does not beat the inflow.

The component is unavailable without a checkpoint — an installation before its first drained window, which does not degrade the overall status — stale, and the appliance degraded, once the checkpoint is more than three hours behind, and degraded when the last pass failed; the admin-only message carries the reason. A value under an hour is the cadence, not a backlog: a new window opens only once the checkpoint is an hour old. What the numbers mean and what to do when the lag grows is on analysis runs.

The receipt ledger

ingest.receipts (since 0.33) describes the SQLite ingest receipt ledger — the replay check, one row per accepted event — as its last retention sweep measured it: rows (recounted every 15 minutes, reduced by the sweeps in between), oldest_at, db_bytes (the whole state database file, of which the ledger is by far the largest table), deleted_last_sweep, swept_at — all null until the first sweep, never zero — and retention_seconds (NDR_RECEIPT_RETENTION, seven days by default). A health request never counts the table itself. What bounds the ledger and why db_bytes does not shrink is on disk sizing.

Response targets are probed

A response target (opnsense-response, unifi-response) never collects, so until 0.37 its connector:<instance> line stayed unavailable however well test --live had gone. Since 0.37 the service probes every active target at start and every ten minutes — the companion’s health on both firewall nodes, the controller session for UniFi — and the line follows the probe: ok with last_event_at = the probe time; degraded with the companion’s findings (node b: elevation_too_broad …, node a: pf is disabled) as the admin-only message; unavailable with the failure class after a failed probe, or No successful collection or health probe recorded yet before the first success; stale after three missed probes. The states and what to do about them are on the response overview.

The update component

The update line is ok when the appliance is current or has a fresh check; unavailable with a message when the policy is off — switched off is not a fault and does not degrade the overall status; stale when the last successful check is more than 72 hours old, when no check has run, when the last attempt was rolled back, or when a security release is available and the policy will not take it; and degraded when the release keyring cannot be read, when the last attempt ended rolled_back_unhealthy, or when the last attempt ended failed and no check has succeeded since it finished.

Since 0.36 a failed attempt is the updater’s word only until a later check succeeds: the line is then stale for 24 hours (a later update check succeeded) and ok after that, while rolled_back_unhealthy never fades. The component carries the most recent history row as four additive fields — last_attempt_at, last_attempt_outcome (available, applied, failed, rolled_back, rolled_back_unhealthy, skipped), last_attempt_stage and the admin-only last_attempt_error — all null on an appliance that has never attempted an update. The history itself is ndr update --history and is never rewritten. The gate, its fallback and the outcomes are on upgrades.

The content rows

Since 0.39 the report says where its Pro content comes from. None of the three rows changes the overall status: on a free installation they describe the free core, which is a fact and not a fault.

The threat-intel enrichment row carries the refresh’s provenance next to loaded_at and entries: source (keyserver for the signed Pro feed, local for the abuse.ch sources built on the appliance, unknown when no refresh was ever recorded), bundle_version and signing_key_id (null for a local or unknown source), pulled_at, and the admin-only pull_error — why the last pull produced nothing, null after a successful one. The three status words are unchanged; provenance never moves them. See threat intelligence.

The models row (kind: analysis) lists built_in, the size of the built-in catalogue, and packs, always a list, with each installed model pack’s id, version, channel, signing_key_id, models, pulled_at and whether it loaded. It is unavailable when no pack is installed — the built-in model runs alone —, ok when every pack loads, stale when a pack’s last pull failed while the installed version runs, and degraded when an installed pack does not load or the built-in catalogue is broken; the admin-only message names the pack and the reason.

The analyst row (kind: analyst) carries the month’s spend against the licence’s quota: quota_microusd, spent_microusd, runs, period (2026-09) and resets_at, all null without a licence quota. It is unavailable while the provider is disabled (NDR_ANALYST_PROVIDER), ok otherwise, and degraded with the admin-only message The monthly analyst quota of this licence is spent; runs resume on … once the quota is used up — runs then answer 429 quota_exhausted until the first of the next month.

Metrics and push

The same health report is available two more ways: as a Prometheus exposition on GET /metrics — every component state, the ingest and feed rates, incidents by status, connector counters, licence and updater, behind a bearer token and off until one is set — and as a push, the health.transition event of the webhook, sent the moment a component or the appliance as a whole changes state. One function renders all three, so they never disagree.

What to watch

Signal	Watch for
Source freshness	A source going stale. Facts age out; they are never deleted, so a stale source is visible rather than absent.
Capture loss	Above one percent. Check CPU, check offloads, check the backup window.
Transport buffer	Growing. Means the backend or the database is not accepting.
Disk	ClickHouse growth against retention.
rejected_last_5m	A steady nonzero rate: a configuration problem, not noise.
analysis	stale means the checkpoint is more than three hours behind and the appliance is degraded with it; degraded means the last pass failed and last_error says why. A failed pass publishes no checkpoint and the next run retries it at the width it reached; a last_error_class with slice_seconds at the floor, 300 (even at the 5m0s floor), needs a decision — raise NDR_ANALYZE_QUERY_MEMORY_BYTES or NDR_ANALYZE_QUERY_TIMEOUT, or find the query in the database’s query_log. slice_seconds below slice_configured_seconds with a recent slice_limit_at is a job working at a width that fits; it grows back six hours after the last bound. lag_seconds under an hour is the cadence. Since 0.35, backlog_rate_hours_per_hour at or under 1.0 while lag_seconds grows is a job the timer’s share of the clock cannot catch up — switch to the catch-up loop and watch eta_seconds fall; a loop that sits in paused_for_load is bounded by the appliance, not the analyzer, and mode should be back to timer once the loop is disabled. See analysis runs.
ingest.receipts	oldest_at older than the retention by more than a few minutes, or rows still growing after seven days: the sweep is not running — check ndr-jobs.timer. db_bytes is expected to stop growing, not to shrink.
telemetry	A degraded line means the heartbeat is failing, which is not a service outage. Nothing about ingest, analysis or the API depends on it.
threat-intel	unavailable with configured but no successful read means the list file never loaded — the file you maintain is invalid, or no refresh has run yet. degraded (since 0.33) means a newer file was rejected or unreadable and the previous set keeps matching; the admin-visible message says which. loaded_at should move after every refresh and entries should match the refresh’s total. On a Pro installation (since 0.39) source: local with a pull_error is a working installation on the fallback — read the error, the key server, the feed keyring or the licence feature is the usual cause; bundle_version should move roughly daily while source is keyserver. See threat intelligence.
models	stale means the last pack pull failed — the key server, the release keyring, a pack that needs a newer appliance — while the installed version runs; degraded means an installed pack no longer loads, and the analyzer runs without it. unavailable is the free core. See model packs.
analyst	degraded means the licence’s monthly quota is spent; runs resume on resets_at, or sooner with a higher quota on the key in the portal. unavailable is the provider switched off. See the licence quota.
connector:<instance>	stale after missed polls, unlicensed when the licence lacks the feature. degraded, with the reason in the admin-visible message, when the last poll refused rows for a stated reason: a Pi-hole or AdGuard Home instance whose clock runs more than five minutes ahead of the appliance reports query time is more than 5m ahead of the appliance clock, and the state clears on the first poll that refuses nothing — fix the clock, the feed keeps running meanwhile. For NetBox the counters must move at every sync even when nothing changed. See NetBox. A Pi-hole or AdGuard Home instance reports queries, blocked and clients of the last poll; a DHCP leases instance reports leases_active, leases_expired, leases_released, leases_declined and hosts_named, recomputed against the clock on every poll; a Proxmox VE instance reports guests, running, ips, agent_ok, templates and nodes, and is truncated above 2,000 guests. A truncated poll on a resolver means the oldest queries of the window were dropped — raise page_size or lower the interval. A webhook or chat sink instance reports delivered, failed, rejected, dropped and pending, counted for the life of the process; it is unavailable with the failure class after a delivery round the receiver did not acknowledge and never stale — a quiet network is not a failing sink. pending above zero for longer than the backoff (15 min) means the receiver stopped answering; a growing dropped means it was down longer than max_pending events. A response target (since 0.37) follows its ten-minute probe: degraded is the companion’s own finding — a sudoers grant wider than the six forms, pf off on a node — and is cleared on the firewall, not on the appliance; unavailable names the failure class of the last probe; stale means three probes in a row did not happen.
update	degraded with release keyring cannot be read is the keyring file; degraded with last_attempt_outcome: rolled_back_unhealthy needs a person and never fades; degraded with failed clears itself once a later check succeeds (24 hours of stale, then ok). stale for more than 72 hours is the channel being unreachable — ndr update --check shows the transport error. See the update component.

Telemetry and instance registration

The appliance registers itself once and sends one small daily heartbeat, which appears as a telemetry health component and in the Telemetry & registration section of the system view.

ndr telemetry show                # the payload that would be sent next; no connection
ndr telemetry send --dry-run      # build, sign, check and record it; no connection
sqlite3 /var/lib/ndr/state.sqlite3 \
  'SELECT sent_at,kind,status,detail FROM telemetry_log ORDER BY sent_at DESC LIMIT 10;'

A failing instance server is visible and harmless: the line turns degraded, the retry backoff runs from 30 s to 15 min with at most five attempts per cycle, and ingest, analysis and the API are untouched. Do not treat a red telemetry line as an outage.

Switch it off with NDR_TELEMETRY=off in /etc/ndr/ndr.env and restart the unit; on the pro plan that also ends entitlement renewal. To force a fresh registration after the server side reset an installation, delete the stored secret and restart — the identifier itself must survive, because on the pro plan it is the seat:

sqlite3 /var/lib/ndr/state.sqlite3 'DELETE FROM telemetry_identity;'
systemctl restart ndr

/etc/ndr/install-id (mode 0600) belongs in the backup. Losing it turns the installation into a new one in the portal and costs a pro seat until that is corrected; a malformed identifier file is reported at start and deliberately not overwritten.

The full field list, what is never sent, the retention and the switch are documented under instance registration and telemetry.

Dashboards

Grafana ships with the appliance and is loopback-bound, administered through an SSH tunnel. It carries the sensor health and capture dashboards and reads ClickHouse through a restricted account that cannot write and cannot run foreign DDL.

Alerting

The shipped alert rule starts paused and with no contact point, so installing the appliance does not start sending messages to a place you have not configured yet.

To enable it:

1. Put your notification endpoint and its credential in the dedicated root-owned mode-0600 environment file. Until those are non-empty, do not install the contact-point example — the point is that a half-configured notification path fails visibly at setup rather than silently at 03:00.
2. Install the contact-point example, restart, require a clean provisioning log, and use the built-in contact-point test.
3. Add the notification settings block to the installed rule, restart, and check that the rule shows the named contact point while it is still paused.
4. Only then unpause it and restart again.

Resolved messages stay enabled, so recovery is sent as well as firing. That matters more than it sounds: an alert path that only ever fires trains you to ignore it.

It is not accepted until it has actually delivered

Live acceptance requires a controlled source outage of more than ten minutes, receipt of the problem notification, restored ingestion, and receipt of the recovery notification. No repository test sends to your notification endpoint and none modifies the live notification policy tree — that path is only ever proven by using it.