Health monitoring

Two health surfaces, for two audiences.

The local check

ndr-health

Exits 0 only when everything it asserts is true:

• the API is live and ready,
• ClickHouse is up with the complete schema,
• the transport’s runtime identity and service account are the expected ones,
• Grafana is up,
• every configured listener is actually listening — each syslog port, each flow socket, each local socket,
• the privacy policy files match their recorded hashes,
• Zeek is running,
• the timers exist.

Wire it into whatever monitors your other machines. A non-zero exit is the signal; the output says which assertion failed.

The check is deliberately read-only about its listeners: it inspects the socket table for a listening process rather than opening a connection to it. An earlier version connected to the syslog ports to prove they were open, and the source-restriction on those ports then correctly refused the loopback connection — producing a pair of alarming journal lines every minute while the actual data flow was perfectly healthy. Checking by observation rather than by probing avoids manufacturing that kind of symptom.

The tenant health API

GET /api/v1/health, which is what the system view renders. It returns 200 for ok and degraded, and 503 for unavailable — the status code is part of the contract, so an uptime check gets a useful answer without parsing the body.

See the system view for what each field means.

What to watch

Signal	Watch for
Source freshness	A source going stale. Facts age out; they are never deleted, so a stale source is visible rather than absent.
Capture loss	Above one percent. Check CPU, check offloads, check the backup window.
Transport buffer	Growing. Means the backend or the database is not accepting.
Disk	ClickHouse growth against retention.
rejected_last_5m	A steady nonzero rate: a configuration problem, not noise.
Analysis timer	A run that did not complete. It publishes no checkpoint and retries, but a persistent failure needs reading.
telemetry	A degraded line means the heartbeat is failing, which is not a service outage. Nothing about ingest, analysis or the API depends on it.

Telemetry and instance registration

The appliance registers itself once and sends one small daily heartbeat, which appears as a telemetry health component and in the Telemetry & registration section of the system view.

ndr telemetry show                # the payload that would be sent next; no connection
ndr telemetry send --dry-run      # build, sign, check and record it; no connection
sqlite3 /var/lib/ndr/state.sqlite3 \
  'SELECT sent_at,kind,status,detail FROM telemetry_log ORDER BY sent_at DESC LIMIT 10;'

A failing instance server is visible and harmless: the line turns degraded, the retry backoff runs from 30 s to 15 min with at most five attempts per cycle, and ingest, analysis and the API are untouched. Do not treat a red telemetry line as an outage.

Switch it off with NDR_TELEMETRY=off in /etc/ndr/ndr.env and restart the unit; on the pro plan that also ends entitlement renewal. To force a fresh registration after the server side reset an installation, delete the stored secret and restart — the identifier itself must survive, because on the pro plan it is the seat:

sqlite3 /var/lib/ndr/state.sqlite3 'DELETE FROM telemetry_identity;'
systemctl restart ndr

/etc/ndr/install-id (mode 0600) belongs in the backup. Losing it turns the installation into a new one in the portal and costs a pro seat until that is corrected; a malformed identifier file is reported at start and deliberately not overwritten.

The full field list, what is never sent, the retention and the switch are documented under instance registration and telemetry.

Dashboards

Grafana ships with the appliance and is loopback-bound, administered through an SSH tunnel. It carries the sensor health and capture dashboards and reads ClickHouse through a restricted account that cannot write and cannot run foreign DDL.

Alerting

The shipped alert rule starts paused and with no contact point, so installing the appliance does not start sending messages to a place you have not configured yet.

To enable it:

1. Put your notification endpoint and its credential in the dedicated root-owned mode-0600 environment file. Until those are non-empty, do not install the contact-point example — the point is that a half-configured notification path fails visibly at setup rather than silently at 03:00.
2. Install the contact-point example, restart, require a clean provisioning log, and use the built-in contact-point test.
3. Add the notification settings block to the installed rule, restart, and check that the rule shows the named contact point while it is still paused.
4. Only then unpause it and restart again.

Resolved messages stay enabled, so recovery is sent as well as firing. That matters more than it sounds: an alert path that only ever fires trains you to ignore it.

It is not accepted until it has actually delivered

Live acceptance requires a controlled source outage of more than ten minutes, receipt of the problem notification, restored ingestion, and receipt of the recovery notification. No repository test sends to your notification endpoint and none modifies the live notification policy tree — that path is only ever proven by using it.