Metrics

GET /metrics on the API is a Prometheus text exposition of the whole appliance: health component states, per-feed ingest rates, incidents by status and severity, connector states and counters, the updater, the licence and the analyst budget. It is Free, it is off until a token is set, and the route does not exist at all — 404 — while it is off.

Enable it

1. Set the token. In /etc/ndr/ndr.env (mode 0600), at least 16 characters:

printf 'NDR_METRICS_TOKEN=%s\n' "$(openssl rand -hex 24)" >> /etc/ndr/ndr.env
systemctl restart ndr-api

2. Scrape on the loopback. The API binds to 127.0.0.1:8080 and nothing else, so the scraper runs on the appliance — a Prometheus, Alloy or Grafana Agent instance, or Grafana’s own Prometheus data source through one of them. The credential is a plain bearer header:

curl -sS -H "Authorization: Bearer $NDR_METRICS_TOKEN" http://127.0.0.1:8080/metrics | head

Request	Answer
Token unset in the service environment	404 — the route is not registered.
Wrong or missing Authorization header	401.
Anything but GET	405.
GET with Authorization: Bearer <token>	200, text/plain; version=0.0.4.

The exposition is not part of the signed /api/v1 surface, and the signed service credential of the API is deliberately not accepted on it: that credential exists so the web layer can act for a user, no scraper speaks it, and the exposition is not tenant-scoped — it is the whole appliance with a tenant label on every tenant-bound series.

A scrape configuration

For Prometheus, with the token in a file the scraper can read and the appliance cannot:

scrape_configs:
  - job_name: nyxtrace
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/nyxtrace-metrics-token
    static_configs:
      - targets: ['127.0.0.1:8080']

The same target in Grafana Alloy:

prometheus.scrape "nyxtrace" {
  targets = [{ __address__ = "127.0.0.1:8080" }]
  scrape_interval = "30s"
  authorization {
    type             = "Bearer"
    credentials_file = "/etc/alloy/nyxtrace-metrics-token"
  }
  forward_to = [prometheus.remote_write.default.receiver]
}

The body is rendered at most every ten seconds and served from that cache in between, so a short scrape interval or several scrapers cost nothing extra: one scrape storm cannot multiply the health reads against ClickHouse.

The metric families

Every metric is a gauge, read at scrape time from the same reports the system view shows, and every name is prefixed nyxtrace_. State sets (health_status, health_component, connector_state, license_state) are enumerated the way kube-state-metrics does: one row per possible value, 1 for the current one.

Metric	Labels	Meaning
build_info	version	Always 1.
health_status	tenant, status	Overall status of /api/v1/health; 1 for the current one of ok, stale, degraded, unavailable.
health_component	tenant, component, kind, status	One row per component and status, 1 for the current status. Components are the ids of the health report: clickhouse, a sensor id, connector:<instance>, license, telemetry, update, analysis, threat-intel, and since 0.39 models (kind="analysis") and analyst (kind="analyst") — the content rows. 0.39 adds no family of its own: the feed’s provenance, the installed packs and the licence quota are read from the health report, not scraped.
health_component_lag_seconds	tenant, component, kind	Seconds since the component’s last event or collection, where the report has one.
ingest_events_5m, ingest_unparsed_5m, ingest_rejected_5m	tenant	The ingest counters of the health report.
feed_events_5m	tenant, sensor	Events received per feed in the last five minutes.
feed_last_event_timestamp_seconds	tenant, sensor	When the feed’s most recent event arrived.
capture_loss_percent	tenant	The mirror’s last capture-loss sample, when one exists.
incidents	tenant, status, severity	Incident count per status and severity.
connector_state	tenant, instance, connector, state	The framework state the scheduler persisted (unknown, ok, stale, error, degraded, disabled, unlicensed), one row with value 1.
connector_consecutive_failures, connector_items_last_hour, connector_last_success_timestamp_seconds	tenant, instance, connector	The health line’s numbers.
connector_count	tenant, instance, connector, counter	The instance’s own counters: an inventory source’s sizes, a webhook sink’s delivered, failed, rejected, dropped, pending.
analysis_lag_seconds	tenant	Seconds the published analysis results are behind now — the age of the analysis checkpoint. Absent before the first drained window. Under 3,600 is the hourly cadence, not a backlog.
analysis_window_open	tenant	1 while the job is paging through a window it has not drained (catching up).
analysis_budget_rows	tenant	Rows per pass and phase, NDR_ANALYZE_FLOW_BUDGET.
analysis_slice_seconds	tenant	Since 0.34 the width the job currently works at — the remembered one, halved after a query bound, grown back slowly. (In 0.33 it was the configured slice.)
analysis_slice_configured_seconds	tenant	The cap, NDR_ANALYZE_SLICE, the width never exceeds (0.34).
analysis_slice_limit_seconds	tenant	The width the job last ran into a query bound at (0.34). Absent while it never has.
analysis_slice_limit_timestamp_seconds	tenant, class	When the job last ran into a query bound and halved its slice; class is memory_limit or timeout (0.34). Absent while it never has.
analysis_last_run_flows, analysis_last_run_rows, analysis_last_run_seconds, analysis_last_run_timestamp_seconds	tenant	The last pass: flow rows consumed, rows of all phases, its duration, when it ran.
analysis_last_run_slice_seconds	tenant	The width the last pass ran under (0.34).
analysis_last_run_failed	tenant	1 when the last pass failed; the reason is analysis.last_error on the health report. See analysis runs.
analysis_last_run_bound	tenant, class	A set: one row per class (memory_limit: max_memory_usage, timeout: max_execution_time), 1 when the last pass ran into that bound (0.34). Both 0 for a pass that succeeded or failed for another reason.
analysis_mode	tenant, mode	A set: one row per mode (timer, loop), 1 for how the job runs — loop while the catch-up service ndr-analyze-loop@ has a fresh heartbeat (0.35).
analysis_cycles	tenant	Cycles of the analysis loop the catch-up service has run since it started. Present only under the loop (0.35).
analysis_paused_for_load	tenant	1 while the catch-up service is waiting for the one-minute load average to fall under its ceiling; 0 otherwise, and under the timer (0.35).
analysis_load_ceiling	tenant	The load average above which the loop pauses, NDR_ANALYZE_LOAD_CEILING (0.35).
analysis_backlog_rate_hours_per_hour	tenant	Hours of analysis windows drained per hour of wall clock over the last hour: 1 keeps up exactly, more catches up, the gaps between timer runs count against it. Absent until the run log is a minute old (0.35).
analysis_eta_seconds	tenant	Seconds until the job is live at its current rate. Absent while it is live, while the rate is not measurable, or while the rate does not beat the inflow (0.35).
receipts_rows, receipts_oldest_timestamp_seconds, receipts_deleted_last_sweep, receipts_retention_seconds	tenant	The ingest receipt ledger as its last retention sweep measured it: rows at the last count, the oldest receipt still kept, what the last sweep deleted, and NDR_RECEIPT_RETENTION. Absent until the first sweep.
state_db_bytes	tenant	Size of the SQLite state database file. It stops growing once the ledger reaches its retention; it does not shrink — see disk sizing.
analyst_budget_microusd	tenant, provider, kind	Today’s analyst budget: limit, reserved, spent.
license_enabled, license_state (state), license_expires_timestamp_seconds, license_grace_until_timestamp_seconds, license_seats	—	The licence report. license_state is a set with off when licensing is switched off.
update_enabled, update_available, update_available_security, update_blocked, update_last_check_timestamp_seconds, update_last_check_failed	—	The updater report.
metrics_generated_timestamp_seconds	—	When the body was rendered — the age of the cached body you are looking at.

A first scrape looks like this:

# TYPE nyxtrace_build_info gauge
nyxtrace_build_info{version="0.29.0"} 1
# TYPE nyxtrace_health_status gauge
nyxtrace_health_status{tenant="example",status="ok"} 1
nyxtrace_health_status{tenant="example",status="stale"} 0
nyxtrace_health_status{tenant="example",status="degraded"} 0
nyxtrace_health_status{tenant="example",status="unavailable"} 0
# TYPE nyxtrace_feed_events_5m gauge
nyxtrace_feed_events_5m{tenant="example",sensor="sensor_example_vsensor"} 820
# TYPE nyxtrace_incidents gauge
nyxtrace_incidents{tenant="example",status="open",severity="high"} 2
# TYPE nyxtrace_connector_count gauge
nyxtrace_connector_count{tenant="example",instance="hook",connector="webhook",counter="delivered"} 41
nyxtrace_connector_count{tenant="example",instance="hook",connector="webhook",counter="pending"} 0

Useful expressions

Expression	Meaning
nyxtrace_health_status{status!="ok"} == 1	Anything not green.
sum by (tenant, severity) (nyxtrace_incidents{status=~"open|investigating"})	Open incidents by severity.
nyxtrace_connector_count{counter="pending"} > 0	A sink that has something to deliver — alert when it stays true longer than the sink’s backoff (15 min): the receiver stopped answering.
nyxtrace_feed_events_5m == 0	A silent feed.
nyxtrace_analysis_lag_seconds > 3*3600	The analysis is more than three hours behind — the same threshold at which the analysis component turns stale.
nyxtrace_analysis_last_run_failed == 1	The last analysis pass failed; read analysis.last_error on the health report.
nyxtrace_analysis_last_run_bound == 1	The last pass ran into a query bound; the class label says which. Transient while the job halves its slice; alert when it stays true and nyxtrace_analysis_slice_seconds == 300 — the floor.
nyxtrace_analysis_slice_seconds < nyxtrace_analysis_slice_configured_seconds	The analyzer is working at a narrower slice than configured. Not an alert by itself — it grows back six hours after the last bound; a dashboard line.
nyxtrace_analysis_lag_seconds > 3600 and nyxtrace_analysis_backlog_rate_hours_per_hour <= 1	The job is behind and not catching up at its current share of the clock (0.35). Under the timer this is the cue to switch to the catch-up loop; under the loop, check nyxtrace_analysis_paused_for_load.
nyxtrace_analysis_eta_seconds	A dashboard line while the loop runs: the time until live at the measured rate. It should fall between scrapes; absent once the job is live.
nyxtrace_analysis_mode{mode="loop"} == 1 for longer than a day	The catch-up loop is still enabled. It is meant to be switched off once nyxtrace_analysis_lag_seconds is under 3,600 — the timer is the default.
time() - nyxtrace_receipts_oldest_timestamp_seconds > nyxtrace_receipts_retention_seconds + 3600	The receipt sweep has stopped: the oldest receipt is older than the retention.
nyxtrace_license_state{state="grace"} == 1	The licence is in its 72-hour grace window.
time() - nyxtrace_update_last_check_timestamp_seconds > 3*86400	The updater has not checked in for three days.

Cardinality

The label set is bounded by construction: tenants, sensor ids, connector instances and closed vocabularies. Nothing on /metrics carries a device, an address, a domain, an incident id or a diagnostic text — those live in the API and the dashboard, where they are role-gated. The number of series therefore grows with the things you configured, never with what the network did, and a scraper can keep it for years without a retention concern of its own.

The same reasoning is why there are no histograms and no per-request timings: the exposition is the state of the appliance, not the performance of the service. A metric you want to alert on and cannot find here is a request for the health report first, because the two always agree.

Relation to the health API and the webhook

/metrics, GET /api/v1/health and the webhook’s health.transition events are three views of one health report — the same function renders all three, so a receiver, a dashboard and a Grafana panel never disagree about a component’s state. Use the health API for an uptime check that wants a status code, the metrics for history and panels, and the webhook for a push the moment a state changes.