Metrics

GET /metrics on the API is a Prometheus text exposition of the whole appliance: health component states, per-feed ingest rates, incidents by status and severity, connector states and counters, the updater, the licence and the analyst budget. It is Free, it is off until a token is set, and the route does not exist at all — 404 — while it is off.

Enable it

1. Set the token. In /etc/ndr/ndr.env (mode 0600), at least 16 characters:

printf 'NDR_METRICS_TOKEN=%s\n' "$(openssl rand -hex 24)" >> /etc/ndr/ndr.env
systemctl restart ndr-api

2. Scrape on the loopback. The API binds to 127.0.0.1:8080 and nothing else, so the scraper runs on the appliance — a Prometheus, Alloy or Grafana Agent instance, or Grafana’s own Prometheus data source through one of them. The credential is a plain bearer header:

curl -sS -H "Authorization: Bearer $NDR_METRICS_TOKEN" http://127.0.0.1:8080/metrics | head

Request	Answer
Token unset in the service environment	404 — the route is not registered.
Wrong or missing Authorization header	401.
Anything but GET	405.
GET with Authorization: Bearer <token>	200, text/plain; version=0.0.4.

The exposition is not part of the signed /api/v1 surface, and the signed service credential of the API is deliberately not accepted on it: that credential exists so the web layer can act for a user, no scraper speaks it, and the exposition is not tenant-scoped — it is the whole appliance with a tenant label on every tenant-bound series.

A scrape configuration

For Prometheus, with the token in a file the scraper can read and the appliance cannot:

scrape_configs:
  - job_name: nyxtrace
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/nyxtrace-metrics-token
    static_configs:
      - targets: ['127.0.0.1:8080']

The same target in Grafana Alloy:

prometheus.scrape "nyxtrace" {
  targets = [{ __address__ = "127.0.0.1:8080" }]
  scrape_interval = "30s"
  authorization {
    type             = "Bearer"
    credentials_file = "/etc/alloy/nyxtrace-metrics-token"
  }
  forward_to = [prometheus.remote_write.default.receiver]
}

The body is rendered at most every ten seconds and served from that cache in between, so a short scrape interval or several scrapers cost nothing extra: one scrape storm cannot multiply the health reads against ClickHouse.

The metric families

Every metric is a gauge, read at scrape time from the same reports the system view shows, and every name is prefixed nyxtrace_. State sets (health_status, health_component, connector_state, license_state) are enumerated the way kube-state-metrics does: one row per possible value, 1 for the current one.

Metric	Labels	Meaning
build_info	version	Always 1.
health_status	tenant, status	Overall status of /api/v1/health; 1 for the current one of ok, stale, degraded, unavailable.
health_component	tenant, component, kind, status	One row per component and status, 1 for the current status. Components are the ids of the health report: clickhouse, a sensor id, connector:<instance>, license, telemetry, update, …
health_component_lag_seconds	tenant, component, kind	Seconds since the component’s last event or collection, where the report has one.
ingest_events_5m, ingest_unparsed_5m, ingest_rejected_5m	tenant	The ingest counters of the health report.
feed_events_5m	tenant, sensor	Events received per feed in the last five minutes.
feed_last_event_timestamp_seconds	tenant, sensor	When the feed’s most recent event arrived.
capture_loss_percent	tenant	The mirror’s last capture-loss sample, when one exists.
incidents	tenant, status, severity	Incident count per status and severity.
connector_state	tenant, instance, connector, state	The framework state the scheduler persisted (unknown, ok, stale, error, degraded, disabled, unlicensed), one row with value 1.
connector_consecutive_failures, connector_items_last_hour, connector_last_success_timestamp_seconds	tenant, instance, connector	The health line’s numbers.
connector_count	tenant, instance, connector, counter	The instance’s own counters: an inventory source’s sizes, a webhook sink’s delivered, failed, rejected, dropped, pending.
analyst_budget_microusd	tenant, provider, kind	Today’s analyst budget: limit, reserved, spent.
license_enabled, license_state (state), license_expires_timestamp_seconds, license_grace_until_timestamp_seconds, license_seats	—	The licence report. license_state is a set with off when licensing is switched off.
update_enabled, update_available, update_available_security, update_blocked, update_last_check_timestamp_seconds, update_last_check_failed	—	The updater report.
metrics_generated_timestamp_seconds	—	When the body was rendered — the age of the cached body you are looking at.

A first scrape looks like this:

# TYPE nyxtrace_build_info gauge
nyxtrace_build_info{version="0.29.0"} 1
# TYPE nyxtrace_health_status gauge
nyxtrace_health_status{tenant="example",status="ok"} 1
nyxtrace_health_status{tenant="example",status="stale"} 0
nyxtrace_health_status{tenant="example",status="degraded"} 0
nyxtrace_health_status{tenant="example",status="unavailable"} 0
# TYPE nyxtrace_feed_events_5m gauge
nyxtrace_feed_events_5m{tenant="example",sensor="sensor_example_vsensor"} 820
# TYPE nyxtrace_incidents gauge
nyxtrace_incidents{tenant="example",status="open",severity="high"} 2
# TYPE nyxtrace_connector_count gauge
nyxtrace_connector_count{tenant="example",instance="hook",connector="webhook",counter="delivered"} 41
nyxtrace_connector_count{tenant="example",instance="hook",connector="webhook",counter="pending"} 0

Useful expressions

Expression	Meaning
nyxtrace_health_status{status!="ok"} == 1	Anything not green.
sum by (tenant, severity) (nyxtrace_incidents{status=~"open|investigating"})	Open incidents by severity.
nyxtrace_connector_count{counter="pending"} > 0	A sink that has something to deliver — alert when it stays true longer than the sink’s backoff (15 min): the receiver stopped answering.
nyxtrace_feed_events_5m == 0	A silent feed.
nyxtrace_license_state{state="grace"} == 1	The licence is in its 72-hour grace window.
time() - nyxtrace_update_last_check_timestamp_seconds > 3*86400	The updater has not checked in for three days.

Cardinality

The label set is bounded by construction: tenants, sensor ids, connector instances and closed vocabularies. Nothing on /metrics carries a device, an address, a domain, an incident id or a diagnostic text — those live in the API and the dashboard, where they are role-gated. The number of series therefore grows with the things you configured, never with what the network did, and a scraper can keep it for years without a retention concern of its own.

The same reasoning is why there are no histograms and no per-request timings: the exposition is the state of the appliance, not the performance of the service. A metric you want to alert on and cannot find here is a request for the health report first, because the two always agree.

Relation to the health API and the webhook

/metrics, GET /api/v1/health and the webhook’s health.transition events are three views of one health report — the same function renders all three, so a receiver, a dashboard and a Grafana panel never disagree about a component’s state. Use the health API for an uptime check that wants a status code, the metrics for history and panels, and the webhook for a push the moment a state changes.