Logs and evidence

The journal

Everything on the appliance is a systemd service, so everything is in the journal:

journalctl -u ndr-api -f          # the Go core
journalctl -u ndr-web -f          # the interface and its backend-for-frontend
journalctl -u vector -f           # transport
journalctl -u zeek -f             # packet source
journalctl -u clickhouse-server   # the store
journalctl -u ndr-analyze@<tenant> --since -2h   # the hourly analysis
journalctl -u ndr-jobs --since -1h               # queued analyst work

Secrets are redacted from logs by construction: source tokens are never logged, they are compared in constant time, and the connector framework strips every configured secret value from error texts, health lines and the whole unwrap chain — including from a wrapped cause that carried one.

Capacity lines

The analysis and identity jobs log explicit capacity notices when they hit a per-run budget:

Notice	Means
adapter_hourly_backlog	More completed-hour aggregates than one run processes
adapter_flow_backlog	More flow groups than one run processes
adapter_rarity_backlog	More rarity rows than one run processes
adapter_flow_group_capacity	A single destination group larger than the chunk limit

A backlog is not a failure. The run commits what it processed, the cursor advances only that far, and the next hourly run continues. A backlog that never shrinks is the thing to investigate.

Quarantine and the rejection ledger

Two different places, for two different failures.

The transport’s local quarantine holds records it could not parse at all. They stay on disk rather than being dropped.

The rejection ledger in the database holds individual records the backend refused, each with a stable hash, a bounded reason, the receipt time and the original line — scoped to the tenant and sensor. Their valid siblings in the same batch were accepted normally.

Both are counted in health as unparsed_last_5m and rejected_last_5m.

Replaying after a misconfiguration

Some ingest failures are yours to fix: a wrong token, a record kind the sensor is not allowed to send, a payload over the size limit. Those return an operator-visible error rather than being silently accepted.

After fixing the configuration, replay from the retained source records — the packet source’s own logs, and the retained hourly firewall files. That is what the retention windows are for.

The flow collector is the exception: it writes no raw file, so a flow gap can only be recovered if the exporter itself can resend it. Plan accordingly.

Evidence, which is not a log

The thing you usually actually want is not a log line but the evidence behind a finding, and that lives in the product rather than on disk. An incident links to its source records; a flow has an evidence endpoint listing every observation of it with its sensor, trust class, receipt time, normalisation warnings and directional counters.

Observation evidence stays resolvable for 90 days, past the raw payload’s own retention. When an identifier cannot be resolved it is reported as unresolved — never quietly relabelled as something else.

What you will not find in a log

Payload contents. Nyxtrace stores metadata, and the browser API never returns raw payload JSON. If you need packet contents, you need a capture tool, and that is deliberately a separate decision with separate consequences.