Replaying a model file

The forecast counts the verdicts the analyzer stored. It answers “what would fire under the file that was in force when the results were written” and nothing else: change a threshold and the forecast does not move until a day of new results has been written under it. The replay is the other half of that question. Since 0.38 it re-evaluates the stored results of the last days against the configured model file and a candidate file and prints, per model and per step of the verdict pipeline, what each would fire — so a threshold, a role or the baseline can be judged before it goes live, on this network’s own data.

It is the tool for the last days of the learning phase, and it stays useful afterwards whenever a model file is about to change.

Running it

# the configured file alone, last day, every configured tenant
ndr analysis replay

# one tenant, the last three days
ndr analysis replay --tenant example --days 3

# a candidate beside the configured file, machine-readable
ndr analysis replay --tenant example --days 3 --model /etc/ndr/models/candidate.yaml --json

Flag	Meaning
--tenant	one tenant; default every configured tenant, one report each
--days	the window, 1 to 7, ending now; default 1
--model FILE	a candidate model file, any path the ndr user can read; it is replayed beside the configured file, so every run prints the comparison
--json	the report as JSON instead of the table

It runs on the appliance, as the service user, against the local stores. It takes no lease and does not wait for the analyzer; a run beside a busy analyzer is fine. Runtime is that of one streamed ClickHouse read of the window plus a pass over the rows in memory — seconds to a minute on the reference installation for three days.

What it reads

Everything the replay needs is already on the appliance; it reads and computes, it does not collect.

• The stored results of the window from ClickHouse: every model_results row of the file’s models, streamed in observed_at order, deduplicated per result (a re-run of an older window may have written a result twice; the latest wins). The window is bounded at a week and the read at a row cap (three million); a window that hits the cap is reported as truncated in the header, and the numbers then describe the rows that were read, not the window.
• The devices of the tenant from the state store: their first-seen timestamps and display names, and the facts the analyzer resolves a role from — inventory device type and tags, the NetBox role and the Proxmox guest record of any of the device’s addresses, Proxmox tags. The replay resolves roles from the same facts the analyzer uses, with the candidate’s roles section for the candidate and the configured file’s for the configured file, so a role you have just added to the candidate is visible in the candidate’s column only.
• The sharing table: how many devices contacted each destination in the last seven days, from the hourly destination features, read with each file’s own maximum_shared_devices.
• The analyzer’s series records from its engine snapshot — the expected periodic destinations it has learned so far — read without taking the analyzer’s lease. The header says how many it found as known series.

Nothing else. The replay does not open the payload tables, does not compute features, and does not read the incident tables.

Reading the report

One block per tenant and file, the configured file first, then the candidate. The header line carries the tenant, the file, the window and its length, the rows read (with truncated when the cap was hit), the number of devices, how many of them carry which role, how many shared destinations the sharing step knew, how many series records the baseline was seeded with, and the daily cap of the file.

Then one row per model:

Column	Meaning
MODEL, SEVERITY	the model and the severity of the file being replayed — a candidate that changes a severity shows it here
EVALUATED	the stored results of the model in the window
STORED	how many of them carried a positive verdict when they were written — what the forecast counts
WOULD FIRE	how many the file being replayed fires after every step of the pipeline
DEVICES	the distinct devices behind WOULD FIRE
SERIES	the distinct (device, destination pattern) pairs behind WOULD FIRE — one series is one incident before the daily cap, so this is the number the incident list would show
DROPPED threshold/role/shared/baseline	what each step of the pipeline withdrew, in pipeline order: below the thresholds, suppressed by a role, a shared destination, an expected series. A step at zero did nothing on this network.
LEGACY	rows written before 0.38, which carry no interval count, span or host fan-out and are redrawn on what they carry (coefficient of variation and periodicity, or the port count); they are counted, not trusted
TOP DEVICES	the five devices with the most verdicts under the file, by name

The total line sums evaluated, stored and would-fire, lists the verdicts by severity, and gives incidents/day — the series of the window, each counted on the day it first fires — before and after the daily cap. Below the tables the report repeats its own limits, the paragraph under what it cannot show.

How to read the two blocks against each other: STORED is the same in both, because it is history. WOULD FIRE of the configured file should be close to STORED — where it is not, the results were written under an older file or the analyzer has since learned series the results predate. WOULD FIRE of the candidate is the number you are tuning, and the DROPPED columns say which knob is doing the work.

The JSON shape

--json prints an array of {label, report} objects, one per tenant and file; label is current (<path>) or candidate (<path>). The report carries tenant_id, window, days, learning_days, rows, truncated, devices, roles (name → count), shared_destinations, known_series, daily_cap, totals (evaluated, stored_would_fire, would_fire, by_severity, series, incidents_per_day, capped_incidents_per_day), models and note. A model carries model_id, title, severity, detector, evaluation (measurement when the stored reason carries one the pipeline redraws, stored when the stored verdict stands), evaluated, stored_would_fire, would_fire, devices, series, legacy, dropped (below_threshold, role_suppressed, shared_destination, expected_baseline) and top_devices in the forecast’s shape.

The candidate workflow

The model file the appliance uses is the bundle’s copy, /opt/nyxtrace/models/network-core.yaml, named by NDR_MODEL_PATH in ndr.env. An update replaces the bundle’s copy, so a site-specific file lives elsewhere — /etc/ndr/models/ is a reasonable home — and NDR_MODEL_PATH points at it.

1. Copy the file in force to the candidate’s path and edit the candidate: a threshold, a role’s match list, a policy. The knobs are all in the file; there is no threshold in the binary.
2. Replay three days with --model and read the candidate’s block. The target for a network that is about to mature is under ten High verdicts a day; the DROPPED columns say which step got you there and the top devices say who is left.
3. A device that keeps firing one series is one incident that will update itself; leave it. A device with many series is a role or a threshold: a poller that reaches many targets on a timer wants the infrastructure role — tag it infra in the inventory, or give it its NetBox role — and a class of devices that all trip the same threshold wants the threshold moved.
4. Replay again until the numbers are what you want, then make the candidate the file in force (NDR_MODEL_PATH, restart the analyzer) and replay once more without --model: the configured file’s block should now read like the candidate’s did. A role or a threshold takes effect at the next pass; a NetBox role or a Proxmox guest is known once its connector has polled, an inventory tag at once.

ndr analysis forecast --model FILE and the API’s ?model= are the cheap view of the same candidate: they join the forecast’s counts with the candidate’s titles and severities and name the file in model_file, but the counts stay the stored verdicts. Use them to see a candidate’s severities on the dashboard’s numbers; use the replay to see what it fires.

What it cannot show

• Every device is evaluated as mature. The question is what fires when the learning window ends, so the replay does not suppress; a device that is still learning on the day you read the report is counted as if it were not.
• The baseline is seeded with what the analyzer has learned so far. With known series 0 — a fresh installation, or an analyzer that has run the 0.38 code for less than baseline_minimum_days — every periodic series looks new for its first three days and the baseline step drops nothing. That is not a bug in the candidate; it is the analyzer not having had time. Read the report again after the three days.
• Rows written before 0.38 are legacy. They carry no interval count, no span and no host fan-out, so the threshold step can only redraw them on the coefficient of variation and the periodicity, or the port count. The LEGACY column says how many; a window that is mostly legacy is a window worth waiting out.
• Peer-isolation and composite results keep their stored verdict. Their inputs are not in the stored reason, so there is nothing to redraw.
• A truncated window is a sample. Three million rows is a lot; a tenant that hits it in seven days should replay fewer.

Read-only, by construction

The replay writes nothing: no result, no incident, no series record, no snapshot, no notice. It takes no lease, so it never delays the analyzer, and the analyzer never waits for it. It reads the ClickHouse results under the same memory bound as the analysis runs, streamed rather than loaded. Running it on a production appliance in the middle of the day is the intended use.