Analysis runs

The detectors run inside one job, ndr analyze, once per tenant. Until 0.33 it ran hourly with a fixed row budget, and on the reference installation that was not enough: the inflow was larger than the budget, the backlog was structural, and after five days one query outgrew the database’s memory and every run failed. Since 0.33 the job is a loop of bounded passes under a per-query memory bound. 0.33 did not catch up either — its rarity query read the whole observation tables whatever the window — so 0.34 changed what the rarity phase reads and made the slice the job works at a piece of state that survives the run. 0.34 no longer ran out of memory, but it drained only about 0.055 hours of backlog per minute of running — every flow query was a full scan of the flow table’s retention — and only when it ran back to back, which drove the load to 21 on sixteen cores. So 0.35 changed what the flow phases read (a time-keyed copy with a pair projection, no FINAL), moved the timer to every two minutes, and added a catch-up loop that runs while the job is behind and pauses when the appliance is busy. This page is how it runs now and how to read it.

The timer and the live gate

ndr-analyze@<tenant>.timer fires every two minutes (OnUnitActiveSec=2min, two minutes after boot; every minute in 0.33 and 0.34). That does not make the analysis run more often: the job opens a new window only once its checkpoint is an hour old, so a job that is caught up returns in milliseconds, takes no lease and changes nothing. The cadence of the published results is still hourly.

What the short slot buys is catch-up. A job that is behind keeps running passes for up to NDR_ANALYZE_MAX_WALL (110 s) of every slot, instead of one budget an hour. A pass is what one hourly run used to be — lease, engine snapshot, the three phases up to the budget, one atomic commit — and the loop runs passes until the job is live (no open window, checkpoint younger than an hour) or the wall cap stops it. systemd never starts a oneshot unit that is still running, and the lease (the wall cap plus ten minutes) serialises the job across tenants and against the catch-up loop; an invocation that finds the lease held simply exits, which is not an error.

Why two minutes and 110 s. The minute slot of 0.33 with its 50 s wall was a load problem whenever the job was behind: on the reference installation it drove the load average to 21 on sixteen cores, and the operator’s answer was an hourly override. Measured on the same appliance, the two-minute slot with a 110 s wall stays at a load of at most 12 with the API answering in under two seconds — the same share of the clock, half the process starts, and a wall long enough for a pass to drain a window. Since 0.35 the bundle ships exactly that, so an appliance that set NDR_ANALYZE_MAX_WALL=110s or a timer drop-in by hand can drop both.

If you still carry an hourly override from the 0.33 incident, remove it once the analyzer is live — live=true in the journal, lag_seconds under an hour — and let the shipped cadence take over. For a backlog, do not shorten the timer; use the loop.

# back to the shipped two-minute cadence
systemctl revert [email protected]
systemctl daemon-reload
systemctl restart [email protected]

Catching up: the loop

The timer is the default and the installer enables it. Since 0.35 the appliance also ships ndr-analyze-loop@<tenant>.service — ndr analyze --tenant <id> --loop, the same binary in a long-running unit (Type=simple, Restart=always, stop timeout 120 s) — as the operator’s tool for working off a backlog: after an update that left the job hours behind, after a database outage, or whenever lag_seconds keeps growing under the timer.

What it does:

• It runs cycles — each one what a timer invocation is, with its own wall budget, the lease and the persisted slice — back to back while the job is behind, and sleeps a minute between cycles once the job is live or when a timer run holds the lease.
• Before every cycle it reads the one-minute load average from /proc/loadavg. Above NDR_ANALYZE_LOAD_CEILING — three quarters of the CPUs by default, 12 on a sixteen-core appliance — it pauses, re-reading the load every fifteen seconds, and says so on the health report (paused_for_load: true) and in the journal. The pause is the pacing: the loop takes what the appliance has to spare and no more.
• A stop signal ends the pass in flight at its next page and the pass commits — a page is one query of at most NDR_ANALYZE_QUERY_TIMEOUT plus the engine’s work on it, which is what the 120 s stop timeout is for — and the loop clears its heartbeat, so analysis.mode on the health report flips back to timer at once.
• A binary replaced on disk (an update) stops the service the same way, and Restart=always brings the new one up: the updater restarts only the units it names, and the loop unit is not one of them.
• After a failed cycle it backs off thirty seconds and tries again.

Exactly one of the two should be enabled per tenant. The lease keeps the timer and the loop from working the same window, but a timer run can take the lease while the loop is paused for load, which defeats the pacing. Switching is by hand:

# to catch up: the loop instead of the timer
systemctl disable --now [email protected]
systemctl enable --now [email protected]

# watch it: the journal, and `analysis` on the health report
journalctl -fu ndr-analyze-loop@example

# back, once lag_seconds is under an hour (live=true in the journal)
systemctl disable --now [email protected]
systemctl enable --now [email protected]

The loop’s journal carries one line at start, one per cycle, one per pause and one at stop:

tenant=example operation=analysis mode=loop started=true load_ceiling=12.00 wall=1m50s
tenant=example operation=analysis mode=loop cycle=17 committed=3 live=false lease_held=false elapsed=1m48.2s load=9.80 lag=41h12m0s rate_hours_per_hour=3.10 eta=19h8m0s
tenant=example operation=analysis mode=loop paused_for_load=true load=13.40 ceiling=12.00
tenant=example operation=analysis mode=loop paused_for_load=false paused=45s load=11.20
tenant=example operation=analysis mode=loop stopped=signal cycles=212 committed=380

committed counts the passes of the cycle, live says whether it ended caught up, lease_held that it yielded to a timer run; rate_hours_per_hour and eta appear once they are measurable. A cycle that failed reads cycle=N error=… backoff=30s; a replaced binary, stopped=binary_replaced.

If the appliance’s other work suffers while the loop runs, lower the ceiling with NDR_ANALYZE_LOAD_CEILING in /etc/ndr/ndr.env and restart the loop unit — it reads its environment at start. The loop is meant to be switched off again: once the job is live it does nothing the timer would not do, at the cost of a long-running process.

Windows and slices

A backlog of any size is worked off in windows of at most the slice. The hourly, flow and rarity phases page through one window, commit their cursors atomically with the engine snapshot, and the scheduler checkpoint advances once every phase has drained the window. The peer pass reads its day one hour at a time. So the memory of every aggregation is bounded by one slice of inflow, never by how far behind the job is.

The slice has a configured cap, NDR_ANALYZE_SLICE (1 h), and — since 0.34 — a working width the job adapts and remembers; the section below is about that. A new window is also sized by its content before it is opened: one cheap count() of the window’s flows, halved down to the floor until the window holds at most NDR_ANALYZE_SLICE_ROWS (100,000; the hour that failed on the reference installation held 215,000). A quiet hour opens at the full width, a busy one narrower, and the remembered width stays what the bounds taught the job.

Within a window, NDR_ANALYZE_FLOW_BUDGET (1,000,000 rows per phase) is the commit granularity of one pass — how much work is durable before the next snapshot is written — not a throughput limit. A window larger than the budget takes several passes; the window stays open between them and the health report says so.

The flow phases read a time-keyed copy

This is the 0.35 change, and the one that made the throughput. The canonical flow table, canonical_flows_v2, is ordered by (tenant, flow id) and partitioned by a hash of the flow id — the right key for the ingest’s upsert and the flow detail, and no key at all for a time range. So until 0.34 every window query of the hourly and flow phases was a full scan of the retained flows: on the reference installation the hour aggregate of one one-hour window read 24 million rows and 3.4 GiB for a window of 75,000 flows, and the cost of a slice grew with the days of data, not with the slice. The flow phase’s changed flows page was worse — a twenty-minute profile at the two-minute cadence had it at 77 % of the analyzer’s ClickHouse time: 215 calls, 24.5 million rows and 2.9 GiB each, the whole retention twice per call because its subquery (which pairs changed since the checkpoint) grew with the lag.

Since 0.35 the five flow reads of the analyzer — the hour aggregate, the changed-flows page and its subquery, the group count, the window’s row count and the earliest flow — read ndr.canonical_flows_hourly, created by migration 013:

• A copy of the fifteen columns the analyzer uses, the same ReplacingMergeTree(version), ordered by (tenant_id, toStartOfHour(start_at), flow_id), daily partitions, 90 days. ClickHouse turns a range on start_at into a range on the key expression, so a one-hour window opens the granules of that hour and nothing else.
• A projection by_pair, ordered by (tenant_id, src_ip, dst_ip, toStartOfHour(start_at), flow_id), for the changed-flows page and the group count. The page is assembled one batch of eight inventory devices at a time — src_ip IN the batch’s addresses, the pairs the slice’s own versions changed, the 24-hour range — which is a handful of key ranges of the projection: the batch’s granules, not the day’s. Granules are 1,024 rows on this copy instead of the default 8,192, so a small device does not read its neighbours’ rows; the price is about 200 MiB of marks in the mark cache and 40 MiB of primary index at ninety days.
• A materialised view copies every insert into canonical_flows_v2 — the ingest’s upsert inserts a new version of a flow, so a replaced flow arrives as a new version here too — and the migration backfills the retention once.
• No FINAL any more. A query with FINAL never reads a projection, so the versions of a flow are resolved by ORDER BY version DESC LIMIT 1 BY flow_id on the page and by argMax(…, version) per flow in the aggregate, over the pruned rows only.
• The changed-pair subquery is bounded to the slice. It selects the pairs with a version inside the window’s own [lower, upper) — the ingest instants of that slice — inside the page’s own day, instead of every version since the checkpoint. Each slice re-evaluates the pairs it changed, the next slice the ones after it: nothing is skipped, and the cost of a slice no longer grows with the lag.

Measured on the integration corpus — eight days, a hundred devices, 4,000 flows an hour — against the 0.34 query shape on canonical_flows_v2: one page call reads 102,400 rows instead of 1,536,000 (15×), at a peak of 2.6 MiB instead of 10; the window’s nine pages 665,600 instead of 13,824,000 (21×); the hour aggregate 18,432 instead of 1,011,712 (55×). The test that measures it also requires both tables to return identical rows, replaced versions included. On the reference installation — about five million flows a day, a thousand devices — a page call is expected to read some hundred thousand rows instead of 24.5 million.

Why the projection lives on a copy and not on canonical_flows_v2: ClickHouse 26.3 refuses a projection on a ReplacingMergeTree unless the table’s deduplicate_merge_projection_mode is rebuild (or drop, which discards it on every merge). On the source that would be an ALTER of the ingest’s table and a rebuild on every merge of the whole table; on the copy it is a setting of a new table that merges only these columns. What the copy costs: one more write per ingested flow, a second sort of every merged part for the projection, and about two fifths of the source table’s disk (table and projection) — see disk sizing. The backfill is one streaming INSERT SELECT of the retention at four threads (on the reference installation, 24 million rows and 3.4 GiB of these columns, one to three minutes of ndr migrate during which ndr-api is not yet up); it needs no new grant — the projection is part of the CREATE, so there is no ALTER and no MATERIALIZE PROJECTION. The order of the restart it fixes is on upgrades.

Every analysis query also runs with os_thread_priority 10 (NDR_ANALYZE_QUERY_NICE) since 0.35: the analyzer’s four ClickHouse threads yield to the API’s and the ingest’s when a core is contended. ClickHouse’s own priority setting was considered and not used — a query with priority 0, which is every query but the analyzer’s, takes no part in it, and giving the API’s pool a priority would let every ingest insert of the same process pause the analyzer.

The rarity phase reads an hour-keyed aggregate

This is the 0.34 change that made the catch-up possible at all. Until 0.34 the first step of the rarity phase — which feature keys were observed inside the window — read the observation tables flow_observations and raw_events with a predicate on received_at. Nothing in either table is keyed or indexed by receipt time (their keys are the tenant and a hashed event or observation id, so rows are not time-ordered inside a part), and the only thing ClickHouse could prune on was the tenant. So the query read every part and every granule of both tables, FINAL, five times over through the dns/tls/http views with their JSON extraction: on the reference installation 65 million rows and 25 GiB per attempt, for a one-hour window of 76,000 keys, refused at the 4 GiB bound seven times out of seven after about 55 s each. Halving the window could never have helped — the cost did not depend on the window — and each run had spent its wall budget on the refused query before it could retry, so every run started again at one hour.

Since 0.34 the three rarity queries — the window’s candidate keys, their history, and the flow phase’s destination lookup — read one narrow table, ndr.feature_hourly, created by migration 012:

• An AggregatingMergeTree with one row per tenant, feature kind, value, device address and hour of receipt, holding constant-size aggregate states: the first observation, the latest receipt and its evidence, and the three most recent receipt instants (for seen before).
• Two materialised views fill it on every insert: one from flow_observations (kinds destination and port), one from raw_events (kinds domain, ja4 and user_agent, with the same JSON expressions the dns/tls/http views use — those views keep serving every other reader unchanged).
• Daily partitions, ordered by (tenant, kind, value, ip, hour), 90 days of history for every kind — where the raw_events kinds used to have thirty.

The candidate query merges the buckets of the window’s hours and keeps the keys whose latest receipt lies in the window, so it opens only the partitions of its window. The history and destination lookups are primary-key reads of exactly the keys they ask for, bound in chunks of 1,000 keys per query: a key bound as a tuple literal costs about 26 KiB of query memory in the parsed text, so a chunk is 27 MiB where the 0.33 page of 20,000 keys was half a gibibyte before it read a row. No rarity query touches payload_json or FINAL any more. On the integration corpus the candidate query of a one-hour window reads 3 of 22 parts and 5 % of the rows at a peak of 6 MiB.

Two semantic details. History is read up to the end of the window’s last hour, not to its exact upper bound (a bucket cannot be cut inside); a value received again inside that boundary hour is evaluated once, at its later receipt, by the window that owns it. And replayed duplicates that the receipt ledger let through merge into the same states — seen before is the second distinct recent instant, so a duplicate is never its own previous sighting.

Migration 012 runs with the pre-start step of ndr-api, backfills the table from the retained observations in the same file (about a minute on the reference installation, eight days and 20 million observations, four threads, spilling to disk beyond 2 GiB) and needs no new grant; the order of the restart it fixes is on upgrades. Its size on disk is on disk sizing.

The slice is state

Since 0.34 the width the analyzer works at is not configuration but a row in the state database — analysis_slice, one per tenant: the width, how many windows drained in a row at it, and the last query bound the job ran into (when, which, at what width, in ClickHouse’s words).

• A run starts at the remembered width, clamped to [5 min, NDR_ANALYZE_SLICE]. The first run after the update finds none and starts at the configured slice.
• A pass ClickHouse refuses for memory (MEMORY_LIMIT_EXCEEDED, code 241) or cuts at the query timeout (TIMEOUT_EXCEEDED, code 159) halves the window — from the window’s own width when it was narrower than the slice — persists the halved width first, and retries at once, even when the wall budget is already spent. The budget bounds successful work, not the search: the retry gets a budget of its own, one NDR_ANALYZE_MAX_WALL. That is what 0.33 lacked; its refused query ate the budget and the retry never happened.
• The search is bounded twice: at most three halvings per run, and never below the five-minute floor. Either way the run fails with the reason in last_error and on the health report, the analysis component goes degraded, and the next run continues at the width that was reached. The two messages, verbatim: analysis window … exceeds the query bounds (memory_limit) even at the 5m0s floor; raise NDR_ANALYZE_QUERY_MEMORY_BYTES or NDR_ANALYZE_QUERY_TIMEOUT, or narrow the window by hand and analysis window … still exceeds the query bounds (timeout) after 3 halvings in one run; the next run continues at 15m0s.
• It grows back slowly. After six windows drained in a row at the current width it doubles, up to NDR_ANALYZE_SLICE, and it never grows back into a width that was refused less than six hours ago. A live job opens one window an hour, so a refused width is probed again about six windows after the bound was last hit, not every seventh window.

A window that was already open is narrowed where its cursors allow — only a flow phase in progress pins a window, because it pages by device rather than by time — and the rows beyond the new upper bound become the next window’s. The engine’s replay keys and result upserts make the re-read idempotent, exactly as for any failed pass.

Every query is bounded

Every ClickHouse query the analyzer runs carries its own settings:

Setting	Value	Why
max_memory_usage	4 GiB (NDR_ANALYZE_QUERY_MEMORY_BYTES)	One eighth of the reference appliance’s RAM; three such queries beside ingest and the API still fit under the server’s limit
external GROUP BY and ORDER BY	at half of it, 2 GiB	Aggregation and sorting spill to disk instead of failing
max_execution_time	30 s (NDR_ANALYZE_QUERY_TIMEOUT; was 120 s in 0.33)	A query of a slice that fits finishes in seconds; one still running after thirty is filling the memory bound. A query cut by it halves the window exactly like a memory limit
max_threads	4 (NDR_ANALYZE_QUERY_THREADS)	The hash tables of an aggregation scale with the threads; this alone divides the peak of a GROUP BY by four against the stock sixteen, and leaves the other cores to ingest and the API
os_thread_priority	10 (NDR_ANALYZE_QUERY_NICE, since 0.35)	The analyzer’s threads are niced, so they yield to the API’s and the ingest’s when a core is contended
max_query_size	16 MiB	The rarity queries bind their keys as tuple sets in the query text — since 0.34 in chunks of 1,000

The API’s reads share the database’s pool but not these bounds.

The variables

All of them live in /etc/ndr/ndr.env; the installer merges the keys from ndr.env.example, and an empty value keeps the default. The defaults fit a 32 GB appliance.

Variable	Default	Range	What it bounds
NDR_ANALYZE_FLOW_BUDGET	1000000	10,000 – 10,000,000	Rows one pass consumes per phase before it commits
NDR_ANALYZE_MAX_WALL	110s (was 50s until 0.34)	5 s – 6 h	Wall-clock cap of one invocation, sized to the timer’s two-minute slot; the lease is this plus ten minutes; a retry after a halving gets one of its own; one cycle of the loop
NDR_ANALYZE_SLICE	1h	5 m – 24 h	The widest window one pass opens — the cap the remembered width never exceeds
NDR_ANALYZE_SLICE_ROWS	100000	1,000 – 100,000,000	Flows a new window may hold before it is opened narrower (since 0.34)
NDR_ANALYZE_QUERY_MEMORY_BYTES	4294967296 (4 GiB)	64 MiB – 1 TiB	max_memory_usage of every analysis query; spills to disk at half
NDR_ANALYZE_QUERY_TIMEOUT	30s	5 s – 10 min	max_execution_time of every analysis query; a query cut by it halves the window like a memory limit (since 0.34)
NDR_ANALYZE_QUERY_THREADS	4	1 – 64	max_threads of every analysis query (since 0.34)
NDR_ANALYZE_QUERY_NICE	10	1 – 19	os_thread_priority of the threads ClickHouse runs analysis queries on (since 0.35)
NDR_ANALYZE_MODE	timer	timer, loop	How ndr analyze runs when invoked without --loop; the flag wins over it. The shipped units set neither — the timer unit runs a cycle and exits, the loop unit passes --loop (since 0.35)
NDR_ANALYZE_LOAD_CEILING	0.75 × CPUs	0.5 – 1024	One-minute load average above which the loop pauses between cycles (since 0.35)
NDR_RECEIPT_RETENTION	168h (7 days)	24 h – 90 d	Not the analyzer’s: how long the ingest receipt ledger keeps a receipt. It is listed here because 0.33 introduced it with the others

Changing any of the analyzer values takes effect on the next timer invocation — within two minutes — with no restart; the loop unit reads its environment at start, so restart it. Lowering NDR_ANALYZE_SLICE below the remembered width clamps the next run to the new cap; raising it lets the width grow back into the room.

Reading the run

The health report. GET /api/v1/health has an analysis object and an analysis component line since 0.33; the slice fields are 0.34, the mode and throughput fields 0.35:

Field	Meaning
lag_seconds	The age of the checkpoint: how far the published results are behind now. null before the first drained window. A live job hovers between zero and an hour, because a new window opens only once the checkpoint is an hour old — a value under 3,600 is not a backlog, it is the cadence.
checkpoint_at	The checkpoint itself
window_open, window_lower, window_upper	Whether a window is being paged through, and which
budget	Rows per pass and phase, NDR_ANALYZE_FLOW_BUDGET
slice_seconds	The width the job works at — the remembered one, halved after a bound, grown back slowly
slice_configured_seconds	The cap, NDR_ANALYZE_SLICE, it never exceeds
slice_limit_at, slice_limit_class, slice_limit_seconds	The last query bound the job ran into: when, memory_limit or timeout, and the width it tried. All null while it never has
last_run_at, last_run_flows, last_run_rows, last_run_seconds, last_run_drained	The last pass: when, how many flow rows and rows of all phases, how long, whether it drained its window
last_run_slice_seconds	The width the last pass ran under
last_error, last_error_class	Why the last pass failed, or null. last_error_class is memory_limit or timeout for a bound failure and null for any other; last_error itself then starts with the class and the width — memory_limit at slice 1h0m0s: … or timeout at slice 30m0s: … — followed by ClickHouse’s own words
mode	loop while the catch-up loop has written a heartbeat in the last five minutes, timer otherwise (0.35)
cycles, paused_for_load, load	From the loop’s heartbeat: cycles run since the service started, whether it is waiting for the load to fall under the ceiling, and the one-minute load average it last read. null/false/null under the timer
load_ceiling	The configured pause threshold, NDR_ANALYZE_LOAD_CEILING, whatever the mode
backlog_rate_hours_per_hour	The throughput: the widths of the windows drained by the passes of the last hour of wall clock, per hour of wall clock. 1.0 keeps up exactly — an hourly timer draining one hour each hour reads 1.0 too, which is the point: it never catches up — more catches up, and the gaps between timer runs count against it. null until the run log is a minute old
eta_seconds	The time until the job is live at that rate: (lag − 1 h) / (rate − 1). null while the job is live, while the rate is not measurable, or while the rate does not beat the inflow (≤ 1.0)

The component is unavailable while there is no checkpoint (an installation before its first drained window; this does not degrade the appliance), stale — and the appliance degraded — once the checkpoint is more than three hours behind, and degraded when the last pass failed; the admin-only message carries the reason. The same numbers are on /metrics as nyxtrace_analysis_*.

How to read slice_seconds. 3600 with no slice_limit_at is a job that never had to adapt. A smaller value with a recent slice_limit_at is a job that hit a bound and is working at a width that fits; it will try to grow back six hours after the last bound. 300 with last_error_class set is the floor: the window is not the problem — raise NDR_ANALYZE_QUERY_MEMORY_BYTES or NDR_ANALYZE_QUERY_TIMEOUT, or look at which query it is (the query log has it under the analysis settings).

How to read the rate. Under the timer a job that is behind runs 110 s of every 120 s, so its rate is what those passes drain; under the loop it is what the cycles drain between pauses. A rate above 1.0 with an eta_seconds that shrinks between readings is a job catching up. A rate at or under 1.0 while lag_seconds grows is a job that is not — switch to the loop, and if the loop spends its time in paused_for_load, the appliance is the bound, not the analyzer.

The journal. One line per pass, one when a run inherits a width, one per halving, one when the width grows, one when a new window was narrowed by its count, one when the job is caught up, and — under the loop — one per cycle and one per pause (the loop’s lines are above):

tenant=example operation=analysis slice=30m0s persisted=true successes=2 last_limit=memory_limit
tenant=example operation=analysis window=…..… flows=215000 slice_rows=100000 fitted_width=30m0s
tenant=example operation=analysis attempt=1 window=…..… drained=true wall_stopped=false elapsed=4.2s
tenant=example operation=analysis attempt=1 window=…..… memory_limit=retry width=15m0s halvings=1 elapsed=… error=…
tenant=example operation=analysis attempt=1 window=…..… timeout=retry width=15m0s halvings=1 elapsed=… error=…
tenant=example operation=analysis slice=1h0m0s grown=true
tenant=example operation=analysis attempt=1 live=true passes=3 elapsed=12.8s
tenant=example operation=analysis mode=loop cycle=17 committed=3 live=false lease_held=false elapsed=1m48.2s load=9.80 lag=41h12m0s rate_hours_per_hour=3.10 eta=19h8m0s

journalctl -u ndr-analyze@example --since -2h        # the timer
journalctl -u ndr-analyze-loop@example --since -2h   # the loop, while it is enabled

The run log, the slice and the heartbeat. Every pass — committed or failed — is one row of analysis_runs in the state database, kept for seven days; since 0.34 it carries the width and the bound class. The slice itself is one row of analysis_slice, and the loop’s heartbeat one row of analysis_service (0.35), gone once the loop stops:

sqlite3 /var/lib/ndr/state.sqlite3 \
  "SELECT run_at,slice_seconds,error_class,drained,error
   FROM analysis_runs ORDER BY run_at DESC LIMIT 20"

sqlite3 /var/lib/ndr/state.sqlite3 "SELECT * FROM analysis_slice"
sqlite3 /var/lib/ndr/state.sqlite3 "SELECT * FROM analysis_service"

When lag grows

Lag under an hour is the cadence. Lag that keeps growing across several readings is a job that is not catching up, and last_error and the rate say which of three things it is:

• last_error_class is set and slice_seconds is at the floor, 300. Even a five-minute window exceeds NDR_ANALYZE_QUERY_MEMORY_BYTES or NDR_ANALYZE_QUERY_TIMEOUT. Since 0.34 the rarity queries cost what the window holds and since 0.35 the flow queries cost what the window and the device batch hold, so a floor failure on a database with both tables is unusual — find the query in ClickHouse’s query_log under the analysis settings before raising the bound (the default is one eighth of 32 GB, and three analysis queries run beside ingest and the API). Do not set the checkpoint by hand.
• last_error is null, passes drain, backlog_rate_hours_per_hour is at or under 1.0. The timer’s share of the clock — 110 s of every two minutes — is not enough for the inflow plus the backlog, or the timer is still hourly (above). Switch to the loop and watch the rate and eta_seconds; a loop that spends most of its time paused_for_load is telling you the appliance has nothing to spare, and the ceiling, not the analyzer, decides the pace. Check last_run_seconds against last_run_rows too — a pass that takes long for few rows is a slow database, not a small budget.
• last_error is something else, or the passes stop. Read the journal of the unit; a database that is down, or a database without canonical_flows_hourly or feature_hourly because ndr-api was not restarted before the analyzer, shows up there by name. Under the loop such a cycle fails, the loop backs off thirty seconds and tries again, and the run log and the health report carry the reason.

After a fix, expect a slice=… persisted=true line, possibly memory_limit=retry or timeout=retry lines followed by drained=true, then one drained window a pass until live=true; under the loop, cycle=N lines with a falling lag= and, once the rate has settled, an eta=.

Known limits

• The copy keys a flow by the hour of its start_at. A flow whose start moved to another hour between versions has one row per hour it was ever in; the page keeps the latest version across hours where the old FINAL read kept the latest per key. The ingest derives start_at from the same observations each time, so this is a replay edge, not a path.
• The window’s row count (FlowCount, the one that sizes a new window) counts without FINAL and so over-counts replaced versions on purpose; it is a sizing number, not a statistic.
• The rarity phase’s cut-off and the peer pass read as they did in 0.34; the copy changes only the five flow reads.
• feature_hourly keys history by the hour of receipt, so the rarity history of a window ends at the end of the window’s last hour, not at its exact upper bound.