Disk sizing

The appliance keeps three things that grow: the product’s own data in ClickHouse, ClickHouse’s diagnostic logs about itself, and the SQLite state database. Only the first is what you installed the appliance for, and until 0.33 it was the smallest of the three.

What grows per day

Measured over the second production week of the reference installation — five feeds, about 4.7 million flow records and 1.3 million DNS records a day. Your numbers scale with your ingest; the shape is what matters.

What	Per day	Bound	Steady state
ndr.raw_events	≈ 1.2 GB	90 days retention	≈ 105 GB
ndr.* read-shaped aggregates (device hourly stats, destinations, contexts)	≈ 1.4 GB	retention	the ndr database settles around 230 GB after 90 days
ndr.feature_hourly, since 0.34	≈ 1.8 million rows (≈ 75,000 an hour on the reference inflow)	90 days TTL	≈ 160 million rows — see below
ndr.canonical_flows_hourly, since 0.35	one row per canonical flow version, the same as canonical_flows_v2 receives	90 days TTL, daily partitions	≈ two fifths of canonical_flows_v2 on disk, table and projection together — see below
ClickHouse system logs, before 0.33	≈ 8.7 GB, of which text_log alone 4.6 GB	none for most of them	unbounded — 69.7 GB after eight days on the reference appliance, three times the product’s data
ClickHouse system logs, since 0.33	bounded	3 days, query_log 7 days	≈ 10–12 GB
SQLite state.sqlite3, before 0.33	≈ 1.7 GB, almost all event receipts	31 days, pruned by one inline DELETE	≈ 52–57 GiB, 183–205 million rows
SQLite state.sqlite3, since 0.33	≈ 1.7 GB	7 days, swept in batches every minute	≈ 11 GiB, about 45 million rows
The transport’s quarantine, before 0.33	13–16 MB, almost all Suricata stats records	none	unbounded
The transport’s quarantine, since 0.33	a handful of rows	none — see below	small
The transport’s dropped-stats ledger, since 0.33	≈ 30 KB	30 days	≈ 1 MB

The middle rows are the finding of that week: the root disk grew by 13 GB a day, and three quarters of it was ClickHouse logging every Trace-level line of every query it ran. Nothing in Nyxtrace reads those tables. They exist for an operator’s diagnosis, and three days is what a diagnosis needs.

The bound

Since 0.33 the bundle ships a ClickHouse configuration fragment, installed to /etc/clickhouse-server/config.d/90-nyxtrace-system-logs.xml by the fresh install and by the bundle update:

Log	Setting
text_log	level warning, TTL 3 days
trace_log, part_log, processors_profile_log, query_thread_log	TTL 3 days
query_log	TTL 7 days — the weekly report reads it
metric_log, asynchronous_metric_log	TTL 3 days

Every log named in the file has a TTL, and the product’s static checks fail when one is added without. A log that is not in the file keeps ClickHouse’s default; if another one grows, it belongs in the file.

Applying it to a running appliance

ClickHouse reads config.d at start, and the system-log settings are not reloaded at runtime. The update installs the file but does not restart ClickHouse — it restarts only the Nyxtrace units and gates on their health — so after updating to 0.33, once:

systemctl restart clickhouse-server

Then check for rotated tables. ClickHouse applies a changed TTL when it opens a log table; if it decides the table’s definition changed, it renames the old table with a numeric suffix (text_log_1, …) and creates a fresh one, and the old table keeps its data — and its size — until you drop it:

# rotated system-log tables, if any
clickhouse-client --query "SHOW TABLES FROM system LIKE '%\\_log\\_%'"
# drop every rotated table listed, one by one, e.g.
clickhouse-client --query "DROP TABLE system.text_log_1"
# what is left, per table
clickhouse-client --query "SELECT table, formatReadableSize(sum(bytes_on_disk)) FROM system.parts WHERE database='system' AND active GROUP BY table ORDER BY sum(bytes_on_disk) DESC"

Whether a TTL change alone triggers the rename is not something to rely on either way — which is why the check is part of the procedure rather than an assumption. An appliance that shows no _N tables needs nothing more.

The system database of an appliance that ran without the bound shrinks to its steady state over the next three days as the TTLs expire the old rows; it does not happen at the restart.

Rolling the bound back

Remove the file from config.d and restart ClickHouse. The TTLs stay on the tables until ClickHouse rotates them at a later start without the configuration; nothing else in the product depends on the file.

The rarity aggregate, feature_hourly

Since 0.34 the analyzer’s rarity phase reads an hour-keyed aggregate instead of the raw observation tables (why). ndr.feature_hourly holds one row per tenant, feature kind, value, device address and hour of receipt — narrow columns and constant-size aggregate states, no payload. Its row count is therefore the number of distinct feature keys seen in an hour, not the number of observations: the hour that failed on the reference installation held 215,000 flows but 76,000 keys (68,000 ports, 8,000 destinations), so the estimate is about 75,000 rows an hour, 1.8 million a day, 160 million at the 90-day TTL for that inflow. Bytes on disk were not measured before the release; expect a small fraction of raw_events. Watch it the same way as the rest of the database:

clickhouse-client --query "
  SELECT formatReadableSize(sum(bytes_on_disk)) AS disk, sum(rows) AS rows,
         min(partition) AS oldest, max(partition) AS newest
  FROM system.parts WHERE database = 'ndr' AND table = 'feature_hourly' AND active"

The TTL is 90 days for every kind — where the raw_events kinds (domains, JA4, user agents) had thirty days of rarity history before — and daily partitions, so an expired day is dropped whole. Migration 012 fills the table from the retained observations when ndr-api first starts after the update (about a minute on the reference installation); it needs no restart of ClickHouse and no grant — see upgrades.

The analyzer’s flow copy, canonical_flows_hourly

Since 0.35 the analyzer’s flow phases read a time-keyed copy of the canonical flows instead of canonical_flows_v2 itself (why). ndr.canonical_flows_hourly holds the fifteen columns the analyzer uses, one row per flow version, ordered by tenant, hour and flow id, plus a projection by_pair that stores the same rows a second time in pair order. So it grows exactly as the source table does — a materialised view copies every insert — and settles at about two fifths of canonical_flows_v2’s disk, table and projection together (the source carries the wider columns; on the reference installation eight days were 24 million rows and 3.4 GiB of these columns before the projection). It uses granules of 1,024 rows instead of 8,192, which costs about 200 MiB of marks in the mark cache and 40 MiB of primary index at ninety days. Watch the table and its projection separately:

clickhouse-client --query "
  SELECT formatReadableSize(sum(bytes_on_disk)) AS disk, sum(rows) AS rows,
         min(partition) AS oldest, max(partition) AS newest, count() AS parts
  FROM system.parts WHERE database = 'ndr' AND table = 'canonical_flows_hourly' AND active"

clickhouse-client --query "
  SELECT formatReadableSize(sum(bytes_on_disk)) AS disk, count() AS parts
  FROM system.projection_parts
  WHERE database = 'ndr' AND table = 'canonical_flows_hourly' AND name = 'by_pair' AND active"

The TTL is 90 days on start_at with daily partitions, so an expired day is dropped whole, and the projection is rebuilt from every merged part (deduplicate_merge_projection_mode = 'rebuild'), so the two counts of parts should match. Migration 013 fills the table from the retained flows when ndr-api first starts after the update (one to three minutes on the reference installation); check the free disk against a fifth of the source table’s bytes_on_disk beforehand — see upgrades.

The SQLite state database

/var/lib/ndr/state.sqlite3 holds checkpoints, leases, the response ledger, the telemetry log — and the ingest receipt ledger, event_receipts: one row per accepted ingest event, kept so that a replayed event is dropped instead of inserted twice. Nothing else reads it; health and freshness use the newest received_at in ClickHouse. On the reference installation it grew by 5.9–6.6 million rows a day, and under the previous 31-day retention it would have settled at 52–57 GiB, pruned by a single inline DELETE inside an ingest transaction that would have held the write lock for seconds.

Since 0.33 the ledger is bounded to seven days (NDR_RECEIPT_RETENTION, default 168h, 24 h to 90 d). Seven days only has to outlast the longest upstream replay — a firewall’s syslog spool of 64 hourly EVE files, Zeek’s hourly rotated logs a lost transport checkpoint would reread, the transport’s own disk buffer — and it does so with days to spare. The sweep is ndr jobs, every minute: 5,000 rows per DELETE, 50 ms between batches, at most 20 s of its tick, walking the existing index. The steady-state backlog of a minute is one batch; a shortened retention is worked off at about a million rows a tick without ever holding the write lock for more than one small DELETE. The ingest path keeps its own inline sweep as a safety net, bounded to 5,000 rows per five minutes. What the sweep measured is on the health report as ingest.receipts and on /metrics as nyxtrace_receipts_* and nyxtrace_state_db_bytes.

db_bytes does not shrink. SQLite reuses freed pages, so the file stops growing once the ledger reaches its retention and stays that size. There is deliberately no VACUUM in the product: on a multi-gigabyte file it holds the database for minutes. If the file has to shrink once — after lowering the retention on an appliance that ran the old one, say — stop everything that writes to it, vacuum as the service user, and start them again:

systemctl stop ndr-api ndr-jobs 'ndr-analyze@*' 'ndr-naming@*'
sudo -u nyxtrace sqlite3 /var/lib/ndr/state.sqlite3 'VACUUM'
systemctl start ndr-api ndr-jobs
systemctl start ndr-analyze@example ndr-naming@example    # per tenant

An appliance updated to 0.33 whose ledger held only a few days sees nothing happen: the first sweeps delete nothing and the ledger stops growing at seven days.

The transport’s quarantine and the dropped-stats ledger

The transport keeps two directories of records it did not forward.

The quarantine holds records it could not parse — see logs. Before 0.33 it was almost entirely Suricata stats records, the engine’s own per-interval counters, which the envelope had no mapping for: 888 of 894 rows on the reference appliance, 13–16 MB a day. Since 0.33 those records are dropped deliberately and counted before they reach the envelope, and every remaining quarantine row carries its reason. What is left is a handful of rows a day. The quarantine directory has no retention of its own; at that rate it is not urgent, and the rows an older appliance accumulated are evidence of the old behaviour and are not touched by the update. Review them, then remove them by hand.

The dropped-stats ledger, /var/lib/vector/dropped/suricata-stats-<day>.ndjson, is one short line per dropped stats record — feed, kind, source timestamp, receipt time, none of the payload — about 30 KB a day. It exists because this build of the transport has no metrics source or sink, so its internal discard counter cannot leave the process; the file is the durable count. wc -l on a day’s file is the number of records dropped that day, and jq -r .feed | sort | uniq -c splits it by firewall. The directory is declared in ndr-tmpfiles.conf with a 30-day age, so systemd’s tmpfiles timer removes older files. Nothing in it is ever replayed.

Watching it

The health page lists Disk against retention as a signal to watch. With the bound in place the number to watch is the ndr database against its 90-day retention; a system database above 15 GB means the bound is not applied — check that the file is in place and that ClickHouse was restarted after the update. For the state database, nyxtrace_state_db_bytes should plateau within seven days of the update; nyxtrace_receipts_oldest_timestamp_seconds older than the retention means the sweep has stopped. feature_hourly should plateau after ninety days; a newest partition that stops moving means the materialised views are not being fed — check that the API is on 0.34 and that migration 012 ran. The same holds for canonical_flows_hourly since 0.35: its newest partition should be today’s, and a projection part count below the table’s part count means merges are not rebuilding the projection.