Upgrades and rollback

An upgrade is a new bundle applied by the same installer, in the same order as the first install. Nothing about the procedure is special-cased for upgrades, which is deliberate: one code path, exercised every time.

Learning that a new version exists

After a successful heartbeat the appliance asks the Nyxtrace server whether a newer version exists and shows the answer — version, channel, release date, whether it is a security release, and a short notice — in the interface. By default that is a notice only: nothing is downloaded and nothing is replaced until you either apply the bundle by hand as below or switch the automatic updater on. With NDR_TELEMETRY=off the heartbeat’s check does not happen and you watch the release page instead. See instance registration and telemetry.

Automatic updates

An appliance can install its own releases, but it never does so on its own initiative: the installer enables neither the policy nor the timer, because an updater that switches itself on would contradict the rule that there is no forced update.

The appliance asks the Nyxtrace server for an Ed25519-signed release manifest, verifies it against /etc/ndr/release-pubkeys.json, downloads the bundle, checks the signed SHA-256, snapshots the current program state, installs the bundle through the installer that came inside it, runs the migrations if the manifest says so, restarts the units that were running and waits up to 120 seconds for the appliance to be healthy. If it is not, the snapshot goes back.

ndr update --policy        # what this appliance will do on its own, and why
ndr update --check         # check once, change nothing; exit 2 means an update is available
ndr update --apply         # install now, within the policy
ndr update --history       # the log of every attempt
ndr update --rollback      # the operator's way back, from the most recent snapshot

The policy is NDR_UPDATE_POLICY in /etc/ndr/ndr.env — off, notify (the default), auto-patch, auto-minor or auto — with NDR_UPDATE_CHANNEL (stable or beta) and an optional NDR_UPDATE_WINDOW (HH:MM-HH:MM, local time). An unknown policy or channel is a start error, not a silent fallback. Then the timer is the operator’s consent:

systemctl enable --now ndr-update.timer ndr-update-now.path

The timer runs daily with up to four hours of randomised delay on top of the window. ndr-update-now.path is how the Update now button on the system view reaches something allowed to install software: the web API cannot install anything and only writes a request file; without the path unit the button is greyed out and the page says so.

ndr-update.service is the one unit in the product that runs as root, because an update replaces the binary and restarts units. Everything dangerous about an update — the signature, the checksum, the expiry, the window, the policy, the rollout — is inside that one program, and --apply --force overrides the window, the policy and the rollout but never the signature, the checksum or the expiry.

What an update check sends: the running version, the channel, and the first 16 hex characters of the SHA-256 of the install id — used only to compute a rollout bucket, and not stored. No licence key, no host names, no counts. The endpoints are identical for Free and Pro, because a security fix behind a licence check would not be defensible.

NDR_UPDATE_POLICY=off switches the whole thing off, independently of NDR_TELEMETRY: someone who does not want telemetry may still want security updates, and the other way round. The update health line then reads unavailable with the note that it is switched off. A rolled_back entry in the history means the update was installed, did not become healthy, and the appliance is running the previous version; rolled_back_unhealthy is the one case that needs a human — and the one outcome the update health line never lets fade.

The release channel is live

Since 2026-09-18 the server side exists. The Nyxtrace server serves the release channel the updater installs from — GET /v1/releases/{channel}/latest, unauthenticated and identical for Free and Pro — and the release keyring is published at https://nyxtrace.com/release-pubkeys.json. It holds one key, id nyx-release-2026a; a rotation adds a key rather than replacing one. The server holds only this public half: it verifies a manifest before the manifest enters the index and never signs one. Signing happens on the publishing host, the only place the private key exists, and a manifest is signed for a bounded time (not_after, 180 days by default, 400 at most) — an expired one is refused by every appliance and re-signed by the publisher, never quietly extended by the server.

An appliance needs the same file at /etc/ndr/release-pubkeys.json (root-owned, world-readable; NDR_UPDATE_KEYRING names another path). Without it — or with the licence keyring in its place — the update health line is degraded with release keyring cannot be read; a keyring that lacks the key a manifest was signed with makes ndr update --check refuse that manifest with unknown_signing_key. The heartbeat’s update report lists the ids the appliance holds as keyring_key_ids, so the fleet’s keyring state is visible where the rest of the updater’s state is.

Two channels, and how a release travels. NDR_UPDATE_CHANNEL is stable (the default) or beta. A release is published to beta first, with a rollout of 0 %: an appliance on beta sees it in ndr update --check as published but outside the rollout — the bucket is computed from the truncated install-id hash, and only ndr update --apply --force installs it on that box — then the rollout is raised from the portal, and the release is promoted to stable once it has held on beta. So under the defaults — channel stable, policy notify, no window, the keyring at its default path — an appliance sees a release only once it is on stable and its bucket is inside the rollout, and it installs nothing until you apply by hand or set an auto-* policy and enable the timer. A withdrawn release leaves latest within a minute and its download answers 410; an appliance that already installed it is unaffected, because there is no remote downgrade.

What this appliance sees, without changing anything:

ndr update --policy    # channel, policy, window, keyring path — and why nothing happens on its own
ndr update --check     # asks the channel once: the running version, the channel, the policy,
                       # and what the channel holds. Exit 0: nothing to install (nothing newer,
                       # or an empty channel). Exit 2: an update is available.
ndr update --history   # every attempt so far, including the ones the policy skipped

--check sends the running version, the channel and the truncated hash of the install id, as described above, and nothing more. Until a release has gone through beta and reached stable, a --check on the default channel reports nothing newer, which is a normal state and not a failure — the update health line stays ok and moves its last_event_at.

The health gate

The updater reads the appliance’s health twice: once in the preflight, before anything is downloaded — updating an already broken appliance makes the diagnosis impossible, because afterwards nobody can say whether the update was the cause — and once after the restart, for up to NDR_UPDATE_HEALTH_GATE_SECONDS (120), where a refusal is a rollback. What it checks is the overall status (ok and degraded pass, unavailable does not — a sensor that has not sent an event in the ninety seconds since the restart is normal) and the clickhouse component, which must be present and not unavailable. The licence is deliberately not gated: no network is not an update failure.

Since 0.36 the gate reads GET /healthz on the loopback bind (NDR_UPDATE_HEALTH_URL, default http://127.0.0.1:8080/healthz). The route is the appliance’s own view of itself, made for exactly this caller: unauthenticated, and instead bound to where the request comes from. It answers only a peer on 127.0.0.1 or ::1 that carries no proxy header — X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Real-IP, Forwarded or Via — and refuses anything else with 403, whatever the address; it is rate limited to five requests a second with a burst of ten (429, Retry-After: 1). What it returns is the overall status, the version and the component list as id, kind and status — the fold of every tenant’s report, the worst status winning — and nothing else: no operator messages, no last_event_at, no counters, none of the license, telemetry, update, ingest or analysis objects, no host paths. HTTP 200 for ok and degraded, 503 for unavailable, like the tenant health API. It is internal: promised to the updater of the same release and to an operator’s curl on the box, not to any client — see the endpoints.

curl -s http://127.0.0.1:8080/healthz | jq .data.status
curl -s -H 'X-Forwarded-For: 192.0.2.1' -o /dev/null -w '%{http_code}\n' \
  http://127.0.0.1:8080/healthz          # 403: a proxy header, whatever the peer

Why it exists: until 0.36 the default was /api/v1/health, a signed, tenant-bound route, and the root updater has neither the secret nor a tenant. The first real ndr update --apply --force on the reference installation stopped in the preflight after three seconds with HTTP 401 from it — fail-closed, nothing downloaded, nothing mutated, failed/preflight in the history — and the update health line then stayed degraded for good, because that row was the updater’s last word (below).

The signed fallback. An operator who points NDR_UPDATE_HEALTH_URL at /api/v1/health anyway gets the probe signed with the service secret ndr update runs with — the first service, the first tenant, role reader, valid for thirty seconds. Any other URL is polled unsigned. That is what carries an appliance whose running API is older than 0.36 and has no /healthz yet (a bundle applied by hand without the restart, or an update started from a 0.36 ndr binary against an older API): set the variable to http://127.0.0.1:8080/api/v1/health in /etc/ndr/ndr.env for that one update and remove it afterwards.

When the probe answers with a status the gate cannot read, ndr update --apply prints two lines beside the outcome — never the body:

health gate: source=signed-api url=http://127.0.0.1:8080/api/v1/health http=401
hint:     that route needs the BFF signature; the gate's default is the loopback /healthz (NDR_UPDATE_HEALTH_URL)

source is healthz, signed-api or other, by the path. The history row keeps the class, the URL (without userinfo, query or fragment) and the status; the hint is printed and not stored.

Symptom	Where to look
failed at preflight, source=signed-api … http=401	NDR_UPDATE_HEALTH_URL points at the signed route and the probe was not signed: a release before 0.36, or a CLI without service secrets. Leave the variable at its default
source=healthz … http=404	The running API is older than 0.36 and has no /healthz. Restart onto the new API first, or use the signed fallback for this one update
source=healthz … http=403	The probe did not arrive from loopback, or something adds a proxy header between the updater and the API. The gate must talk to the bind address directly
source=healthz … http=429	Something else on the box polls the route faster than five a second

The update component after an attempt

A failed attempt degrades the update health line only while it is the updater’s most recent word. Once a later check has succeeded — a check that finished after the attempt; the check inside the failed run itself never counts — the line is stale for 24 hours with the message a later update check succeeded, and ok after that. rolled_back_unhealthy does not fade: that is the one outcome that needs a human. The history (ndr update --history) is never rewritten either way.

Since 0.36 the component also says what the updater last did, without the update object: last_attempt_at, last_attempt_outcome (the history’s vocabulary — available, applied, failed, rolled_back, rolled_back_unhealthy, skipped), last_attempt_stage and, for administrators, last_attempt_error — the row’s detail, which after a gate refusal is the same the health endpoint (signed-api, http://…) answered HTTP 401 the CLI printed. All four are null on an appliance that has never attempted an update. See health monitoring.

The install step and the rollback

The first complete update through the updater — 0.36.0 → 0.36.1 on the reference installation — got through the preflight, the download, the signature, the checksum and the snapshot, and then found two defects: one in the install step, one in the way back. Both are fixed in 0.37. The second attempt ran to applied/done in about seventy seconds — download, Ed25519 signature, SHA-256, snapshot, installer, restart and health gate — and that run is the end-to-end proof the release channel had been waiting for since 0.35.

The installer no longer needs NDR_ARTIFACT_DIR. The updater runs the installer out of the unpacked bundle: <state>/<version>/bundle/bundle/scripts/install-appliance --from-bundle <state>/<version>/bundle --apply. Until 0.37 the script resolved its first-install default — the archive next to it, else ../dist — before it looked at --from-bundle, and under set -e the cd into a directory that does not exist there ended every real updater install in stage install, fail-closed, followed by a rollback. The reference installation got past it by pointing NDR_ARTIFACT_DIR at an existing root-owned path for one run. That workaround is not needed any more and must not stay in ndr.env: since 0.37 the artefact directory is resolved only on the first-install path (a missing one is a plain message instead of a shell error), --from-bundle takes everything from the bundle tree, and ndr update starts the installer with NDR_ARTIFACT_DIR removed from its environment, so the variable can never become load-bearing again — a value still set is ignored. The installer’s own test (make test-ops-installer, needs docker) runs the bundled installer out of an unpacked tree without ../dist and asserts the install, the merged ndr.env, and that every file under /etc/ndr keeps its owner and mode.

A rollback gives every file back to its owner. The snapshot under <state>/<version>/previous carries manifest.json (0600): owner, group, Unix mode and mtime of every entry of /opt/nyxtrace, /usr/local/bin/ndr, /etc/ndr and the ndr-* units, read from the live system when the snapshot is taken. The restore reads the manifest before it writes anything, puts the trees back, re-applies the manifest (the updater runs as root; a symlink is only ever lchowned, never chmoded through; children before parents; /etc/systemd/system itself never) and then checks every entry against it — type, owner, group, mode. A difference is an error that names the path, and the attempt ends rolled_back_unhealthy with that error in ndr update --history — rather than with a service that cannot read its configuration. Until 0.37 the snapshot kept modes and lost owners: the reference installation’s first rollback restored sites.yaml as root:root 0640, ndr-api could not open it, and the health gate refused the restored version — 85 seconds to rolled_back_unhealthy. /etc/ndr is restored as a copy, not as a link tree into the snapshot, so a licence key the service later writes in place does not change the snapshot.

A snapshot without a manifest is refused. A snapshot taken by an updater older than 0.37 carries no manifest, and ndr update --rollback refuses it (the snapshot carries no ownership manifest) instead of repeating the defect; nothing is touched. The way back from such a snapshot is by hand: copy the trees, then restore the owners from the documented set — sites.yaml, connectors.yaml and /etc/ndr/secrets/* root:nyxtrace 0640 (the directory 0750), install-id and license.key nyxtrace:nyxtrace 0600, every *.env root:root 0600 — and run ndr-health before the services start. Do not “fix” a configuration file to 0600 on the way: the service reads it through its group, and a 0600 root:root sites.yaml is exactly the file ndr-api cannot open. The next successful ndr update --apply takes a snapshot with a manifest and prunes the old ones (the two newest stay).

The forward path keeps the same rule: --from-bundle does not touch the operator’s files, and when it merges a key a new release introduced into ndr.env the file keeps the owner and the mode it has. After the first update that ran under 0.37, the check is short:

test -f /var/lib/nyxtrace-update/<version>/previous/manifest.json && echo manifest
stat -c '%U:%G:%a %n' /etc/ndr/sites.yaml /etc/ndr/connectors.yaml \
  /etc/ndr/install-id /etc/ndr/license.key
# root:nyxtrace:640 twice, nyxtrace:nyxtrace:600 twice

The one-time trap: the first update onto 0.37

An update is carried out by the ndr binary that is running — the old one — with the installer from the new bundle. On the first update onto 0.37 that means: the install step is already 0.37 and goes through without NDR_ARTIFACT_DIR, but the snapshot and a rollback are still the old updater’s — without a manifest. If exactly this one update falls back, the way back by hand above applies, owners included; check them before the services start. From the next update on, both halves are 0.37. A snapshot lying around from before (one 0.36 took, with its owner loss) is refused by the new --rollback and pruned by the next successful apply.

The sequence

# 1. Back up first, and verify it is readable.
# 2. Stage and verify the new bundle.
export NDR_ARTIFACT_DIR=/root/nyxtrace-release
/root/nyxtrace-release/install-appliance            # preview, changes nothing
/root/nyxtrace-release/install-appliance --apply    # apply

# 3. Validate before restarting anything.
ndr validate-config

# 4. Restart, then check.
systemctl restart ndr-api ndr-web vector
ndr-health

A repeat apply preserves your secrets byte for byte. It does replace the generated policy files with the exact bundled versions and removes legacy duplicate keys from the older environment files, so a setting you hand-edited into one of those generated files will not survive — put local overrides where the documentation says they belong.

Migrations

Schema migrations are additive and repeatable, and the service unit runs them as a pre-start step before every start. You do not run migrations by hand on a healthy system.

Three properties worth knowing:

• Migration artifacts are validated before any database connection is opened. A missing or empty migration file fails before anything is mutated.
• An upgrade that changes a table’s identity keeps the old table and backfills the new one. The old data is retained rather than dropped, so a rollback has something to go back to.
• The migration ledger is repeatable: running it twice produces the same state, and it records what it did.

Migrations that need a new database grant

Two migrations rewrite views, and each needed a right the appliance’s migration user did not have before. The installer ships the updated ClickHouse users file (users.d/ndr-users.xml) and puts it in place; if you maintain the database users yourself, install the new file before the restart, because the pre-start step runs the migration first and the service does not start on Not enough privileges.

Migration	Statement	Grant it needs
008	Rewrites the two destination views in place with ALTER TABLE … MODIFY QUERY — it creates and drops nothing, so no ingest write falls into a gap	ALTER VIEW MODIFY QUERY ON ndr.*
009	Replaces the dns_events view with DROP VIEW IF EXISTS plus CREATE VIEW, so it admits the records of the Pi-hole and AdGuard Home connectors and gains the additive columns source and blocked; every existing column keeps its name, type and position	DROP VIEW on the database

The 009 statement is a drop and a create rather than MODIFY QUERY because dns_events is an ordinary view and MODIFY QUERY works only on materialised ones; CREATE OR REPLACE VIEW would have needed DROP TABLE on the whole database through its temporary table, which is a far wider grant than the one view. A test creates a user with exactly the grants of the shipped file and runs every migration as it, so a missing grant fails in the test suite and not on the appliance.

Migration 010 (0.28) also rewrites a materialised view, with the same MODIFY QUERY statement as 008, so the grant that 008 needed covers it and no new grant is required. It does need one operator step, described next. Migrations 011 (0.33), 012 (0.34) and 013 (0.35) create a column, tables and materialised views — statements the migration user has held since R16/R17 — and need no new grant either; 012 and 013 each carry a backfill and fix a restart order, described below and further below.

Migration 010 and the one-time backfill

Migration 010 adds a blocked column to the hourly device_context_stats aggregate the device detail reads and rewrites the aggregate’s materialised view in place, so the records of the Pi-hole and AdGuard Home connectors — and the resolver’s verdict on them — reach the detail’s Domains (DNS) panel. Before it, the aggregate did not know those records at all; migration 009 had only widened a view the detail no longer reads. The pre-start step runs 010 in seconds, it is safe to repeat, and it creates and drops nothing, so no ingest write falls into a gap.

What the pre-start step cannot do is rewrite history. A materialised view only sees rows inserted after it was created, so the resolver’s records that already sit in the event table reach the aggregate only through the backfill. Run it once after the restart, while ndr-api keeps serving. The command reads the full configuration, so it runs as the service user with the service environment:

systemd-run --pipe --quiet --collect --uid=nyxtrace \
  -p EnvironmentFile=/etc/ndr/ndr.env -p EnvironmentFile=/etc/ndr/netflow.env \
  /usr/local/bin/ndr backfill-stats --days 30

Thirty days is the retention of the raw event table, so nothing older exists to write. The command works in 24-hour batches, logs one line per batch, may be interrupted and repeated — every row it writes is either replaced on its key or an aggregate state a repeat does not change — and finishes in about 30 seconds per three million events. The flow tables are rewritten too, which is harmless for the same reason.

Without the backfill nothing breaks: a device’s domain list shows the resolver’s names and the blocked by resolver mark only for the time since the upgrade, never a wrong verdict, and the rest of the detail is unaffected. To check that the verdicts have arrived, once the resolver has refused anything:

SELECT uniqExactIfMerge(blocked) FROM ndr.device_context_stats WHERE context = 'dns'

is above zero.

Migration 011 and the order of the restart

Migration 011 (0.33) adds one column to the model-results table, triggered Bool DEFAULT false — the verdict a detector reached, stored next to the score so the learning-phase forecast can count it. It is additive, repeatable, and needs no new grant.

It does fix the order of one restart. The pre-start step of ndr-api runs the migration; the 0.33 analyzer writes the new column by name and would fail against a table that does not have it yet. So after applying the 0.33 bundle, start ndr-api before the hourly analysis timer runs again — in the sequence above that is the normal restart, and the analyzer’s next run finds the column. A 0.32 analyzer against a migrated table keeps working: it writes its values positionally and the new column takes its default.

Updating to 0.33: the restarts, in order

0.33 changes more than the binary, and three of the changes take effect only after something is restarted that the update does not restart itself. After install-appliance --apply and ndr validate-config, in this order:

# 1. The API first: its pre-start step runs migration 011, and the health
#    fields of this release need the new binary.
systemctl restart ndr-api ndr-web

# 2. ClickHouse once, for the system-log bound (config.d is read at start).
systemctl restart clickhouse-server
clickhouse-client --query "SHOW TABLES FROM system LIKE '%\\_log\\_%'"   # drop any _N table listed

# 3. The transport: vector.yaml changed (the Suricata stats drop stages and
#    the null-safe envelope). The installer validated the configuration.
systemctl restart vector

# 4. The timers: the analyzer's unit now fires every minute.
systemctl daemon-reload
systemctl restart [email protected]      # per tenant

ndr-health

Why this order. ndr-api before the analyzer because the 0.33 analyzer writes the triggered column that migration 011 adds (above); restarting the API first means the column exists before the analyzer’s next minute. ClickHouse once because the 0.33 bundle installs a configuration fragment that bounds the database’s own diagnostic logs, which had been growing without limit; ClickHouse reads config.d only at start. Check for rotated _N system tables afterwards — the whole procedure, the numbers behind it and the rollback are on disk sizing. Vector because its configuration changed and the process reads it at start. The analyzer timer last because the unit file changed from hourly to every minute (OnUnitActiveSec=1min) and a timer keeps its old schedule until it is restarted after the daemon-reload; the job itself still opens a window only once its checkpoint is an hour old, so the cadence of the analysis is unchanged — see analysis runs.

The first run after the update finds the continuation the failing runs of an appliance that had fallen behind left open, narrows it if its queries still exceed the memory bound, and works the backlog off at up to fifty seconds a minute: expect memory_limit=retry lines followed by drained=true in the unit’s journal, then live=true. No ClickHouse migration is involved; the run log and ledger tables in SQLite are additive and an older binary ignores them, so the rollback is the previous bundle plus its vector.yaml.

Also since 0.33, the bundle update path installs the threat-intelligence refresh units and script it had previously only placed on a fresh install; see threat intelligence.

Migration 012 and the backfill

Migration 012 (0.34) creates the hour-keyed aggregate the rarity phase of the analyzer reads since 0.34, feature_hourly, with its two materialised views — and backfills it in the same file from the retained rows of flow_observations and raw_events, after the views exist so nothing inserted meanwhile is lost. It is one bounded GROUP BY per source (four threads, spilling to disk beyond 2 GiB), so it fits beside ingest. On the reference installation — eight days, about 20 million observations — it is about a minute of ndr migrate, which runs as the pre-start step of ndr-api; expect the first API start after the update to take that much longer. It needs no new grant (CREATE TABLE, CREATE VIEW and INSERT are held since R16/R17, and migrations 006 and 007 created materialised views the same way), it does not touch the raw tables, and running the file twice is harmless: the states merge. Why the table exists is on analysis runs; what it costs on disk is on disk sizing.

Updating to 0.34: the API before the analyzer timer

0.34 is the analyzer catch-up fix. Unlike 0.33 it restarts nothing but the API and the timers: no ClickHouse restart, no transport restart, no configuration fragment. After install-appliance --apply and ndr validate-config, in this order:

# 1. The API first: its pre-start step runs migration 012 — table, views and
#    the backfill, about a minute — and the health fields of this release
#    need the new binary.
systemctl restart ndr-api ndr-web

# 2. Check that the table is filled before the analyzer's next run.
clickhouse-client --query "SELECT count(), min(hour), max(hour) FROM ndr.feature_hourly"

# 3. The timers, after the API is up.
systemctl daemon-reload
systemctl restart [email protected]      # per tenant

ndr-health

Why this order. ndr-api before the analyzer timer because the 0.34 analyzer reads feature_hourly and fails against a database that does not have it yet — the failure is named in the unit’s journal, and nothing is lost, but every such run is a wasted minute. Restarting the API first means the table exists and is backfilled before the analyzer’s next run. The SQLite state gains the analysis_slice table and two columns on analysis_runs, both additive and applied when the binary opens the database.

The timer cadence after the update. The shipped unit fires every minute (every two minutes since 0.35). An appliance whose timer was overridden to hourly during the 0.33 incident keeps that override until the analyzer is live — live=true in the journal, lag_seconds under an hour — and then goes back to the minute cadence; the procedure and the reason are on analysis runs. The first run after the update finds no remembered slice, starts at one hour, narrows the window by its row count (on the reference installation to thirty minutes) and drains it: expect window=… fitted_width=30m0s and drained=true per pass. With the hourly timer a 75-hour backlog is roughly twenty runs; with the minute timer well under an hour.

Rollback is the previous bundle. feature_hourly and its views stay and keep filling harmlessly — the 0.33 rarity queries read the raw tables as before — and the 0.33 binary ignores the new SQLite table and columns. Dropping the table and the views needs DROP TABLE, which the appliance’s migration user does not hold, so it is a default-user decision for when the space is needed, not part of the rollback.

Migration 013 and the flow copy

Migration 013 (0.35) creates the time-keyed copy of the canonical flows the analyzer’s flow phases read since 0.35, canonical_flows_hourly — the fifteen columns the analyzer uses, ordered by tenant, hour and flow id, with a projection by_pair ordered by tenant, source, destination, hour and flow id — its materialised view from canonical_flows_v2, and backfills it in the same file: one streaming INSERT SELECT of the retained flows at four threads, after the view exists so nothing inserted meanwhile is lost, without FINAL (every version is copied and the engine deduplicates). On the reference installation — eight days, 24 million rows, 3.4 GiB of these columns — that is one to three minutes of ndr migrate, the pre-start step of ndr-api; the API is not reachable for that long, as with 012. It needs no new grant (CREATE TABLE, CREATE VIEW and INSERT; the projection is part of the CREATE, so there is no ALTER and no MATERIALIZE PROJECTION), it changes nothing in canonical_flows_v2, and running the file twice is harmless for the same reason a replayed version is. Why the copy exists and what it costs at run time is on analysis runs; its size — about two fifths of canonical_flows_v2, table and projection — is on disk sizing. Check the free disk before the update:

df -h /var/lib/clickhouse
clickhouse-client --query "
  SELECT formatReadableSize(sum(bytes_on_disk))
  FROM system.parts WHERE database = 'ndr' AND table = 'canonical_flows_v2' AND active"

Updating to 0.35: the API before the analyzer, then the loop

0.35 is the analyzer throughput release. Like 0.34 it restarts nothing but the API and the timer: no ClickHouse restart, no transport restart. It is also the first release published to the release channel as a signed bundle for the updater: the updater runs ndr migrate before it restarts anything, so the order below takes care of itself, and only the loop switch at the end stays a hand step. On the reference installation the updater’s first attempt at it stopped in the preflight — the health gate of a release before 0.36 read a signed route — and the bundle was applied by hand; the first update that went through the updater end to end was 0.36.0 → 0.36.1. By hand, after install-appliance --apply and ndr validate-config, in this order:

# 1. The API first: its pre-start step runs migration 013 — table, projection,
#    view and the backfill, one to three minutes — and the health fields of
#    this release need the new binary.
systemctl restart ndr-api ndr-web

# 2. Check that the copy is filled before the analyzer's next run: the two
#    counts should be about equal, and every active part should carry the
#    projection.
clickhouse-client --query "SELECT count() FROM ndr.canonical_flows_hourly FINAL"
clickhouse-client --query "SELECT count() FROM ndr.canonical_flows_v2 FINAL"
clickhouse-client --query "
  SELECT (SELECT count() FROM system.projection_parts
           WHERE table = 'canonical_flows_hourly' AND name = 'by_pair' AND active) AS projection_parts,
         (SELECT count() FROM system.parts
           WHERE table = 'canonical_flows_hourly' AND active) AS parts"

# 3. The timer, after the API is up: the unit now fires every two minutes
#    with a 110 s wall, and a timer keeps its old schedule until restarted.
systemctl daemon-reload
systemctl restart [email protected]      # per tenant

ndr-health

Why this order. ndr-api before the analyzer because the 0.35 analyzer reads canonical_flows_hourly and fails against a database that does not have it yet — named in the unit’s journal and on the health report, nothing lost, but every such run is a wasted slot. The timer after the daemon-reload because its cadence changed from one minute to two and its default wall from 50 s to 110 s; an appliance that had set NDR_ANALYZE_MAX_WALL=110s or a timer drop-in by hand can drop both — the bundle now says the same. The SQLite state gains the analysis_service table (the loop’s heartbeat), additive and applied when the binary opens the database.

Then the backlog. The first run after the update should show drained=true in the journal, and in ClickHouse’s query_log the hour aggregate now reads canonical_flows_hourly with read_rows of the order of an hour of flows instead of the whole retention, the changed-flows page with projections = ['ndr.canonical_flows_hourly.by_pair']. If the job is hours behind, switch to the catch-up loop — systemctl disable --now [email protected] && systemctl enable --now [email protected] — watch analysis.backlog_rate_hours_per_hour and eta_seconds on the health report, and switch back to the timer once lag_seconds is under an hour. The loop, its load ceiling and the rule that exactly one of the two is enabled are on analysis runs.

Rollback is the previous bundle. canonical_flows_hourly and its view stay and keep filling harmlessly — the 0.34 flow queries read canonical_flows_v2 as before — and the 0.34 binary ignores the new SQLite table. Disable the loop unit by hand if it was enabled and re-enable the timer. Dropping the copy and the view needs DROP TABLE, which the appliance’s migration user does not hold, so it is a default-user decision for when the space is needed, not part of the rollback.

Updating to 0.36 and 0.37: the API first, nothing else

Neither release carries a ClickHouse migration, a unit change or a transport change: install-appliance --apply, ndr validate-config, systemctl restart ndr-api ndr-web, ndr-health — or, from 0.36 on, the updater. ndr-api first because /healthz comes with the API binary and the updater of the new release asks for it. 0.37 adds two things to the SQLite state — the kind column on incidents (default detection) and the response_canaries table — both additive, applied when the binary opens the database, ignored by an older binary. After the 0.37 restart the log of an appliance with a response target shows no response_reconcile … invalid node attempt line any more and, within the first ten minutes, operation=connector instance=<instance> kind=action probe=ok; see the target’s health line. The rollback is the previous bundle; an open canary proposal would then have to be rejected by hand and its incident set to resolved.

Updating to 0.39: the feed keyring, two timers, nothing to migrate

0.39 carries no ClickHouse migration and no new database grant. The bundle brings the two-privilege ndr-threat-intel.service, the new ndr-model-packs.{service,timer}, the state directories /var/lib/ndr/threat-intel and /var/lib/ndr/model-packs (created by the installer, owned by the service user) and four environment keys with defaults — NDR_FEED_KEYRING, NDR_THREAT_INTEL_STATE_DIR, NDR_MODEL_PACK_DIR, NDR_CONTENT_CHANNEL — that the updater’s environment merge carries into ndr.env. install-appliance --apply, ndr validate-config, systemctl restart ndr-api ndr-web, ndr-health; or the updater.

On a free installation that is all: the next threat-intel refresh runs as the service user, builds the file from the abuse.ch sources without Spamhaus DROP — the cidrs list of the file becomes empty, entries drops by the DROP prefixes, which is expected — and the models and analyst health rows appear as unavailable. On a Pro installation two things follow, in this order: the licence server has to be on 0.4.0 before the appliance sends its first heartbeat with the analyst usage report (an older server answers 400, the appliance drops the object and retries — harmless, but a refusal a day until then), and the feed keyring goes into place as /etc/ndr/feed-pubkeys.json (root, 0644) before the first refresh, or the pull logs the keyring that verifies feeds is not usable and the local sources apply. Then systemctl start ndr-threat-intel.service and read the threat-intel row: source: keyserver with a bundle_version and signing_key_id: nyx-feed-2026a. systemctl enable --now ndr-model-packs.timer is a separate decision; see model packs.

Rollback is the previous bundle; the new units and directories do not disturb it. The threat-intel rollback stays the 0.33 one — timer off, previous YAML back, no restart —, a pack rollback is a current.json edit, and a quota does not need one: an older binary ignores the token fields.

A replaced view that went missing is recreated

Migration 009 is a drop and a create, which leaves a window — an interrupted run, or a create refused for a missing grant — in which the view is gone, whether or not the ledger records the migration. Since 0.27 ndr migrate closes that window itself: for every migration file that drops and recreates a view, it checks after the run whether the view still exists and recreates a missing one from the CREATE VIEW statement of that file. A healthy system sees no change. While a view is missing, the device detail answers with a warning rather than an error, and it is whole again after the next start.

Version pinning

The bundle pins its own component versions: the database, Grafana, the packet source, the flow collector and the transport. Third-party artifacts are verified against recorded checksums before install, and the database and Grafana are accepted only as exact candidate versions from preconfigured signed repositories.

The transport is a privately built runtime, shipped inside the checksum-covered archive and never downloaded by the appliance. The reason is specific: stock builds reported source progress before the asynchronous disk write had completed, which under low volume could lose buffered records on shutdown. The fix waits for write completion before advancing progress.

If you rebuild that runtime, repeat its regression gate — ten consecutive runs of the buffer test and ten of the end-to-end topology test under the race detector. A version bump and one clean run is not equivalent proof; that is how the original problem survived a release.

Rolling back

The previous release directory is kept, so a bad web or binary release can be reverted by pointing back at it and restarting.

For anything that touched the schema or the host configuration, the way back is the backup: boot the preserved disk or restore the pre-upgrade machine configuration, and keep the newer data archived rather than merging it.

A removal of Nyxtrace touches only what Nyxtrace owns — its units, its configuration, its own firewall table, its own capture resources. It never reloads management networking, and it never removes firewall, DNS or proxy objects you created yourself. Those are yours; the rollback does not assume it may delete them.

Firewall-side changes

If an upgrade changes the firewall feed configuration, treat it as a firewall change and not as an application upgrade: configuration backup first, written rollback, standby node before the active one, and verification of the cluster state, packet filter, interfaces, DNS, management access and one real cross-VLAN path after each step. See firewall feeds.