Backup and restore

What holds state

Path	Holds
ClickHouse data directory	Telemetry, raw evidence, model results
The Nyxtrace state directory	Inventory, incidents, workflow, checkpoints, budgets
The transport’s data directory	Disk buffers and quarantined records
The packet source’s log directory	Retained source records, which is what makes replay possible
/etc/ndr	Configuration and all secrets, mode 0600 — including install-id, the installation identity

Secrets exist only in /etc/ndr, in root-owned mode-0600 files. Nothing sensitive is in a YAML file, a database row or a log line.

/etc/ndr/install-id deserves a separate mention: it is the identity the instance registration uses, and on the pro plan it is the licence seat. A restore that loses it produces a new installation in the portal rather than the same one.

Backing up

A whole-machine backup (a VM snapshot with a filesystem freeze, or an equivalent) is the simplest correct approach, because it captures ClickHouse and the state database at the same instant. The two must agree: a backend ledger from 03:00 restored next to ClickHouse data from 02:00 is not a consistent system.

Recommended targets to plan against: a recovery point objective of 24 hours for the machine, and around 5 minutes for telemetry while the transport buffers have capacity. A recovery time objective of a few hours is realistic.

Verifying an archive is not testing a restore

This distinction is the whole reason this page exists.

A streaming read of a backup archive that completes to 100% proves the archive is readable and complete. It proves nothing about whether the system inside it works. Those are different claims, and conflating them is how people discover their backups are useless at the worst possible moment.

What a restore test must prove

Restore into an isolated environment, without production networking, and then compare:

• ClickHouse raw evidence against what it should contain,
• the state ledger against ClickHouse,
• that the two agree after the restore.

A process-only check — “the services came up” — is insufficient. Services coming up on an inconsistent data set is precisely the failure you are testing for.

Record the result before you rely on the backup, and record it again whenever the schema changes.

During a migration or cutover

Keep the previous disk and the previous machine configuration until the complete acceptance test has passed. Take a fresh backup immediately before the cutover and record the machine’s configuration alongside it.

If you later remove the old disk, the surviving archive is your only way back. Do not delete it, and do not treat “I restored it once, months ago” as current evidence.

Rolling back after a cutover

Boot the preserved disk and restore the saved machine configuration. Data created after the cutover stays archived for investigation rather than being merged into the old system — merging two divergent data sets produces a third, wrong one.

Restoring configuration only

If you only need to recover the appliance’s configuration, the mode-0600 files under /etc/ndr are sufficient, together with the bundle. A repeat installer apply preserves existing secret material byte for byte, so re-running the installer over a restored /etc/ndr does not regenerate your tokens and break every source at once.