OPNsense / pfSense quarantine

The opnsense-response connector puts a device into the tenant’s quarantine alias on an OPNsense or pfSense HA pair, standby node first, and takes it out again. It is Pro (licence feature response). Read automatic response first for what a quarantine can and cannot do; this page is the procedure, in the order it has to be done.

The pfSense note: the companion is a static FreeBSD binary and the pfctl forms are the same, but the steps below name OPNsense GUI paths and the os-sudo package; on pfSense the equivalents are the Firewall → Aliases and System → User Manager pages and the sudo package, and the paths /usr/local/bin/sudo, /usr/local/etc/sudoers.d are assumed. There is no pfSense package yet.

Why a companion and not the API

The OPNsense API can edit aliases — with an API key whose scope is the whole firewall configuration. A key that can add an address to an alias can also delete the alias, the rule and the interface. The response executor refuses that trade: it needs exactly five mutations and one health check, so that is what the companion offers, and nothing else is reachable through it.

The companion is an SSH forced command for a dedicated account: the key can run nothing else, gets no PTY and no forwarding.
It reads one signed JSON request from stdin, answers with one signed reply and exits. The signature is HMAC-SHA256 over the exact request bytes under a shared secret; the reply is signed the same way, replayed request ids are refused, and a clock skew beyond 120 seconds is refused.
The operation set is closed: add_member, remove_member, list_members, kill_states, probe_identity, health.
pfctl runs through argument arrays only — no shell — and only in six exact argument forms, which the sudo grant repeats literally and the companion re-checks itself on every call.
Alias names, allowed ranges and protected scopes are in the companion’s own root-owned configuration. The request carries an address and a tenant id, never an alias name; a request for an unknown tenant is refused.
A ledger under /var/db/nyx-companion records which entries the companion added. An entry without a ledger row is foreign and never deleted, and an alias is never flushed.
Every request leaves an audit line in /var/log/auth.log (nyx-companion: op=… tenant=… ok=…).

Before you start

Both nodes’ management addresses (never the CARP VIP: alias contents are not synchronised, each node is changed on its own), and which one is currently CARP BACKUP — that is the standby, done first.
A config backup on each node before the first change: cp -p /conf/config.xml /conf/config.xml.pre-nyx-response-<UTC>.
The tenant’s routed ingress interfaces — the ones a device in the tenant’s ranges enters the firewall through. Never management, DNS, DHCP, CARP or pfsync interfaces.
The complete list of what must never be quarantined: node and VIP management addresses, DNS, DHCP, CARP/pfsync, the appliance, the identity provider, the edge proxy, storage and your own operator access. You will write this list twice — on the appliance and on each firewall — on purpose.
The companion binary nyx-companion_freebsd_amd64 from the release, and the SHA-256 checksum printed in the release notes.

Part 1 — each firewall node, standby first

Everything in this part is done on both nodes, B (standby) first, then A, with the one difference in step 3.

1. Aliases and rules

Firewall → Aliases → Add, for the tenant:

Field	Value
Name	`NYXTRACE_EXAMPLE_Q4` (and `NYXTRACE_EXAMPLE_Q6` if the tenant has IPv6)
Type	External (advanced)
Expire	900 seconds

External means the GUI holds no content and pfctl manages the table; the expiry installs a cron that runs pfctl -qt NAME -T expire 900 every minute. That cron is the fail-open backstop: it removes every entry 900 seconds after insertion whether or not the appliance is alive, and it is the reason the appliance’s ttl must not exceed it.

Then one block rule per routed ingress interface of the tenant, source = the alias, placed above the pass rules, logging on. HA synchronises the aliases and the rules to the other node; the alias contents it does not.

2. The companion binary

install -m 0755 -o root -g wheel nyx-companion_freebsd_amd64 /usr/local/bin/nyx-companion
sha256 /usr/local/bin/nyx-companion

Compare the digest against the release notes before the first call. A companion you did not verify is a root-adjacent binary you did not verify.

3. Configuration

/usr/local/etc/nyx-companion/config.json, mode 0644 root:wheel — the companion never writes it:

{
  "node": "b",
  "secret_file": "/usr/local/etc/nyx-companion/secret",
  "state_dir": "/var/db/nyx-companion",
  "elevate": "sudo",
  "max_ttl_seconds": 900,
  "protected": ["192.0.2.0/24", "198.51.100.0/28"],
  "tenants": {
    "example": {
      "alias4": "NYXTRACE_EXAMPLE_Q4",
      "alias6": "NYXTRACE_EXAMPLE_Q6",
      "allowed": ["198.51.100.0/24", "203.0.113.0/24"],
      "protected": ["198.51.100.254/32"]
    }
  }
}

Key	Meaning
`node`	`b` on the standby node, `a` on the active node — the appliance names the node it expects in every request, and a mismatch is refused (`node_mismatch`)
`max_ttl_seconds`	Equals the alias expiry; a request with a longer TTL is refused (`ttl_out_of_range`)
`protected`	The global list — every node and VIP management address, DNS, DHCP, CARP/pfsync, Nyxtrace, identity provider, edge proxy, storage, operator access. This is the second, independent copy of the appliance’s list
`tenants.<id>`	Alias names, allowed ranges and additional protected addresses per tenant; a request names a tenant id and the companion looks up the rest

On top of protected the companion adds every interface address of the firewall at each start.

4. The shared secret

umask 027
openssl rand -base64 48 > /usr/local/etc/nyx-companion/secret
chown root:nyxresp /usr/local/etc/nyx-companion/secret
chmod 0640 /usr/local/etc/nyx-companion/secret

Generate it once, on the first node, and copy the same bytes to the second node and to the appliance (step 3 of part 2). At least 32 characters. Never generate it twice: the appliance signs every request with one secret and both companions must verify with the same one.

5. The account and its rights

Create the local user nyxresp (System → Access → Users): no shell login, not in wheel, no GUI privileges. Create the account before step 4’s chown.

The companion runs arp, ndp and ifconfig unprivileged and pfctl through sudo -n -- /sbin/pfctl … in exactly six argument forms. The forced command is not the only barrier: an account with a bare NOPASSWD: /sbin/pfctl could run pfctl -F all, so the grant lists the six forms with the alias names literal and nothing wider. Install os-sudo and create /usr/local/etc/sudoers.d/nyx-companion, mode 0440:

Cmnd_Alias NYX_PFCTL = /sbin/pfctl -t NYXTRACE_EXAMPLE_Q4 -T show, \
    /sbin/pfctl -q -t NYXTRACE_EXAMPLE_Q4 -T add *, \
    /sbin/pfctl -q -t NYXTRACE_EXAMPLE_Q4 -T delete *, \
    /sbin/pfctl -si, /sbin/pfctl -ss -vv, /sbin/pfctl -k id -k *
nyxresp ALL=(root) NOPASSWD: NYX_PFCTL

Repeat the three alias lines for NYXTRACE_EXAMPLE_Q6 and for every further tenant alias. The * stands for the one address or state id the companion appends; sudo matches it against the remaining arguments and the companion never passes more than one.

The companion checks the grant itself. On every invocation it runs sudo -n -l; if the output grants ALL or /sbin/pfctl with no or arbitrary arguments, every mutating operation (add_member, remove_member, kill_states) is answered with elevation_too_broad and the reason appears under warnings in health — visible in ndr connectors test … --live — until the rule is narrowed. doas can express only exact argument lists and therefore cannot carry add, delete or -k id -k <id>; with "elevate": "doas" the companion asks doas -C … /sbin/pfctl -F all and refuses to mutate when the answer is permit. Use sudo.

Finally the state directory for the ledger:

mkdir -m 0700 /var/db/nyx-companion && chown nyxresp /var/db/nyx-companion

6. The forced command

In the user’s Authorized keys (System → Access → Users → nyxresp) put the appliance’s public key (generated in part 2, step 1) as one line, options first:

command="/usr/local/bin/nyx-companion",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 AAAA… nyxtrace-response

With that line the key can run nothing but the companion. Make sure the SSH service allows public-key login for the account and is reachable from the appliance address only (a rule on the management interface).

7. The host key pin

cat /etc/ssh/ssh_host_ed25519_key.pub

That line goes into connectors.yaml as standby_host_key or active_host_key. Read it on the node’s console or an existing trusted session — not from a first SSH connection, which is the one a pin is meant to protect.

8. Prove the companion locally

Before the appliance ever calls it, run one health request by hand as the account:

printf '%s' '{"body":{"v":1,"op":"health","tenant":"example","node":"b","request_id":"manual-1","issued_at":'"$(date +%s)"'},"sig":"…"}' \
  | sudo -u nyxresp /usr/local/bin/nyx-companion

The signature is HMAC-SHA256 over nyx-companion-request-v1\n followed by the exact body bytes, keyed with the secret. Computing it by hand is fiddly; the appliance’s ndr connectors test does it for you, so the practical local check is tail /var/log/auth.log after the appliance-side test in part 2 — the audit line is written either way, including for a refusal.

Repeat part 1 on node A with "node": "a".

Part 2 — the appliance

1. A dedicated SSH identity

ssh-keygen -t ed25519 -N '' -C nyxtrace-response -f /etc/ndr/secrets/response-ssh-key
chown root:nyxtrace /etc/ndr/secrets/response-ssh-key
chmod 0640 /etc/ndr/secrets/response-ssh-key
cat /etc/ndr/secrets/response-ssh-key.pub     # → part 1, step 6, on both nodes
rm /etc/ndr/secrets/response-ssh-key.pub

Never reuse this key for anything else. The connector uses no agent, no known_hosts and no PTY; the only host keys it accepts are the two pinned in the configuration.

2. The shared request secret

The same bytes as /usr/local/etc/nyx-companion/secret on both nodes:

install -m 0640 -o root -g nyxtrace /dev/null /etc/ndr/secrets/response-request-secret
read -rs secret && printf '%s' "$secret" > /etc/ndr/secrets/response-request-secret

Both files follow the secret rules: 0640 root:nyxtrace or 0600 nyxtrace, never a value in YAML.

3. Declare the instance

In /etc/ndr/connectors.yaml:

connectors:
  - id: opnsense-response
    instance: fw-example
    tenant: example
    enabled: true
    config:
      pair: fw-example
      standby_host: 192.0.2.3            # node B, never the VIP
      standby_host_key: "ssh-ed25519 AAAA… root@fw-b"
      standby_node: b
      active_host: 192.0.2.2             # node A
      active_host_key: "ssh-ed25519 AAAA… root@fw-a"
      active_node: a
      port: 22
      user: nyxresp
      alias_ipv4: NYXTRACE_EXAMPLE_Q4
      alias_ipv6: NYXTRACE_EXAMPLE_Q6
      policy_revision: 1
      allowed_targets: "198.51.100.0/24, 203.0.113.0/24"
      protected_scopes: "192.0.2.0/24, 198.51.100.0/28"
      ttl: 900s                          # must not exceed the alias expiry
      kill_states: true
      auto_enabled: false                # the automatic mode; see the overview before you flip it
      kill_switch: false
      # learning_until: 2026-09-24       # UTC date; the word none switches the learning phase off
    secrets:
      ssh_key: file:/etc/ndr/secrets/response-ssh-key
      request_secret: file:/etc/ndr/secrets/response-request-secret

The instance name needs at least three characters. Connection fields:

Field	Meaning
`pair`	Label of the pair in memberships and attempts
`standby_host`, `active_host`	Node addresses — the CARP BACKUP node is changed first, the MASTER second
`standby_host_key`, `active_host_key`	One public key line each, from part 1 step 7
`standby_node`, `active_node`	Must equal `node` in the companion configuration on that firewall (`b` / `a`)
`port`, `user`, `ssh_timeout`	`22`, `nyxresp`, `30s` (5 s to 2 min)
`alias_ipv4`, `alias_ipv6`	Display names of the tenant’s aliases, for membership keys; the request never carries them. Leave `alias_ipv6` empty when the tenant has none
`kill_states`	Ask the companion to terminate the target’s routed states once both nodes hold the entry (default `true`)

Policy fields, shared with the UniFi target:

Field	Default	Meaning
`policy_revision`	required	Increase whenever a number below changes; every decision records it
`allowed_targets`	required	Comma-separated ranges a target may lie in; a prefix broader than `/16` or `/48` needs `allow_broad_targets: true`
`protected_scopes`	required	Never quarantined — plus both node addresses and the appliance’s own addresses, added implicitly
`ttl`	`900s`	Membership lifetime, 1 min to 24 h; must not exceed the alias expiry
`auto_enabled`	`false`	Allow the automatic mode at all. With it on and the calibration gate closed, the producer proposes instead of executing
`kill_switch`	`false`	`true` refuses every new execution and stops the producer; releases still run
`proposal_score`, `automatic_score`	`70`, `95`	The decision ladder; `automatic_score` must be above `proposal_score`
`automatic_min_severity`	`high`	Lowest incident severity the automatic mode considers (`medium`, `high`, `critical`); the ladder applies on top
`min_confidence_percent`, `min_evidence_classes`	`90`, `2`	Automatic-mode gates: the evidence confidence and the number of independent classes
`max_manual_active`, `max_automatic_active`	`5`, `1`	Memberships held at once per mode; the automatic cap also counts actions approved but not yet applied
`cooldown`	`60m`	Per-device quiet period after a release before the automatic mode may quarantine it again
`learning_until`	`2026-09-24`	UTC date (`YYYY-MM-DD`) before which nothing runs automatically, whatever the feedback says. The word `none` switches the learning phase off — an empty value is the default, not “off”
`calibration_window_days`	`14`	How many days of feedback the calibration gate counts (1–90)
`calibration_min_labels`	`5`	How many incidents with a true/false-positive verdict the window must hold (0–1000; `0` opens without labels once the learning phase is over)
`calibration_min_precision_percent`	`80`	Share of true positives among those verdicts the automatic mode requires
`allow_broad_targets`	`false`	Waives the `/16` / `/48` floor

4. Validate and test

As the service user:

sudo -u nyxtrace env NDR_CONNECTORS_ENABLED=true NDR_CONNECTORS_PATH=/etc/ndr/connectors.yaml \
  ndr connectors validate
sudo -u nyxtrace env NDR_CONNECTORS_ENABLED=true NDR_CONNECTORS_PATH=/etc/ndr/connectors.yaml \
  ndr connectors test fw-example --live

test --live runs the companion’s health operation on B, then A, and nothing else: no table is touched. It proves the forced command, the host key pin, the SSH key, the request secret, that pf is enabled, the CARP role and the presence of the aliases on each node, printed as one note per node (node b: pf=true carp=BACKUP elevation=sudo aliases=NYXTRACE_EXAMPLE_Q4,…). A warnings entry here is the companion’s own privilege check (elevation_too_broad) telling you the sudoers rule is wider than the six forms.

Result	Where to look
`host key does not match the pin`	Part 1 step 7 — the pin, or you are talking to the VIP
`SSH key refused by the node`	Part 1 step 6 — the authorized key line
`reply could not be verified (request_secret?)`	Part 1 step 4 / part 2 step 2 — the secrets differ
`forced command ended without a reply`	The companion did not run: path, mode or the `command=` option
`refused: unknown_tenant`	Part 1 step 3 — the tenant id in `config.json`
`refused: node_mismatch`	`node` in `config.json` versus `standby_node` / `active_node`
`timed out`	Reachability from the appliance to the node’s management address

Then restart ndr-api and confirm the worker started:

operation=response_worker tenant=example instance=fw-example pair=fw-example nodes=2 ttl=15m0s auto_enabled=false kill_switch=false policy_revision=1 protected=… started=true

Without the licence feature the worker logs mode=release_only instead: an approval stays approved with the reason in the policy decision, while an existing hold stays releasable — see release is never gated.

Part 3 — the canary

In this order, on the pair you can afford to break, before any production pair.

1. Walk it through with the kill switch on. Set kill_switch: true, restart, create a proposal on an incident for one address you own and approve it. The action fails visibly with executor kill switch engaged in the policy decision and no node is contacted. That proves the path from the Response page to the worker without touching the firewall.

2. One address, ten minutes. Set kill_switch: false, restart, and approve a proposal for the same operator-owned test address. Watch the Response page: Executing → Executed (held on every node) with Held until and the executor line held on 2 node(s) until …; b: n/m states killed; a: n/m states killed. On each node, pfctl -t NYXTRACE_EXAMPLE_Q4 -T show lists the address. From that address, new routed connections fail and existing ones survive unless kill_states ended them.

3. Release with readback. Press Release: Release requested → Released (absence confirmed) only once both nodes confirmed the entry is gone. -T show on both nodes is empty of the address.

4. Expiry with Nyxtrace stopped. Approve again, then systemctl stop ndr-api. Watch the cron remove the entry on both nodes on its own within 900 seconds of insertion. Start ndr-api again: the worker reconciles, finds no entry, and ends the action as Expired (deadline passed).

Only after all four: seven days of notification only is the spec’s recommended operating stage before anything more, and auto_enabled stays false throughout. Switching the automatic mode on afterwards has six steps of its own, the first of which is this canary.

Release and expiry

Release: the Release button on the Response page (Response or Administration role), or POST /api/v1/response-actions/{id}/release with If-Match and an Idempotency-Key. Standby first here too. The action is rolled_back only after every node confirmed absence; a node that could not confirm leaves it rolling_back with the reason, and the worker retries.
Release is never licence-gated. What was applied under a licence stays reversible after the licence lapsed; the instance then runs release-only.
Expiry: the appliance releases at the deadline; the firewall cron releases at the alias expiry regardless. A second action on the same address shares the first deadline and never extends it.
Emergency stop: kill_switch: true and a restart refuses every new execution; releases still run.
By hand: pfctl -t NYXTRACE_EXAMPLE_Q4 -T delete <ip> on both nodes — never -T flush, the alias may hold entries an operator added — then Release in the interface so the ledger agrees.

What the interface and the API show

The Response card shows Held until with the fail-open note, the executor’s last explanation and the policy decision. Per-node attempts — one row per companion call with node label, operation, readback_present as null / false / true, states seen and killed and the failure class — are on GET /api/v1/response-actions/{id}; see the endpoints. Node labels are your own (a, b); alias names, node addresses and state ids never leave the server, and only a failure class is stored.

The instance is listed as connector:fw-example on the System page like every other connector.

Uninstall

Disable the instance, remove the authorized key line, pkg delete os-sudo if it was installed for this alone, remove /usr/local/bin/nyx-companion, /usr/local/etc/nyx-companion and /var/db/nyx-companion, and the aliases and rules last, B then A.

What has and has not been proven

Proven, with the code as shipped in 0.31 and against fakes: the executor adds standby then active and reads back both; a partial application is compensated and nothing is held; a compensation that sticks stays reconciling with the membership on record; a first-node failure applies nothing anywhere; release confirms absence on every node; overlapping actions share the deadline; expiry retires actions after the deadline and continues past a failing one; reconciliation reports foreign entries untouched and restores missing ones; limits, cooldown and the kill switch hold; a failed connection persists a failure class and never the endpoint. The companion’s parsing of pfctl -ss -vv is pinned to FreeBSD 14 fixtures, the wire format is fuzzed, and the contract tests show that unknown statuses survive a reader.

Open, and stated here so nobody reads the page above as a completed acceptance:

add / list / delete through the companion on a real node.
The independent 900-second expiry with Nyxtrace stopped.
Reboot: fail-open, then correct reconciliation.
A CARP failover during an active membership.
Existing connections survive, new routed connections fail.
kill_states ends exactly the target’s verified states; a foreign state in the same network stays.
Each companion refusal against alias, URL, command, tenant and state-id injection, individually.

Until the project has run these on an isolated pair, treat the feature as a build you are the first to prove — which is why the canary above starts with the kill switch engaged and one address you own.