Proxmox vSensor

Firewall feeds cover routed traffic. To see inside a VLAN you need packets, and on a hypervisor you can get them without any extra hardware: mirror the traffic of selected guests into an isolated bridge that the appliance listens on.

This is the one change in the whole product that touches a hypervisor’s networking, so it is built to be narrow and reversible.

Shape

guest tap ──(tc mirror, ingress + egress)──► ndr veth ──► vmbrMON ──► appliance capture NIC ──► Zeek
                                                            │
                                            no IP, no route, no uplink

An isolated monitor bridge with no address, no route and no uplink.
One veth pair per mirrored access VLAN, injecting into a port whose PVID is that VLAN. The appliance’s port on the bridge is a tagged trunk.
Learning disabled on the injection ports, so known-unicast forwarding cannot decide to skip the sensor.
A separate trunk injection pair for tagged uplink traffic (ingress mode only, see below).

Access taps are untagged at the point where traffic is captured, which is why each access VLAN gets its own PVID port rather than everything being dumped into one bridge and hoped for.

What is mirrored

Since 0.33 the mirror copies both directions of every mirrored guest port (NDR_MIRROR_DIRECTIONS=both, the default): the ingress hook of the tap sees what the guest sends, the egress hook sees what the bridge delivers to the guest. Each frame fires exactly one hook on a given tap, so one tap never yields two copies of the same frame, and every session a mirrored guest takes part in is seen in both directions regardless of where its peer lives — another guest on the same host, a physical device behind the uplink, or the firewall.

What that means for a few common cases:

Traffic	Copies the sensor sees
Outside → mirrored guest	one, at the tap egress
Mirrored guest → outside	one, at the tap ingress
Broadcast from outside	one per mirrored tap
Mirrored guest A → mirrored guest B, same host	two — A’s ingress and B’s egress, like a physical SPAN of two switch ports

The last row is the one to know about. Two mirrored guests on one host that talk to each other arrive at the sensor twice. Zeek treats a duplicate TCP segment as a retransmission; for UDP the orig_pkts/orig_bytes counters count double. Those are distinct segment observations, not globally duplicate packets, and the backend correlates them by tuple, Community ID, time overlap, VLAN, sensor and trust — it never adds overlapping counters. Deduplicating at the tap is a possible later refinement; today it is documented rather than hidden.

The host uplink is not a source in this mode: its ingress copy would be a second copy of every inbound frame already seen at the tap egress.

Why both directions

The previous design mirrored tap ingress only, plus ingress on the host uplink, which yields one observation per layer-2 segment. That is correct when the uplink is part of the mirror. In a scoped pilot — two guest taps added by hand, uplink never added — it fails in a specific and instructive way: the guests’ DNS queries were copied, the resolver’s replies came in through the uplink and were never copied. Zeek saw queries without a single answer, could not complete one transaction, wrote no dns.log, and every TCP session was half-seen. Mirroring the tap egress closes that hole for every peer of the guest, whatever its location, without touching the uplink at all.

The switch

NDR_MIRROR_DIRECTIONS=both      # default since 0.33: tap ingress + tap egress, uplink not a source
NDR_MIRROR_DIRECTIONS=ingress   # the pre-0.33 behaviour: tap ingress + uplink ingress

ingress restores the previous behaviour exactly and is the rollback switch. Any other value is a start error, not a silent fallback. The two modes are the only ones; there is no egress-only mode, because it would see nothing a guest sends.

Explicitly excluded, by construction and in both modes:

Every interface belonging to the sensor guest itself.
The configured sensor addresses and MAC addresses, filtered at the capture hook — the same classic-BPF bytecode runs on both hooks — and again in the backend as defence in depth.
Any source you list in NDR_EXCLUDE_SOURCES — and exclusions are logged on every reconciliation, so an omission stays visible.

A tagged guest trunk is rejected during preflight unless you list it explicitly as excluded. Silently omitting a port you cannot handle correctly is worse than refusing to start.

VLAN tags at the two hooks

Access taps are untagged at both hooks. On ingress the guest sends untagged frames and the bridge assigns the PVID afterwards; on egress the bridge strips the PVID tag before the frame reaches the tc egress hook. So the same untagged bytecode works on both, and each access VLAN is still injected through its own PVID port. The egress half is proved by the namespace test (kernel 6.1, iproute2 6.1) with a foreign protocol 802.1Q counter on the egress hook that stays at zero while the protocol all counter rises; on the Proxmox VE kernel it is assumed rather than measured — and if the assumption failed, a copy that still carried the PVID tag would land in the same VLAN anyway, because the injection port admits only that VID.

Install it carefully

The tooling has three commands and the first two change nothing:

ndr-preflight proxmox          # inspect and report
install-proxmox-mirror         # print the exact actions
rollback-proxmox-mirror        # print the exact undo

Before applying, record the current state: interface detail, bridge VLAN membership, the queueing disciplines and filters on every source, and the guests’ current connectivity. Archive that output.

The apply path backs up the network configuration and the sensor guest’s configuration, adds only a drop-in file, brings up only the monitor bridge, attaches the sensor guest’s passive interface, and enables a reconciliation timer. It never reloads the main bridge and never reloads networking as a whole.

It also reserves exactly one capture filter preference and handle — on both the ingress and the egress hook of every source — and preflight aborts if something else already owns them on either hook. The cleanup path reads its own state file and deletes only those exact filters plus its own veth devices — it deliberately leaves every shared clsact queueing discipline and every foreign filter in place. That is what makes it safe to run on a host that has other tooling installed.

The legacy `ingress` qdisc

Egress mirroring needs a clsact qdisc on the tap. A Proxmox rate= limit on a guest NIC creates a legacy ingress qdisc instead, with a police filter on it. In ingress mode that qdisc is tolerated as before. In both mode ndr-mirror replaces it with clsact only when the only filters on it are its own; otherwise it stops in preflight, before touching anything, and names the tap and the three ways out: NDR_MIRROR_DIRECTIONS=ingress, listing the tap in NDR_EXCLUDE_SOURCES, or migrating that qdisc by hand.

Upgrading an existing mirror to both directions

An install that ran before 0.33 keeps its ingress filters and gains the egress filter on the next reconciliation; the state file (/run/ndr-mirror/filters) grows a fourth column, hooks, and rows without it mean ingress. Install the new script and environment, then run the reconciliation once, the way the host was scoped:

Timer-based install (the reconciliation timer is enabled): ndr-mirror sync. It reconciles every non-sensor guest port of the production bridge and drops the uplink as a source.
Add-based install (timer disabled, sources added by hand with ndr-mirror add): re-run ndr-mirror add <tap> for each source you had added. Do not run sync there — it would widen the scope to every guest port on the bridge.

Both commands are idempotent. The whole sequence for an add-based install with two taps:

# 0. record the state you are about to change
mkdir -p /var/backups/ndr-mirror/$(date -u +%Y%m%dT%H%M%SZ) && cd $_
cp /usr/local/sbin/ndr-mirror ndr-mirror.pre; cp /etc/ndr/ndr-mirror.env ndr-mirror.env.pre
cp /run/ndr-mirror/filters filters.pre
for t in tap100i0 tap101i0; do
  tc -j qdisc show dev $t >$t.qdisc.json
  tc -j filter show dev $t ingress >$t.ingress.json
done

# 1. the new script and the switch (the last entry in the env file wins)
install -m 0755 scripts/ndr-mirror /usr/local/sbin/ndr-mirror
grep -q '^NDR_MIRROR_DIRECTIONS=' /etc/ndr/ndr-mirror.env \
  || printf 'NDR_MIRROR_DIRECTIONS=both\n' >>/etc/ndr/ndr-mirror.env

# 2. reconcile the sources you had — preflight aborts on a foreign filter
ndr-mirror add tap100i0 && ndr-mirror add tap101i0

Record the before and after tc -j filter output for the touched taps.

Verify it

On the host, per mirrored tap:

tc filter show dev tap100i0 ingress   # pref 45000 … bpf … mirred (Egress Mirror to device ndrs<vlan>)
tc filter show dev tap100i0 egress    # the same filter on the egress hook
ndr-mirror status                     # one row per hook; both counters rise under traffic
cat /run/ndr-mirror/filters           # … ingress,egress

On the appliance, within minutes: /var/log/zeek/current/dns.log appears and its rows carry answers (answers, rcode_name); conn.log shows resp_pkts and resp_bytes above zero for sessions of the mirrored guests; and capture_loss.log stays below one percent. If two mirrored guests on the host talk to each other, their frames appear twice — expected, see above.

After applying, compare against the state you recorded, then run a bidirectional packet capture check during both idle and backup load. Capture loss must stay below one percent. Keep an eye on the mirror load on the host during the next backup window: every frame delivered to a mirrored guest is now copied too, and egress copies that came in through the uplink can be GRO superframes of up to 64 KiB.

Start small. Two active guests are enough for a first proof, and the global reconciliation should stay disabled until each additional source has been reviewed.

Rolling back

Set NDR_MIRROR_DIRECTIONS=ingress in /etc/ndr/ndr-mirror.env and run the same reconciliation command again (sync, or the same add per source). It deletes exactly the egress filters it owns, keeps the ingress filters and, on timer installs, restores the uplink as a source.

Going back to the previous script also works, with one rule: run ndr-mirror clear with the new script first. The old script reads the state file’s first three columns and ignores the fourth, so it runs, but it can only delete ingress filters — it would leave the egress filters in place. clear removes both hooks and the ndr* veths and leaves clsact and every foreign filter standing; then copy the old script and environment back from your backup and repeat your add commands.

Limits

A hypervisor mirror does not see:

physical devices talking to each other on a switch,
other hypervisors,
guests you did not select.

Firewall NetFlow and Suricata cover routed traffic, not those blind spots.

One more known gap worth stating: whether a VLAN tag survives into the metadata depends on what the packet-level source writes. If the tag is not in the source record, the flow’s VLAN list is empty even though the capture itself proves the tag was on the wire. The VLAN comes from the source or it comes from nowhere; Nyxtrace does not infer it.

Two things are known and not yet verified on a Proxmox VE kernel: the untagged egress hook described above, and whether a Proxmox rate= police filter (prio 50, on a legacy qdisc) can end classification before the mirror’s filter runs, which would leave the ingress mirror of that tap silently blind. Neither affects a tap without a rate limit.