Internal ingest

The transport posts source records to the Go core’s internal listener. This is a loopback-only endpoint: a token is never accepted over a non-loopback listener, and a remote sensor endpoint using mutual TLS is future work outside v1.

POST /internal/v1/ingest/events
Content-Type: application/x-ndjson
Authorization: Bearer <source-token>

Identity comes from the token

The deployment-owned registry maps each token to exactly one enabled sensor, one tenant, one set of allowed record kinds and one trust class.

The body contains no tenant and no sensor identity, and payload data can never select either. Tokens are redacted from logs, compared in constant time, and refused outside loopback.

sensors:
  - sensor_id: sensor_example_zeek
    tenant_id: example
    token_env: NDR_SENSOR_EXAMPLE_ZEEK_TOKEN
    allowed_kinds: [zeek.conn, zeek.dns, zeek.ssl, zeek.http, zeek.ssh]
    trust: packet

A flow sensor additionally lists expected_exporters with the canonical sampler addresses. A mismatch is kept as raw unparsed evidence and never becomes a flow observation.

Record shape

One record per non-empty line:

{"kind":"zeek.conn","observed_at":"2026-09-10T11:59:58.220Z","payload":{"uid":"C8Tful1TvM3Zf5x8fl"}}

payload is the untouched decoded source object. The transport unwraps a syslog JSON envelope and does no field normalisation whatsoever — normalisation is the Go core’s job, and keeping that boundary sharp is what makes the raw record trustworthy as evidence.

A request carries 1 to 1,000 lines and at most 5 MiB.

Record kinds

zeek.conn, zeek.dns, zeek.ssl, zeek.http, zeek.ssh, zeek.stats, zeek.capture_loss, suricata.flow, suricata.dns, suricata.tls, suricata.http, suricata.ssh, suricata.alert, suricata.anomaly, suricata.smb, suricata.dhcp, suricata.krb5, netflow.flow, unbound.query.

Event identity

The core derives an event identifier as the SHA-256 of a version marker, the trusted tenant and sensor identifiers, the kind, the normalised UTC observed_at, and the payload JSON encoded canonically: UTF-8, object keys sorted by code point, no insignificant whitespace, non-ASCII unescaped, non-finite numbers rejected.

Two consequences, both intentional:

• The same logical event keeps its identity if the transport retries or changes JSON whitespace or key order.
• Source timestamps and source-native identifiers are in the hash, so repeated legitimate events do not collapse into one merely because their other fields match.

Success

{"data":{"accepted":1,"duplicates":0,"rejected":0,"committed_at":"2026-09-10T12:00:00.100Z"},
 "meta":{"request_id":"req_01J...","generated_at":"2026-09-10T12:00:00.100Z"}}

HTTP 200 only after the store accepted every valid raw row and every rejected-line record. Event identifiers are logically idempotent: raw rows use the event identifier as a replacing key and reads collapse it. A retry may leave physical duplicate parts but cannot duplicate a logical observation, incident, cost entry or rejection record.

Failures

Condition	Response	Meaning
A malformed individual line	200, rejected incremented	Recorded in the durable ledger with a stable hash, a bounded reason, the receipt time and the original line. Valid siblings continue.
Un-normalisable but valid JSON	200	Committed as raw with parse_status=unparsed
Body over 5 MiB	413	Operator-visible
Empty or over-1,000-line batch	422	Operator-visible
Bad token or forbidden kind	401 / 403	Operator-visible
Rate limited	429 with Retry-After	Transport backs off
Store unavailable	503	Transport retries with full-jitter exponential backoff, 1 s base, 60 s cap, while its disk buffer has room

A malformed line returning 200 is deliberate: the alternative makes the transport either discard a whole good batch or loop on it forever. The rejection is durable and counted, so nothing is lost and nothing is hidden.

401, 403, 413 and 422 are your errors to fix. Correct the token or the transport configuration, then replay from the retained source records. The transport does not claim response-aware dead-letter routing, and it blocks before dropping buffered data.

Liveness and readiness

GET /internal/v1/ingest/health/live    # unauthenticated process liveness
GET /internal/v1/ingest/health/ready   # requires a source token; 200 only when the store can commit

Source freshness is not reported here — it is reported by the tenant-scoped health route.

Community ID

Nullable. The core computes the standard direction-independent value with seed 0 when the tuple suffices, records a warning otherwise, and verifies rather than blindly trusting a provided value.

Correlation requires either a non-empty verified Community ID plus temporal overlap, or a normalised tuple plus overlap. An empty identifier never joins records. Sequential same-tuple connections and address-translated tuples stay distinct unless stronger evidence links them.