Chat notifications

The four chat sinks — telegram, discord, slack and matrix — deliver the same events as the webhook as a message a person reads on a phone: the severity and title, the device names, the first line of the summary, and a link into the dashboard. They are Pro: the licence feature is connector.<id>, which the licence service maps onto the coarse connectors grant every tier carries. Without it the instance is listed as unlicensed on the system view, keeps its bookmark and sends nothing — nothing is lost while a key is missing.

Eight things reach a chat: a new incident, an incident whose score rose, an incident an operator closed, a health component that changed state, and — since 0.32 — the four outcomes of the automatic response mode: a proposal the appliance held for a person, an automatic quarantine applied, one that could not be applied, and one released. Evidence, model results and the analyst narrative stay behind the link, as for the webhook.

All four are one family behind the same outbound event path, so everything the webhook page says about the subscription (events, min_severity), the bounds (timeout, retry_delay, max_pending), the at-least-once bookmark, the counters and reset-cursor applies unchanged. This page is what differs per platform.

What every platform has in common

https only, no redirects. A platform behind a proxy or a self-hosted homeserver with its own certificate is pinned like every other endpoint (tls_fingerprint_sha256, read with the openssl s_client line of UniFi Network); api.telegram.org, discord.com, hooks.slack.com, slack.com and a public homeserver are trusted through the host trust store. There is no allow_insecure_http here, and a redirect is refused with unexpected_shape rather than followed.

The credential is a secret file, never a value in connectors.yaml, following the secret rules:

umask 077
printf '%s' '<credential>' > /etc/ndr/secrets/<platform>-…
chgrp nyxtrace /etc/ndr/secrets/<platform>-…
chmod 0640 /etc/ndr/secrets/<platform>-…

A Discord or Slack webhook URL is the credential — its path carries the token — so it goes into a secret file too.

The subscription and the severity floor are the webhook’s fields with the webhook’s meaning:

Field	Meaning
`events`	Comma-separated: `incident.created`, `incident.escalated`, `incident.closed`, `health.transition`, `response.proposed`, `response.executed`, `response.failed`, `response.released`. Empty means every type. An explicit list written before 0.32 does not include the response events until you add them.
`min_severity`	Holds back incident and response events below the floor (`info`, `low`, `medium`, `high`, `critical`); default `info`. A response event carries its incident’s severity. A health transition and the test message are never held back by it.
`timeout`	Per HTTP request, default 5 s, 1 s..30 s.
`retry_delay`	Wait before the second attempt on a `5xx` or a transport failure, default 1 s; doubled for the third, each plus jitter.
`max_pending`	How far the instance may fall behind while the platform is unreachable, default 1000, 10..10000. Beyond it the oldest events are dropped and counted.
`public_url`	The dashboard origin for the link in every message. Defaults to `NDR_PUBLIC_URL` of the service; without either the message carries no link.

min_severity: medium is the sensible floor for a channel people are paged by; a channel that is a log can take info.

What the platform answers decides what happens to the queue, exactly as for the webhook:

Answer	What the appliance does
`2xx`	Acknowledged. The bookmark moves; the event is never sent again.
`401`	Held as `auth` — the token or webhook URL is wrong. Fix the secret; nothing is lost meanwhile.
`403`	Held as `forbidden` — the bot is not in the chat, the webhook may not post.
`429` (also `408`, `425`)	Held as `rate_limited`, for the wait the platform asked for: Telegram’s `parameters.retry_after`, Discord’s `retry_after`, Matrix’s `retry_after_ms`, otherwise the `Retry-After` header — bounded to 1 s..15 min and used as the floor of the backoff. The delivery is not retried inside the call.
`5xx`, timeout, connection failure	Retried up to three times within the delivery, then held for the backoff.
Any other `4xx`	Rejected for good: a chat that does not exist (Telegram `400`), a deleted webhook (Discord `404`), an archived channel (Slack `410`) — counted as `rejected`, skipped, the queue behind it moves on. Watch that counter after a change on the platform side.

Pacing. Each platform’s own message rate is declared as the instance’s limit and kept by the framework before a request leaves: Telegram 20 per minute (burst 5), Discord 30 per minute (burst 5), Slack 60 per minute (burst 5), Matrix 12 per minute (burst 10, Synapse’s default). A burst of incidents queues rather than trips the platform’s limiter; a 429 that arrives anyway is honoured as above.

A hostile title cannot page a channel. An incident title is produced from network observations — a hostname is whatever the device claimed — and a chat platform interprets markup. Every string is therefore cleaned first (valid UTF-8, every control, format and line-break character replaced by a space), bounded in characters (title 300, summary line 600, device names 64 each, at most three), and then escaped for the platform it goes to: HTML entities for Telegram and Matrix, a backslash before every Markdown character for Discord, &, < and > as entities for Slack — which is what stops <!channel> and <@U…> from being parsed. Discord’s mentions are switched off wholesale with an empty allowed_mentions, so @everyone in a title pings nobody whatever the escaping. Escaping and cutting happen in one pass, so a message never has an entity cut in half and never exceeds the platform’s cap; where text was cut, … marks the place.

What a message looks like

The same summary is rendered for every platform, with the platform’s markup on top — bold labels and an italic footer in HTML, an embed on Discord, Block Kit on Slack. In its plain form (the Matrix body, the Slack notification fallback):

🟠 New incident: Periodic TLS destination
Severity: HIGH
Score: 82
Status: open
Device: nas-01
One device reached the same external endpoint every 300 s for 6 hours.
https://nyxtrace.example.invalid/incidents/inc_01J...
— Nyxtrace 0.30.0, instance telegram

The glyph is the severity (🔴 critical, 🟠 high, 🟡 medium, 🔵 low, ⚪ info) or, for a health transition, the new state (🟢 ok, 🟡 degraded, 🔴 unavailable). Discord colours the embed the same way.
The headline is New incident, Incident escalated (the score reads 70 → 82), Incident closed (the status reads resolved or false_positive), Health transition or Test message.
A response event has its own glyph and headline: ✋ Response proposal awaits approval (response.proposed), 🛑 Automatic response applied (response.executed), ⚠️ Automatic response not applied (response.failed), 🟢 Automatic response released (response.released). The message carries the incident’s title, severity and device as above, the target address, the identity proof grade, the executor’s or the gate’s explanation, and the lease deadline while one is held — the same fields as the webhook’s response object. No alias name, no node address.
Device names are resolved at delivery time: the appliance looks up the current display name of every device on the incident — a manual name first, then the resolved name, then the inventory’s own — at the moment it renders the message, so a week-old event carries today’s name. A device the inventory does not know is shown by its id; more than three read as nas-01, printer, cam-02 (+4 more). The same names travel in the webhook envelope as incident.device_names.
The prose is the first line of the incident summary only. The analyst narrative and the evidence are behind the link.
A health transition reads Component, Kind, State (ok → unavailable) and Reason — the same admin-level diagnostic the webhook carries, which can name a host-local address. A chat the appliance posts into should be as trusted as an administrator of the appliance.
The test message (ndr connectors test <instance> --live) says so: 🧪 Test message: Nyxtrace chat notifications are configured, with the instance and the version, and no incident behind it.

1. Create the bot

Message @BotFather, send /newbot, keep the token it prints (<digits>:<secret>). The bot needs no rights beyond posting: leave privacy mode on (it never reads the chat) and give it no admin rights in a group. A channel is the exception — a bot posts into a channel only as an admin, so grant it Post messages and nothing else.

2. Add it to the chat and find the chat id

Add the bot to the group or channel. The id is numeric — negative for a group, -100… for a supergroup or channel — and is shown by @userinfobot or @getidsbot, or by https://api.telegram.org/bot<token>/getUpdates after one message in the chat. A public channel can be named as @channelusername instead. A topic of a forum group takes its message_thread_id in addition.

3. Secret and instance

umask 077
printf '%s' '<token from @BotFather>' > /etc/ndr/secrets/telegram-bot-token
chgrp nyxtrace /etc/ndr/secrets/telegram-bot-token
chmod 0640 /etc/ndr/secrets/telegram-bot-token

connectors:
  - id: telegram
    instance: telegram
    tenant: example
    enabled: true
    config:
      chat_id: "-1001234567890"     # numeric, or a public @channelusername
      # message_thread_id: 42       # a topic of a forum group
      # silent: true                # deliver without a notification sound
      # api_url: https://bot-api.example.invalid   # a self-hosted Bot API server only
      # tls_fingerprint_sha256: "<sha256 of that server's leaf>"
      # events: incident.created, incident.escalated, incident.closed, health.transition
      min_severity: medium
    secrets:
      bot_token: file:/etc/ndr/secrets/telegram-bot-token

The token’s shape (<id>:<secret>) is checked by ndr connectors validate, so a pasted username is a start error and not a 404 at the first incident.

What arrives

One sendMessage per event in HTML parse mode, link previews off, disable_notification when silent is set, cut at Telegram’s 4096 characters with a visible marker. Telegram allows about twenty messages a minute into one group; the instance is paced to that and honours a 429 with the retry_after the Bot API names. A 401 is a wrong token (auth, held), a 403 a bot that is not in the chat or was blocked (forbidden, held), a 400 a chat that does not exist (rejected).

Discord

1. Create the webhook

In the channel: Edit Channel → Integrations → Webhooks → New Webhook, name it, Copy Webhook URL. No bot, no application and no OAuth scope is needed — and the webhook can post only into the channel it was created for. The URL carries the token and is the whole credential.

2. Secret and instance

umask 077
printf '%s' 'https://discord.com/api/webhooks/<id>/<token>' > /etc/ndr/secrets/discord-webhook-url
chgrp nyxtrace /etc/ndr/secrets/discord-webhook-url
chmod 0640 /etc/ndr/secrets/discord-webhook-url

connectors:
  - id: discord
    instance: discord
    tenant: example
    enabled: true
    config:
      username: Nyxtrace            # optional: the name the message is posted under
      # tls_fingerprint_sha256: "<only for a proxy in front of Discord>"
      min_severity: medium
    secrets:
      webhook_url: file:/etc/ndr/secrets/discord-webhook-url

What arrives

One embed per event, coloured by severity or by the new health state: the headline and title as the embed title (which Discord renders without Markdown, so it is only cut, at 256 characters), the fields inline, the summary line as description, the incident link on the title, the event time as the embed timestamp and the instance in the footer. allowed_mentions is empty. Discord names a fractional retry_after on a 429, which is honoured. A deleted webhook answers 404, and every event is counted as rejected until the URL in the secret file is replaced — a growing rejected counter after someone tidied the channel’s integrations is that.

Slack

Two modes, one app. Create the app at api.slack.com/apps → Create New App → From scratch.

Incoming webhook (mode: webhook, the default): enable Incoming Webhooks, Add New Webhook to Workspace, pick the channel, copy the URL. The URL is the credential and posts only into the channel it was created for. This mode needs no scope beyond the webhook.

Web API (mode: api): give the app the bot scope chat:write (and chat:write.public if it should post into a public channel it has not joined), install it, copy the Bot User OAuth Token (xoxb-…), invite the bot into the channel (/invite @Nyxtrace) and set channel to the channel id (C…, from the channel details) or #name. Nothing else — no channels:read, no history.

Secret and instance

umask 077
printf '%s' 'https://hooks.slack.com/services/<T…>/<B…>/<secret>' > /etc/ndr/secrets/slack-webhook-url
chgrp nyxtrace /etc/ndr/secrets/slack-webhook-url
chmod 0640 /etc/ndr/secrets/slack-webhook-url
# api mode instead, same owner and mode:
# printf '%s' 'xoxb-…' > /etc/ndr/secrets/slack-bot-token

connectors:
  - id: slack
    instance: slack
    tenant: example
    enabled: true
    config:
      mode: webhook                 # webhook | api
      min_severity: medium
      # mode: api
      # channel: C0123456789        # api mode: channel id or #name; the bot has to be a member
      # api_url: https://slack.example.invalid/api   # api mode, a proxy only
    secrets:
      webhook_url: file:/etc/ndr/secrets/slack-webhook-url
      # bot_token: file:/etc/ndr/secrets/slack-bot-token

validate refuses a webhook instance without webhook_url, an api instance without bot_token or channel, and a token with whitespace in it.

What arrives

A Block Kit message: a header (a plain_text object, never parsed, 150 characters), the fields as mrkdwn, the summary line, and a context line with Open in Nyxtrace and the instance; text carries a plain fallback for the notification banner, and link unfurling is off. &, < and > are escaped in every mrkdwn string, which is what keeps <!channel> and <@U…> from being parsed.

In webhook mode the status is the verdict (400 invalid_payload and 404 channel_not_found are rejected, 403 action_prohibited is held, 410 channel_is_archived is rejected). In api mode the Web API answers a refused message with HTTP 200 and ok: false, so the connector reads the error word: invalid_auth, token_revoked and their kin are auth (held), channel_not_found, not_in_channel, is_archived and missing_scope are forbidden (held), msg_too_long and invalid_blocks are rejected, ratelimited honours Retry-After. An error word the connector does not know holds the event as unexpected_shape rather than dropping it.

Matrix

1. A bot account and its token

Register a user for the appliance on your homeserver, or on any homeserver the room federates with. Log it in once and take the access token — from a client’s Settings → Help & About → Access Token, or with the login API:

curl -s -X POST https://matrix.example.invalid/_matrix/client/v3/login \
  -d '{"type":"m.login.password","identifier":{"type":"m.id.user","user":"nyxtrace"},"password":"…","initial_device_display_name":"Nyxtrace NDR"}'

The account needs nothing but membership of the room: invite it and accept the invite from its session (or, on Synapse, join it with the admin API). No power level, no moderator rights.

2. The room id

!…:server from Room Settings → Advanced in Element — the id, not the #alias. An alias is deliberately not resolved: that would be a second request with its own failure modes, and every client shows the id in the room settings. A room id containing anything that could open a path segment is refused by validate.

Use a room without encryption. The sink speaks the plain client-server API and holds no device keys, so it cannot encrypt; an encrypted room is not supported.

3. Secret and instance

A self-hosted homeserver with its own certificate is pinned like every other endpoint; a homeserver behind a path prefix keeps it in homeserver_url.

umask 077
printf '%s' '<access token>' > /etc/ndr/secrets/matrix-access-token
chgrp nyxtrace /etc/ndr/secrets/matrix-access-token
chmod 0640 /etc/ndr/secrets/matrix-access-token

connectors:
  - id: matrix
    instance: matrix
    tenant: example
    enabled: true
    config:
      homeserver_url: https://matrix.example.invalid
      room_id: "!abcdefghijklmnop:matrix.example.invalid"
      # tls_fingerprint_sha256: "<sha256 of the homeserver's leaf, self-signed only>"
      msgtype: m.notice             # m.notice (a bot, no sound) | m.text (pages like a person)
      min_severity: medium
    secrets:
      access_token: file:/etc/ndr/secrets/matrix-access-token

What arrives

One m.room.message per event with an HTML formatted_body (org.matrix.custom.html) and a plain body every client can show. The message is sent with PUT …/send/m.room.message/{txnId} and the event id is the transaction id, so the at-least-once repeat after a lost acknowledgement is the same transaction and the homeserver posts it once — Matrix is the one platform where a retry never shows twice. The test message uses a fresh id, so a second test is a second message. A 429 (M_LIMIT_EXCEEDED) carries retry_after_ms, which is honoured; a 401 (M_UNKNOWN_TOKEN) is a wrong or expired token (auth, held), a 403 (M_FORBIDDEN) a bot that is not joined to the room (forbidden, held). The instance is paced to Synapse’s default message rate (0.2 per second, burst 10).

Prove it and switch it on

For every platform: validate as the service user, send one test message — a real message in the chat, marked as a test, with no incident behind it — and restart:

sudo -u nyxtrace env NDR_CONNECTORS_ENABLED=true ndr connectors validate
sudo -u nyxtrace env NDR_CONNECTORS_ENABLED=true ndr connectors test telegram --live
systemctl restart ndr-api

A failure names its class: auth, forbidden, tls (wrong or missing pin), unreachable, timeout, rate_limited, rejected, unexpected_shape (a redirect, or a Slack error word the connector does not know). An instance without the licence feature answers unlicensed and sends nothing.

From then on the instance appears as connector:<instance> in GET /api/v1/health and on the system view with the webhook’s five counters — delivered, failed, rejected, dropped, pending — and the same numbers on the metrics endpoint. It never becomes stale. Events from before the instance existed are not delivered; to send everything still held again, reset the bookmark with the service stopped.

Known limits

One chat per instance. Two channels, two rooms or two platforms are two instances, each with its own bookmark, backoff and counters.
A deleted Discord webhook rejects every event until the secret file holds a new URL; the queue does not wait for it.
Matrix: no alias resolution, no encrypted rooms, no threads.
Investigating and reopening are not events; nor is new evidence at an unchanged score.
The payload shapes follow the platforms’ public API documentation and are proven against recorded answers in the project’s test harness; they have not yet been exercised against a live workspace or homeserver by the project.