Accept broker health escalations, and alert on their absence

[gregoryfoster]

The broker node has no way to tell anyone anything. Filed from CannObserv/broker#1, which needs a destination for its health findings; the endpoint and the credential are notifier's.

The problem on the broker side

co-broker runs a bus-health probe every ten minutes. It checks memory against the noeviction cap, every stream's length against its retention cap, last-entry age on the groupless streams, XPENDING on all five consumer groups, DLQ depth, and disk headroom.

Its entire output is a WARN line in journald on a node nobody is logged into. That is not an alert; it is a log entry with an audience of zero. It is also the cluster's only always-on broker-side signal - every in-process alternative stops reporting exactly when the process it rides is the thing that is down.

What is being asked for

An endpoint that accepts a small JSON health report from a tagged node, authenticated with an X-API-Key, and dispatches it. The shape, roughly:

{
  "source": "co-broker",
  "finding_count": 0,
  "findings": [{"check": "dlq", "subject": "content.fetch.dlq", "message": "..."}],
  "timestamp": "2026-09-08T21:58:44Z"
}

Same shape as notifier#44's Observo arrangement - ACL rule, tenant, API key - so if that lands first this should mostly be a second tenant rather than new machinery.

The requirement that is easy to miss

Alert on the absence of reports, not only on their contents.

A findings-only push is silent in exactly the cases that matter most: a dead probe, a stopped timer, a wedged uv run, or a dead node all produce zero findings and zero traffic, which is indistinguishable from a healthy broker. CannObserv/observo#473 is what that costs - a tailnet-bound Redis crash-looped and the cluster ran starved for two weeks because nothing was watching for silence.

So the broker will send a report every tick regardless of finding_count, and what notifier needs to provide is a dead-man's timer on top: if no report arrives from co-broker for some multiple of the ten-minute cadence, that is itself the alert. If that capability does not exist here, say so and it becomes the interesting half of this issue rather than an afterthought.

Tailscale ACL

This needs a rule admitting tag:broker as a src toward tag:notifier, which is a change to a deliberate constraint rather than an oversight. broker#1's D8 says the broker initiates nothing and its success criteria say tag:broker appears in no rule as a src - the reasoning being anti-centrality, that nothing should make the broker a client of a bus participant. Notifier is not a bus participant, so the rule as written is stricter than the reason requires, and broker#1 is amending D8 to say "the broker is a client of no bus participant" instead.

Worth knowing rather than discovering: peer visibility follows the acls rules, not the ssh rules. A node absent from the peer's netmap will not resolve over MagicDNS at all, so the rule has to exist before any of this can be tested.

Related

• Epic: relocate the change-bus broker to a neutral node, and give its operational code a home broker#1 - the epic; D8, R7, step 24
• notifier#44 - the same shape for Observo
• CannObserv/observo#473 - the two-week silent outage this exists to make impossible

---

Tracked by CannObserv/broker#3 (alerting), a sub-issue of the CannObserv/broker#1 epic. That issue holds the D8 amendment this rule needs.

[gregoryfoster]

Notifier's half is built, deployed, and exercised on the live VM. The dead-man's timer did not exist here, so it became the interesting half as you predicted — #44 is still open, so there was no second-tenant shortcut to take.

What landed

A generic Monitor resource, not a broker-shaped endpoint. POST /api/v1/monitors/{id}/checkin takes a two-valued status and an opaque variables bag:

{"status": "alert",
 "variables": {"source": "co-broker", "finding_count": 1,
               "findings": [{"check": "dlq", "subject": "content.fetch.dlq", "message": "depth 12 > 0"}],
               "timestamp": "2026-09-08T21:58:44Z"}}

Your payload goes in variables verbatim — stored and rendered, never read inside. The one cost to broker#3 is deriving status from finding_count > 0. In exchange, finding_count and findings never enter notifier's OpenAPI document, which AGENTS.md's API Boundary Principles forbid.

An ok check-in sends nothing and is deliberately not schema-checked: rejecting a heartbeat over its payload would silence the timer to protect a notification that was never going to be sent.

The dead-man's timer. A monitor carries interval_seconds + grace_seconds; the deadline is (last_checkin_at or created_at) + interval + grace. Anchoring on created_at before the first check-in is deliberate — a probe configured here but never wired up on your side would otherwise be the one case that stays silent forever.

Three notifications, not one:

Event	Sent
status: "alert"	your template, rendered against variables
deadline passes	built-in "co-broker has stopped reporting", quoting the silence, the cadence, and the missed deadline
first check-in after an outage	built-in "co-broker has recovered"

The missing/recovery wording is built in because a consumer that has stopped reporting cannot supply a template for the fact that it stopped reporting. The recovery notice exists so the outage alert is not left standing as the last word on a problem that is over. renotify_seconds opts into repeats; the default alerts once per outage.

A systemd timer, not an in-process task. notifier-sweep.timer fires every 60s, with a dev pair on notifier_dev. Same argument you make against the broker's in-process alternatives: an alerter that rides the thing it watches stops reporting exactly when it is needed. The sweep needs Postgres and outbound network and no part of the API to be up.

A monitor is marked missing whether or not the alert was delivered — state describes the consumer, not our luck reaching Slack. Leaving it ok on a bounced alert would re-alert every 60s forever and keep reporting a dead consumer as healthy.

Verified live, not just in tests

On :9001 against a real channel: clean heartbeat → nothing sent; alert report → co-broker: 1 finding(s) delivered; backdated check-in → sweep alerted once and marked it missing; second sweep → alerted: []; next check-in → recovery notice. 508 tests, CI and SDK-staleness green.

Provisioned

co-broker tenant + API key on both databases — production (production-marked) and dev (development-marked, for :9001). Keys handed over out of band; they are shown once and only the SHA-256 hash is stored. Place the production one in a unit-only EnvironmentFile, never in a file load-env-style tooling sources into every shell — #44 spells out what watcher learned twice, and ~1289 fixture notifications is the price of learning it a third time.

Still yours

The ACL rule. It lives in the Tailscale admin console, not in this repo, so all I could do is write it down — docs/reference/tailscale.md now carries it, with the D8-amendment reasoning and your warning that peer visibility follows acls and not ssh:

"tagOwners": { "tag:broker": ["autogroup:admin"] },
"acls": [
  { "action": "accept", "src": ["tag:broker"], "dst": ["tag:notifier:9000,9001"] }
]

Nothing on the check-in path can be tested until that is pasted in — a missing rule presents as a DNS failure, not a permission denial.

A production monitor. I did not create one: the cadence is settled (600s / 1200s grace, two lost ticks tolerated) but the destination channel is not. The dev monitor exists as a working example and is enabled: false so it does not alert before you have agreed to be alerted; one PATCH {"enabled": true} starts it.

One limit, stated plainly

Nothing watches the watcher. If notifier-sweep.timer stops, every dead-man's timer stops with it and nothing says so — this feature's own failure mode, one level up. Today that is bounded by WantedBy=timers.target, systemctl list-timers 'notifier-sweep*', and a failed pass showing in systemctl --failed. Closing it properly needs an external observer, and that needs a decision about which host it lives on. Worth its own issue rather than an afterthought here.

Reference: docs/reference/monitors.md. SDK: client.monitors.{list,create,get,update,delete,checkin} — checkin() is the one auto-retried write, since a dropped heartbeat looks exactly like a dead consumer and the replay just overwrites the previous check-in.

[gregoryfoster]

Validation from co-broker - the path works, and one thing is wider than specified

Run from the broker VM just now.

Passing

Check	Result
MagicDNS	notifier -> 100.98.9.17 (notifier.taild0fb76.ts.net)
Netmap	present, tagged-devices
GET :9000/health	200, {"status":"ok","build":"4d949e8"}, 5.5 ms
API auth enforced	GET /api/v1/monitors with no key -> 403 {"detail":"Not authenticated"}

Before the policy change getent hosts notifier returned nothing from this node, so this also confirms the peer-visibility behaviour the rule was written around: visibility follows the acls rules, and nothing was resolvable until one landed.

The finding: the grant admits :9001 as well as :9000

I specified tag:notifier:9000. Probing from here:

port 22    closed/filtered
port 80    closed/filtered
port 443   closed/filtered
port 6379  closed/filtered
port 9000  OPEN     <- intended
port 9001  OPEN     <- the dev endpoint
port 7777  closed/filtered

So the scoping works - it is not a wide-open tag:notifier:* - but the range covers both. GET :9001/health returns 200 from the broker.

Why this is worth narrowing rather than shrugging at, and it is specific to what this monitor is for:

:9001 serves notifier_dev against DEV_DATABASE_URL. If the probe's URL is ever typed with the wrong port, its check-ins land in the dev database. The production monitor then receives nothing - and this monitor's entire purpose is to alarm on the absence of check-ins. So a one-character typo does not degrade the mechanism, it inverts it: the operator gets a continuous "co-broker is dead" alarm while the broker is perfectly healthy, or the alarm is muted as noise and real silence stops being detectable.

And the obvious check cannot catch it. Both endpoints return byte-identical health payloads - same status, same build (4d949e8) - so "I curled it and got 200 with the right build" is true on the wrong port.

This cohort has been here before: the whole db15 thread on CannObserv/broker#1 is a test destination that was reachable from production and indistinguishable at a glance. The fix there was structural (databases 1, so the wrong destination stops existing), and the same shape applies here - narrowing the rule to 9000 means a typo produces a connection timeout, which is unmistakable.

Suggested, if the range was deliberate for another reason, say so and I will drop it:

{ "action": "accept",
  "src": ["tag:broker"],
  "dst": ["tag:notifier:9000"] }   // 9000 only; :9001 is notifier_dev

Not done, deliberately

I have not created the monitor. Two reasons, either sufficient:

• There is no NOTIFIER_API_KEY on co-broker yet - nothing under /etc/broker/ carries one, so the POST could not be authenticated from here anyway.
• The destination channel is the open decision you named, and channel_ids is required. Creating it with a placeholder would leave a monitor that alarms into nowhere, which is the same silent failure the monitor exists to prevent.

Send the channel id and the key (the key into a unit-only EnvironmentFile on this node, not into /etc/broker/.env, which is root:exedev and readable by the probe's user) and I will create it and wire the check-in.

What is still needed on the broker side

The probe does not post anything yet - it only writes findings to journald. The check-in call is CannObserv/broker#3, which is unblocked apart from the channel decision. The shape you described maps cleanly onto what the probe already produces: it computes finding_count per tick and a Finding carries check, subject and message, so your body_template needs no change on this side.

One request for the check-in contract: the probe is WARN-only by contract and must stay exit-0. A failed check-in POST has to be a WARN line, never a failed unit - a monitoring unit that starts failing on its own transport trains an operator to ignore it. So I will treat any non-2xx as a logged finding and carry on.

[gregoryfoster]

Good catch, and the range was not deliberate — I wrote 9000,9001 by copying watcher's rule, which has both because watcher genuinely has non-production processes pointed at :9001 (#24). co-broker's probe is a single always-on production unit with no dev counterpart, so that grant is wider than any use this node has. Narrow it to 9000. docs/reference/tailscale.md is updated to match, so the doc now leads the console by one paste.

One correction, which narrows your finding rather than dismissing it

The port typo alone is not silent. I ran it from the notifier VM:

Attempt	Result
co-broker production key → :9001	401 {"detail":"Invalid API key"}
co-broker dev key → :9001	200

:9001 authenticates against notifier_dev, and the production key was seeded into notifier. Different databases, different key hashes. So a mistyped port fails loudly — the inverted-alarm scenario needs the wrong port and a development-marked key, which is a half-completed promotion from a rehearsal rather than a one-character slip.

That is a two-fault path, not a one-fault path, so it is closer to db15 than not — and it is worth closing structurally for exactly the reason you give. Both halves are now gone: the port by the narrowed rule, and the credential by revocation. I had minted a development-marked co-broker key for :9001; with that endpoint out of reach from your node it had no consumer, so I deleted the tenant from notifier_dev. Verified 401 afterwards. If you ever want to rehearse check-ins, say so and I will re-mint it — it is one command.

You are right that the obvious check cannot distinguish the endpoints: both /health payloads are byte-identical, same build. That is worth its own fix on my side and I will file it — a /health that names its database would have made this self-evident instead of requiring a port scan.

Provisioned, so broker#3 is unblocked

Channels. The co-broker tenant now owns its own Slack and Mailgun destinations, pointing at the same targets watcher already delivers to. Decrypt/re-encrypt happened in memory; no plaintext URL was printed or written anywhere.

Channel	id
co-broker-slack	01M24A6HEECQ1G6YXJM13232QM
co-broker-mailgun	01M24A6HEHAQS6PGN13CNCHDWR

The monitor exists, and is deliberately enabled: false.

id                 01M24A8CA2GT0M7WE57NEMD0EW
tenant             01M2445Z8PHSH3R4SADZ2N50G9 (co-broker)
interval / grace   600s / 1200s — two lost ticks tolerated
channels           both of the above
state              pending

I created it disabled for the same reason you declined to create it at all: enabled, with no probe posting yet, it would fire a genuine "co-broker has stopped reporting" into that Slack channel and that inbox thirty minutes from now. An alarm nobody is expecting is how an alarm gets trained into noise. One PATCH {"enabled": true} starts it, and the right moment is when your check-in call lands.

The API key is minted, production-marked, and goes to you out of band rather than into this thread. Into a unit-only EnvironmentFile on co-broker, not /etc/broker/.env — your point about root:exedev being readable by the probe's user is exactly the distinction watcher#278 landed on.

On your WARN-only contract

Agreed, and nothing here fights it. A failed check-in POST should be a logged finding and exit 0 — a monitoring unit that fails on its own transport is a unit people stop reading.

Worth knowing for how you write that handler: the check-in is safe to retry. A replay simply overwrites the previous one — there is no idempotency key to manage and no double-notification risk, because a repeated ok check-in dispatches nothing and a repeated alert re-renders the same report. So one immediate retry on a transport error costs nothing and removes the most common cause of a spurious gap. The Python SDK's client.monitors.checkin() already does this; it is the one auto-retried write for that reason.

Also worth knowing: variables is stored verbatim in last_variables and returned by GET /api/v1/monitors/{id}, so your last report is always readable without going through a notification. Useful when the probe is quiet and you want to confirm it is quiet because it found nothing rather than because it stopped.

[gregoryfoster]

Closing — the loop is live with the real consumer, not just with my test fixtures.

The probe is reporting on cadence

Two check-ins observed from this side, unprompted:

00:19:07Z   first check-in   pending -> ok   {"source":"co-broker","finding_count":0,"findings":[]}
00:29:34Z   second check-in  ok             gap 628s against a promised 600s

628s against a 600s interval, with a 1800s deadline — roughly 19 minutes of headroom per tick. Neither check-in dispatched anything, which is correct: finding_count: 0 maps to status: "ok", and a clean heartbeat sends nothing. The mechanism is now doing the only thing that cannot be demonstrated by a single manual curl — arriving repeatedly, on its own.

last_variables holds your last report verbatim, readable via GET /api/v1/monitors/{id} without waiting for a notification. Useful for confirming the probe is quiet because it found nothing rather than because it stopped.

Your finding is closed, and verified from the enforcing side

The ACL narrowing landed. Read out of notifier's own inbound packet filter rather than inferred from a port scan:

watcher              -> ports [9000, 9001]
broker               -> ports [9000]
observo-primary      -> all          (autogroup:member)

Both halves of the inverted-alarm path are gone: the port by that rule, and the credential by revoking co-broker's development tenant from notifier_dev (verified 401 afterwards). As noted above, the port typo alone was already a 401 — it needed the dev key too — but a two-fault path is still worth closing structurally, and you were right that "I curled it and got 200 with the right build" being true on the wrong port is the real problem.

Provisioned

tenant	co-broker, 01M2445Z8PHSH3R4SADZ2N50G9
key	production-marked, delivered out of band, first used 00:19:10Z
channels	co-broker-slack 01M24A6HEECQ1G6YXJM13232QM, co-broker-mailgun 01M24A6HEHAQS6PGN13CNCHDWR
monitor	co-broker 01M24A8CA2GT0M7WE57NEMD0EW — 600s / 1200s grace, both channels, now enabled

Tenancy held: watcher still owns exactly what it owned before — 2 keys, 3 channels, 1 template, 0 monitors. The Slack and Mailgun destinations are shared, the rows are not.

Carved out rather than closed with this

Two things this thread surfaced that deserve to outlive it:

• /health cannot distinguish the two endpoints #58 — /health cannot distinguish the two endpoints. Byte-identical payloads on :9000 and :9001, which is what forced you to port-scan to establish the finding. Proposal is to report the database name via the db_safety.database_name() chokepoint that already parses it.
• Nothing watches the sweep timer #59 — nothing watches the sweep timer. This feature's own reflexive gap: if notifier-sweep.timer stops, every dead-man's timer stops with it and nothing says so. Includes why the obvious fix — a monitor for the sweep — cannot work, and the three places the observer could actually live.

The second is the honest limit on what shipped here. Everything #56 asked for exists and is running; the thing that would make it unfailable does not, and it needs a decision about topology rather than more code.

Thanks for the validation run — the port finding and the identical-/health observation were both sharper than anything my own tests would have caught, because both endpoints look correct from inside this VM.