Skip to content

fix(grafana): stop short writer gaps from paging false alerts (#1695) - #1697

Merged
groupsky merged 2 commits into
masterfrom
fix/1695-nodata-alerts
Oct 11, 2026
Merged

groupsky merged 2 commits into
masterfrom
fix/1695-nodata-alerts

Conversation

@groupsky

@groupsky groupsky commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1695.

Why

A short gap in InfluxDB data (for example a deploy that restarts mqtt-influx) paged false alerts:

What changed

noDataState for the 8 rules (each has a short comment with the reason):

uid new why
boiler-temperature-sensor-failure Alerting, for 30m → 5m absence rule: the count gives no rows when data stops, so no data is the signal
sunseeker-connection-lost Alerting, for 30m → 5m same
boiler-excessive-power-consumption OK high-power rule: no data = no high reading
webhook-lA5nSTt4k (heat pump overload) OK same
webhook-lA5nSTtVk (main power) OK same
webhook-z157Sot4z (water pump) OK same
webhook-rAc7STtVk (dishwasher done) OK reminder: no data = no finished cycle
webhook-XA5nIopVk (laundry done) OK same

For the 2 absence rules, the 30 min query window already does the waiting. With Alerting, for starts only after the window is empty, so for: 30m would page 30 min later than before. With for: 5m they page about 35 min after data stops (before: about 30 min, as a "DATA SOURCE ISSUE"). A short writer gap cannot empty a 30 min window. Both have execErrState: Alerting, so a query error now pages after 5 min instead of 30; an InfluxDB restart takes about 90 s.

Before this change all 8 were Normal on the production host, so Alerting does not page on deploy.

Total Power Outage: for: 1m → for: 5m.

Trade-off: a writer gap of up to 5 min is silent. A real blackout now pages about 4 min later: about 6.5–7.5 min after the meters stop (90 s query window + 5 min for + up to 1 min to the next evaluation), instead of about 2.5–3.5 min. A real outage still pages. I chose this over a "writer is alive" check because it is one line and needs no new data source.

CI check. validate-grafana-alerts now fails when a provisioned rule uses noDataState: NoData, or leaves it unset (Grafana 9.5 rules_types.go defaults a blank value to NoData). There are tests for both, for .json files, and for the real directory.

Test

  • npm test: 155 passed. Before the rule edits, the real-directory test failed and listed the 8 rules.
  • npm run validate: all checks pass.
  • After deploy: restart mqtt-influx and check the Telegram send log. The result goes in the issue.

- 8 rules used noDataState: NoData. Grafana 9.5 sends NoData at once as a
  DATA SOURCE ISSUE page and ignores for (#1682). Threshold and reminder
  rules now use OK; the 2 absence rules use Alerting.
- Total Power Outage: for 1m -> 5m, so a writer gap of up to 5 min is
  silent. A real blackout pages about 4 min later.
- validate-grafana-alerts fails when a provisioned rule uses NoData or
  leaves noDataState unset (Grafana defaults it to NoData).
Their 30 min window already does the waiting. With Alerting, for starts
only after the window is empty, so for: 30m would page 30 min later
than before. Use for: 5m.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant