Repository navigation
fix(grafana): stop short writer gaps from paging false alerts (#1695) - #1697
Merged
Merged
Conversation
- 8 rules used noDataState: NoData. Grafana 9.5 sends NoData at once as a DATA SOURCE ISSUE page and ignores for (#1682). Threshold and reminder rules now use OK; the 2 absence rules use Alerting. - Total Power Outage: for 1m -> 5m, so a writer gap of up to 5 min is silent. A real blackout pages about 4 min later. - validate-grafana-alerts fails when a provisioned rule uses NoData or leaves noDataState unset (Grafana defaults it to NoData).
Their 30 min window already does the waiting. With Alerting, for starts only after the window is empty, so for: 30m would page 30 min later than before. Use for: 5m.
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1695.
Why
A short gap in InfluxDB data (for example a deploy that restarts
mqtt-influx) paged false alerts:noDataState: NoData. Grafana 9.5 sends NoData at once as a "DATA SOURCE ISSUE" page and does not wait forfor(routy: deploy restarting influxdb + grafana sends false "DATA SOURCE ISSUE" pages — grafana evaluates before influxdb is up #1682).webhook-totalout01(Total Power Outage) uses no data as its blackout signal, withfor: 1m.current_powerhas one writer,mqtt-influx, so a writer gap of about 2.5 min looked like a blackout.What changed
noDataStatefor the 8 rules (each has a short comment with the reason):Alerting,for30m → 5mAlerting,for30m → 5mOKOKOKOKOKOKFor the 2 absence rules, the 30 min query window already does the waiting. With
Alerting,forstarts only after the window is empty, sofor: 30mwould page 30 min later than before. Withfor: 5mthey page about 35 min after data stops (before: about 30 min, as a "DATA SOURCE ISSUE"). A short writer gap cannot empty a 30 min window. Both haveexecErrState: Alerting, so a query error now pages after 5 min instead of 30; an InfluxDB restart takes about 90 s.Before this change all 8 were Normal on the production host, so
Alertingdoes not page on deploy.Total Power Outage:
for: 1m→for: 5m.Trade-off: a writer gap of up to 5 min is silent. A real blackout now pages about 4 min later: about 6.5–7.5 min after the meters stop (90 s query window + 5 min
for+ up to 1 min to the next evaluation), instead of about 2.5–3.5 min. A real outage still pages. I chose this over a "writer is alive" check because it is one line and needs no new data source.CI check.
validate-grafana-alertsnow fails when a provisioned rule usesnoDataState: NoData, or leaves it unset (Grafana 9.5rules_types.godefaults a blank value toNoData). There are tests for both, for.jsonfiles, and for the real directory.Test
npm test: 155 passed. Before the rule edits, the real-directory test failed and listed the 8 rules.npm run validate: all checks pass.mqtt-influxand check the Telegram send log. The result goes in the issue.