Repository navigation
fix(grafana): no false DATA SOURCE ISSUE page when influxdb restarts (#1682) - #1683
Merged
Merged
Conversation
…1682) - grafana waits for influxdb to be healthy (depends_on service_healthy) - heat pump 'not working' rule: noDataState Alerting, so a short no-data gap waits for 'for: 15m' instead of paging at once - delete 3 old UI-made rules that duplicate provisioned ones
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1682.
Why a page was sent although
execErrState: OKGrafana 9.5 logs from the 2026-10-11 deploy:
execErrState: OKturned this into Normal. No page came from this step.current_power, phase B, last 2 min) then ran without error but got no rows. The writers had not yet back-filled the minutes when InfluxDB was down. The data is there now, but it was not there at that moment.noDataState: NoDataput the rule into the NoData state. In Grafana 9.5, NoData does not wait forfor:. It is sent at once as an alert namedDatasourceNoData, and telegram-bridge prints that as "📵 DATA SOURCE ISSUE" (docker/telegram-bridge/src/message-utils.js).3A5nIop4k) and "Термопомпа не работи (Webhook)" did this, so 2 pages came.Changes
grafanawaits forinfluxdb: condition: service_healthy. InfluxDB already has a real healthcheck:/healthon port 8086. That port opens only after the store is open, and the 300 s start period covers slow opens. This is the smallest way to do it; a wait loop in Grafana's entrypoint would do the same job with more code. Limits:up, as a deploy that recreates both does. After a reboot, Docker's restart policy does not wait. Change 2 covers that case for the heat pump rule.deploy.shusesup --no-deps. If only grafana changes, compose v2.18 does not wait. If both change, it waits up to ~300 s, which is insideCOMPOSE_TIMEOUT=600.infrastructureCI test now startsgrafana hawith--no-deps. That test needs only the ingress network, and InfluxDB cannot start in that job (its bind-mounted data dir is owned by root). It could not start before this PR either; nobody saw it because nothing waited for it.noDataState: Alerting. In 9.5 the Alerting state does wait forfor: 15m. A short no-data gap after a restart stays silent. If no rows come back for more than 15 min, the rule still pages under its own title. A full InfluxDB outage, where queries fail, is still seen through the rules that haveexecErrState: Alerting(boiler controller, sunseeker). Change 1 alone is not enough: the empty window comes from the writers back-filling late, not from Grafana starting early.deleteRulesfile (same way asdelete-boiler-manual-mode-alert.yaml). They copy provisioned rules, so each alert was sent twice. Before the delete, all 9 rules in that folder were exported on the host (full rows fromgrafana.db, plus a copy of the DB file). There is no Grafana admin login on the host, so the Grafana API could not be used.Old "General Alerting" rules (UI-made, not in the repo)
Contact point: every rule below, and every twin, has no labels that match a route, so all of them use the default
telegram-webhook.(*) The only differences: the expression step uses the old datasource id
-100instead of__expr__(both mean "expression"), and3A5nIop4kkeeps an unused raw-query text from an old editor mode (rawQuery: false).For the owner: the 6 kept rules are still only in the Grafana DB.
xjXziW_4kalerts at >1000 W but its message says 3000 W, and it keepsnoDataState: NoData.Not changed (said here instead of built)
Other provisioned rules still use
noDataState: NoData. The only one with a short window is Main Power (Webhook), with a 5 min window. If InfluxDB is killed and needs ~200 s to open, that window could also be empty for a moment. Its Alerting state means ">10 kW", sonoDataState: Alertingwould be a wrong page there. If it ever fires on a deploy, it needs its own fix.Done-when proof
The deploy check (both containers recreated, then the Telegram send log) will be added as a comment on #1682 after merge.