Skip to content

fix(grafana): no false DATA SOURCE ISSUE page when influxdb restarts (#1682) - #1683

Merged
groupsky merged 2 commits into
masterfrom
fix/1682-grafana-wait-influxdb
Oct 11, 2026
Merged

groupsky merged 2 commits into
masterfrom
fix/1682-grafana-wait-influxdb

Conversation

@groupsky

@groupsky groupsky commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1682.

Why a page was sent although execErrState: OK

Grafana 9.5 logs from the 2026-10-11 deploy:

  1. 11:00:00 tick: InfluxDB was still opening its store, so the queries failed. execErrState: OK turned this into Normal. No page came from this step.
  2. InfluxDB began to answer at 11:00:04. The heat pump query (current_power, phase B, last 2 min) then ran without error but got no rows. The writers had not yet back-filled the minutes when InfluxDB was down. The data is there now, but it was not there at that moment.
  3. noDataState: NoData put the rule into the NoData state. In Grafana 9.5, NoData does not wait for for:. It is sent at once as an alert named DatasourceNoData, and telegram-bridge prints that as "📵 DATA SOURCE ISSUE" (docker/telegram-bridge/src/message-utils.js).
  4. Both "Термопомпа не работи" (old UI rule 3A5nIop4k) and "Термопомпа не работи (Webhook)" did this, so 2 pages came.

Changes

  1. grafana waits for influxdb: condition: service_healthy. InfluxDB already has a real healthcheck: /health on port 8086. That port opens only after the store is open, and the 300 s start period covers slow opens. This is the smallest way to do it; a wait loop in Grafana's entrypoint would do the same job with more code. Limits:
    • The wait works only when compose starts both in one up, as a deploy that recreates both does. After a reboot, Docker's restart policy does not wait. Change 2 covers that case for the heat pump rule.
    • deploy.sh uses up --no-deps. If only grafana changes, compose v2.18 does not wait. If both change, it waits up to ~300 s, which is inside COMPOSE_TIMEOUT=600.
    • The infrastructure CI test now starts grafana ha with --no-deps. That test needs only the ingress network, and InfluxDB cannot start in that job (its bind-mounted data dir is owned by root). It could not start before this PR either; nobody saw it because nothing waited for it.
  2. The heat pump "not working" rule gets noDataState: Alerting. In 9.5 the Alerting state does wait for for: 15m. A short no-data gap after a restart stays silent. If no rows come back for more than 15 min, the rule still pages under its own title. A full InfluxDB outage, where queries fail, is still seen through the rules that have execErrState: Alerting (boiler controller, sunseeker). Change 1 alone is not enough: the empty window comes from the writers back-filling late, not from Grafana starting early.
  3. 3 old UI-made rules are deleted with a deleteRules file (same way as delete-boiler-manual-mode-alert.yaml). They copy provisioned rules, so each alert was sent twice. Before the delete, all 9 rules in that folder were exported on the host (full rows from grafana.db, plus a copy of the DB file). There is no Grafana admin login on the host, so the Grafana API could not be used.

Old "General Alerting" rules (UI-made, not in the repo)

Contact point: every rule below, and every twin, has no labels that match a route, so all of them use the default telegram-webhook.

Legacy uid Title Provisioned twin Query Threshold for Contact Action
3A5nIop4k Термопомпа не работи webhook-3A5nIop4k same (*) same (<30 W) same 15m same deleted
lA5nSTtVk Main Power alert webhook-lA5nSTtVk same same (>10 kW) same 5m same deleted
z157Sot4z Water pump power alert webhook-z157Sot4z same same (>1 kW) same 15m same deleted
xjXziW_4k Термопомпа претоварена (webhook-lA5nSTt4k is close) 15 min window, not 10 min >1000 W, not >3000 W 15m, not 10m same kept
3057STtVkz AC alert webhook-3057STtVkz other (min/max, not median) 210/248 V, not 207/253 V 0, not 4m same kept
RJcnSop4z External alert none – – 5m default kept
rA5nSotVkz Temperature alert none – – 5m default kept
j05nSotVz Humidity alert none – – 5m default kept
eAc7ITtVk Output alert none – – 5m default kept

(*) The only differences: the expression step uses the old datasource id -100 instead of __expr__ (both mean "expression"), and 3A5nIop4k keeps an unused raw-query text from an old editor mode (rawQuery: false).

For the owner: the 6 kept rules are still only in the Grafana DB. xjXziW_4k alerts at >1000 W but its message says 3000 W, and it keeps noDataState: NoData.

Not changed (said here instead of built)

Other provisioned rules still use noDataState: NoData. The only one with a short window is Main Power (Webhook), with a 5 min window. If InfluxDB is killed and needs ~200 s to open, that window could also be empty for a moment. Its Alerting state means ">10 kW", so noDataState: Alerting would be a wrong page there. If it ever fires on a deploy, it needs its own fix.

Done-when proof

The deploy check (both containers recreated, then the Telegram send log) will be added as a comment on #1682 after merge.

…1682)

- grafana waits for influxdb to be healthy (depends_on service_healthy)
- heat pump 'not working' rule: noDataState Alerting, so a short no-data
  gap waits for 'for: 15m' instead of paging at once
- delete 3 old UI-made rules that duplicate provisioned ones
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

routy: deploy restarting influxdb + grafana sends false "DATA SOURCE ISSUE" pages — grafana evaluates before influxdb is up

1 participant