Skip to content

fix(grafana): move or remove the last 6 UI-made alert rules (#1690) - #1694

Merged
groupsky merged 2 commits into
masterfrom
fix/grafana-ui-rules-1690
Oct 11, 2026
Merged

groupsky merged 2 commits into
masterfrom
fix/grafana-ui-rules-1690

Conversation

@groupsky

Copy link
Copy Markdown
Owner

Closes #1690.

Owner decision from #1690: remove 2 UI-made rules that a provisioned rule covers, and move 4 into the repo. After this, folder "General Alerting" has no UI-made rules.

Heat pump overload: the provisioned rule did not cover the UI rule

The issue asked to confirm this before deleting xjXziW_4k. It did not match:

UI rule xjXziW_4k provisioned webhook-lA5nSTt4k (before)
Query mean(p) of current_power, heat_pump, grouped by phase max(p), all phases in one series
Condition avg over 15 min > 1000 W on any phase avg over 10 min > 3000 W
for 15m 10m

So the provisioned rule needed one phase above 3000 W. Data on routy:

  • 2026-01-10..24 (winter): phase B was above 1000 W (15 min mean) in 42 of 1344 windows. The highest reading on any phase was 1956 W, so the old provisioned rule could never fire.
  • The UI rule really fired 23 times since 2023-11 (last 2026-04-09), from the Grafana state history.
  • Last 7 days: highest 15 min mean per phase is 658 W, so the change does not fire now.

Decision (option A, agreed): webhook-lA5nSTt4k now has the UI rule's meaning: per-phase mean > 1000 W over 15 min, for 15m. The message says "над 1000 W на поне една фаза ... (3 x 1000 W = 3000 W общо)". Then xjXziW_4k is deleted.

Other rules

UI uid What happens
3057STtVkz AC alert (210/248 V, no for) deleted; provisioned webhook-3057STtVkz (207/253 V, 4m) stays
RJcnSop4z External alert moved to webhook-RJcnSop4z in heat-pump-alerts.yaml: min(s2Temp) of monitoring2 heatpump-ctrl, avg < 5 °C, for 5m
eAc7ITtVk Output alert moved to webhook-eAc7ITtVk: max(s3Temp), avg > 60 °C, for 5m
rA5nSotVkz Temperature alert moved to webhook-rA5nSotVkz in new controlbox-alerts.yaml: mean(t) of xymd1 controlbox (t < 80), outside 5-37 °C, for 5m. The message now says 5-37 (it said 5-35).
j05nSotVz Humidity alert moved to webhook-j05nSotVz: mean(h), outside 15-85 %, for 5m

The 4 moved rules use the exact query, reducer, window and for read from the Grafana DB copy. noDataState and execErrState stay OK as in the UI rules, so a short InfluxDB restart does not page (#1682). Deletes go through deleteRules in delete-legacy-duplicate-alerts.yaml, like #1683.

Deploy without a page

On routy at 15:04 UTC, all 6 UI rules and webhook-lA5nSTt4k are Normal, so the new copies start Normal and do not page.

Tests

  • validate-grafana-alerts: passes for the 3 changed files.

- heat pump overload webhook-lA5nSTt4k: per-phase mean > 1000 W over
  15 min, for 15m (the meaning of UI rule xjXziW_4k); the old max over
  all phases > 3000 W could never fire
- move External/Output alert (heatpump-ctrl) and controlbox
  temperature/humidity alerts into provisioned rules with the same
  query, threshold and for; messages match the thresholds
- delete the 6 UI rules through deleteRules
@groupsky

Copy link
Copy Markdown
Owner Author

Review and local test

Local Grafana 9.5.21 + InfluxDB 1.12.4 (rule interval 10s and for: 0s only for speed):

  • normal data (all phases 500 W, s2Temp 10, s3Temp 40, t 20, h 50): all 5 rules Normal.
  • bad data (phase B 1500 W, A/C 500 W, s2Temp 2, s3Temp 70, t 40, h 90): all 5 rules Alerting. So the overload rule fires when one phase is above 1000 W.
  • t = 100 only: controlbox temperature stays Normal (the t < 80 filter drops sensor errors).
  • All rules health ok, no evaluator or provisioning errors.

Subagent review: no blockers. It checked each moved rule field by field against the Grafana DB export. Fixed: the overload message now says plainly that the limit is per phase. Kept as before: the overload rule keeps noDataState: NoData (same as the old provisioned rule and the UI rule; a short restart is shorter than its 15 min window).

Before deploy I check again that all 6 UI rules and webhook-lA5nSTt4k are Normal, so no deleted rule sends a "resolved" message.

@groupsky
groupsky merged commit 53d2a7b into master Oct 11, 2026
19 checks passed
@groupsky
groupsky deleted the fix/grafana-ui-rules-1690 branch October 11, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

routy: move or remove the last 6 UI-made Grafana alert rules (owner decision)

1 participant