feat(influxdb): alert when a shard fails to open (#1685)

[groupsky]

Closes #1685.

What

• docker/influxdb/shard-check.sh: a small loop inside the influxdb container, started by secure-entrypoint.sh.
  • It waits until /health answers. influxd opens all shards before it starts HTTP, so it never runs during the ~2 min shard loading.
  • Then, and every 15 min, it counts shards that are in SHOW SHARDS (meta store) and have a directory on disk, but are missing from SHOW STATS FOR 'shard' (only open shards are listed). All databases are checked.
  • It writes influxdb_health failed_shards=<n>i to the homy database and logs the broken shards as shard-check: ....
  • Shards with no directory are skipped: InfluxDB creates next week's shard in the meta store ahead of time, but creates the directory only on the first write.
  • It uses the admin user from the existing Docker secrets (via INFLUX_USERNAME / INFLUX_PASSWORD, so the password is not in the process list).
• Grafana rule influxdb-shard-open-failed ("InfluxDB Shard Failed To Open"): last(failed_shards) over 1 h.
  • Fires when the value is above 0.
  • Fires when there is no value for 1 h (noDataState: Alerting), so it also fires if the check stops.
  • execErrState: OK, so an influxdb restart (query errors) does not page.
  • Goes to the default Telegram route.
• docker/influxdb/test-shard-check.sh: the test from the issue.
• docs/influxdb-schema.md: the influxdb_health measurement, and the alert named in "Field Types and Broken Shards" next to the repair steps.

Tests

docker/influxdb/test-shard-check.sh (throwaway 1.12.4 container built from this image, auth on):

PASS: failed_shards=0     # healthy instance
PASS: failed_shards=1     # INSERT zz f=1i, DROP MEASUREMENT zz, INSERT zz f=1.5, docker kill, start

• shellcheck: clean for the new scripts.
• validate-grafana-alerts: the new file passes (1 evaluator).

Not done (on purpose, to keep it small)

• The test script is not wired into CI. It needs Docker and about 30 s. It can be added to the CI later if wanted.

[groupsky]

Review and extra tests

Grafana rule, tested in a local Grafana 9.5.21 + InfluxDB 1.12.4 (rule interval set to 10s for speed only):

• failed_shards=1: rule fires, alertname "InfluxDB Shard Failed To Open".
• newer failed_shards=0: rule goes Normal.
• no influxdb_health data: rule fires (noDataState: Alerting), same alertname, not DatasourceNoData.
• No evaluator or provisioning errors in the Grafana log.

Subagent review: no blockers. Fixed in the second commit:

• Alert also fires when InfluxDB cannot be queried for 15 min (execErrState: Alerting, for: 15m). A deploy restart (2-5 min) does not page. last() over 1h keeps a value above 0 for the whole 15 min, so a broken shard still pages.
• Shards that look failed are checked once more after 10 s, so a shard created by a write between the two queries is not counted.
• The test was flaky (1 in 5): right after docker start, Docker can still show the old healthy status. It now waits for the Failed to open shard log line. 10 runs in a row passed after the fix.

Not changed: on the very first start of an empty volume, the first value comes 15 min late (the init server answers /health). This is harmless and never happens on the real host.