internet-latency-collector: add clickhouse exporter - #4320
Draft
thijsvanemmerik wants to merge 1 commit into
Draft
thijsvanemmerik wants to merge 1 commit into
thijsvanemmerik wants to merge 1 commit into
Conversation
Batches records into a bounded buffer and inserts them asynchronously, dropping the oldest when the buffer is full and counting what it drops. A write returns nil while the buffer still has room and an error only when the buffer is at its cap, so a ClickHouse outage does not stop the RIPE collector before it updates its cursor and make it re-fetch a growing window every cycle. Flush still returns its error to its callers, and a flush a write triggers is logged and counted, so a failed insert stays visible. Close bounds its final flush at 10 seconds. The driver sets no read timeout and defaults to 300, so a ClickHouse that accepts a connection and then stops answering would hold shutdown open far past the usual 30-second container kill. The constructor pings the connection it opens. clickhouse.Open builds a pool without dialing, so a wrong address, user, password or database would surface only at the first insert, where a failed flush is a warning and the read position advances anyway. Every ClickHouse setting also has a working-looking default, so the pod would start, spend RIPE credits and leave the table empty. The ping fails the startup instead, and sits next to the Open call it validates so the injected-inserter seam the tests use stays untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of measuring latency between AWS regions with the internet latency collector, which needs
those samples in ClickHouse.
The collector writes latency samples to the DoubleZero ledger or to CSV, and has no way to write to
ClickHouse. This adds a ClickHouse exporter behind the same exporter interface, so a caller can use
either.
It buffers records and inserts them in batches of 500. A failed insert puts the batch back at the
front of the queue, so a ClickHouse restart delays records instead of losing them. The buffer holds
50,000 records, about two days, and drops the oldest beyond that rather than growing without limit.
The constructor pings ClickHouse, so a wrong address, password or database name fails at startup
instead of warning on every flush. Four Prometheus metrics report records written, records dropped,
insert errors and buffer depth.
Nothing builds this exporter yet, so merging changes nothing that runs today.
Test:
go test ./controlplane/internet-latency-collector/internal/exporter/.... The tests thatstart a real ClickHouse container need Docker.