internet-latency-collector: lossy RIPE Atlas circuits get their sample timestamps compressed onchain

[nikw9944]

Summary

The RIPE Atlas exchange export path records only results that carried an RTT. The telemetry program stores samples append-only at next_sample_index and reconstructs each sample's time as start_timestamp + index × sampling_interval. A missed ping therefore consumes no slot, and a lossy circuit's timeline is compressed by its loss ratio. On mainnet-beta, Columbus (cmh) circuits at ~17% success have their RTTs recorded up to ~30 hours earlier than they happened.

Evidence (mainnet-beta, 2026-09-15)

• cmh → dub, epoch 218, from fact_dz_internet_metro_latency: event_ts advances by exactly 10 minutes per sample_index (14:10:42, 14:20:42, 14:30:42 …) while ingested_at for those same 40 samples runs from 2026-09-12 14:27 to 2026-09-14 05:24. Sample index 39 is recorded at 2026-09-12 20:40 and was ingested at 2026-09-14 05:24 — it happened roughly 32 hours after its recorded time.
• Per epoch (212 through 219), cmh-as-origin ripeatlas samples span roughly 14 hours of event_ts against roughly 44 hours of real time — a ~3:1 compression that matches the measurement's success rate in the collector's own export logs (valid_latencies / raw_results, steady at 13–26% per hour).
• Ingest lag ingested_at − event_ts over 2026-09-13/14: cmh-as-origin ripeatlas p50 828 min, p90 1345 min, max 1954 min. Every other metro: p50 3–14 min. Control: wheresitup rows for the same cmh circuits lag 56 min at p50 — same metro, same table, same indexer, so this is the ripeatlas collector's encoding, not the pipeline.
• The lag is uniform across all 27 cmh circuits regardless of their sample count, confirming it is the write encoding rather than anything per-circuit downstream.

Mechanism

• controlplane/internet-latency-collector/internal/ripeatlas/collector.go, exportSingleMeasurementResults: records are appended only if latency > 0; a result whose ping array has no rtt produces no record.
• controlplane/internet-latency-collector/internal/exporter/submitter.go, SubmitSamples: each batch is written as Samples []uint32 with StartTimestampMicroseconds = min(batch timestamps); there are no per-sample timestamps.
• smartcontract/programs/doublezero-telemetry/src/processors/telemetry/write_internet_latency_samples.rs: start_timestamp_microseconds is set only when it is 0 (first write); write_index = next_sample_index; samples are appended. The Go SDK documents StartTimestampMicroseconds as "the timestamp of the first written sample".

Measurements are created with Packets: 1, so a result with no RTT is unambiguous loss for that interval.

Consequences

1. Customer-facing latency views place a lossy circuit's RTTs at the wrong times, drifting later through each ~44h epoch and snapping back at rollover.
2. Any analysis of coverage or freshness keyed on event_ts is wrong for lossy circuits.
3. internet-telemetry alerts: 63-hour detection gap, rules key on header write time rather than sample recency #4170 proposes keying the Internet Telemetry: No Samples alert on sample recency instead of header written_at. With compression in place that alert would fire continuously on every lossy circuit. This must land first, or that alert must key on ingested_at.
4. Neither current alert reads event_ts (Account Not Found uses ingested_at within 2h; No Samples uses header written_at), so this does not by itself cause the current alert noise on cmh.

Proposed fix

Record a loss as a sample, so every sampling interval consumes a slot. The convention already exists in two places: the device telemetry agent records a lost probe as RTT: 0 (controlplane/telemetry/internal/telemetry/pinger.go), and #4321 does the same for cloud-mode RIPE measurements (TotalLossBecomesAZeroRTTRecord). Apply it to the exchange path in exportSingleMeasurementResults: a result with no RTT emits a record with RTT: 0 at the result's timestamp.

Capacity is not a concern: MaxInternetLatencySamplesPerAccount = 3000 against ~264 slots per circuit per 44h epoch at the 10-minute interval.

Before building, settle two things

• Downstream treatment of rtt_us = 0. Confirm that the ClickHouse views and any public-page queries exclude zero RTTs from averages and percentiles. The view definitions are not in this repository (they live in lake); this needs checking there. The device latency path already produces zero-RTT rows, so the convention is probably already handled, but verify.
• LastExportAt and the export cursor must not advance on a loss record. Both currently advance only on a positive RTT, and that is what makes the never_exported / no_recent_exports detection in configureMeasurements work. A loss record that satisfied it would mean a dead target is never marked. Keep the cursor semantics keyed to successful samples; only the exporter record set changes.

Size: roughly 40 lines plus tests in the collector, plus the downstream verification.

Related

• internet-latency-collector: export total packet loss in cloud mode #4321 — the same fix for cloud mode.
• internet-telemetry alerts: 63-hour detection gap, rules key on header write time rather than sample recency #4170 — must not land its sample-recency change before this.
• internet-latency-collector: rank lossy ripe atlas targets last and split the probe blacklist by role #4331, internet-latency-collector: exclude probes tagged system-ipv4-doesnt-work #4334 — probe selection work on the same metro; independent of this, but the ClickHouse coverage numbers used to motivate them were distorted by this bug.