feat(dashboards): release markers on tile charts - #2827
Closed
teeohhem wants to merge 9 commits into
Closed
Conversation
Correlating a latency or error spike with a release meant leaving the product. Dashboard tiles can now overlay the moment each release went out, derived from changes in the OpenTelemetry `service.version` resource attribute, so no instrumentation change is needed. Markers reuse the annotation overlay built for alert firing/recovery lines, which was designed source-agnostic for exactly this. Three properties make the markers trustworthy rather than noisy: - Scoped to the tile. The query runs against the tile's own source with the tile's own filters, so a chart filtered to one service is not annotated with another service's releases. - Tinted to the series. On a chart grouped by service, each marker takes its service's line color, so a release of one service cannot be read as another's. - Suppressed when unattributable. An aggregate line spanning several services gets no markers at all, since a marker naming a service with no visible line invites false attribution. The shared annotation renderer also gains label collapsing, so dense clusters read as "N deploys" instead of overlapping text, sized from the estimated label width rather than a fixed gap.
🦋 Changeset detectedLatest commit: f05da93 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThe PR adds configurable release-version expressions and overlays source-scoped release markers on dashboard time charts.
Confidence Score: 4/5The PR is not yet safe to merge because rollback and same-version redeployment transitions can still be omitted from release markers. The current query aggregates each service/version pair to its earliest timestamp and then drops aggregates predating the window, so a service returning to a previously observed version produces no marker. Files Needing Attention: packages/app/src/hooks/useReleaseAnnotations.tsx
|
| Filename | Overview |
|---|---|
| packages/app/src/hooks/useReleaseAnnotations.tsx | Builds source-scoped release queries and converts results into annotations; repeated returns to an earlier version remain collapsed by the version-level aggregation. |
| packages/app/src/components/charts/chartAnnotations.tsx | Extends annotation layout with merging, clustering, series resolution, and tooltip member metadata. |
| packages/app/src/HDXMultiSeriesTimeChart.tsx | Integrates annotation coloring, responsive layout, hover targets, and tooltips into time charts. |
| packages/app/src/DBDashboardPage.tsx | Adds the URL-backed release-marker toggle and supplies tile-specific query context to the annotation hook. |
| packages/api/src/models/source.ts | Persists the optional service-version expression for log and trace sources. |
| packages/common-utils/src/types.ts | Adds the shared source-field contract used to configure release-version extraction. |
Sequence Diagram
sequenceDiagram
participant User
participant Dashboard
participant ReleaseHook
participant ClickHouse
participant Chart
User->>Dashboard: Enable release markers
Dashboard->>ReleaseHook: Tile source, filters, and time range
ReleaseHook->>ClickHouse: Query first-seen service versions
ClickHouse-->>ReleaseHook: Release rows
ReleaseHook-->>Dashboard: Chart annotations
Dashboard->>Chart: Merge alert and release annotations
Chart-->>User: Render clustered markers and tooltips
Reviews (9): Last reviewed commit: "test(api): use a provisioned connection ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
E2E Test Results✅ All tests passed • 285 passed • 1 skipped • 1074s
Tests ran across 4 shards in parallel. |
Contributor
|
placeholder |
Deployment markers assumed every team emits the OpenTelemetry `service.version` resource attribute. Plenty don't: under GitOps the release identifier is the container image tag, reaching telemetry as `container.image.tag` rather than a version attribute. Those teams saw no markers at all and had no way to say where their version lives short of changing instrumentation. Log and trace sources now carry an optional `serviceVersionExpression`. It defaults to `service.version`, so nothing changes for teams already following resource semconv, and points anywhere else for teams that aren't. This mirrors how ServiceNow Cloud Observability lets you register the attribute that identifies a release. The integration tests round-trip the field through the database rather than only through the Zod schema, because a field missing from the Mongoose discriminator is dropped silently on write.
github-actions
Bot
added
review/tier-4
Contributor
Deep ReviewScope: 🟡 P2 — recommended
🔵 P3 — nitpicks🔵 P3 nitpicks (3)
Reviewers (2): ce-security-reviewer, orchestrator diff analysis. The security pass returned no findings: Testing gaps:
|
When two services deployed within a label's width of each other, the cluster collapsed to "N deploys" in the anchor's series color. On a chart grouped by service that reads as "this service shipped twice" when in fact two different services shipped once each, which is exactly the misattribution the per-series tinting exists to prevent. A cluster spanning several series now uses a neutral color. The count stays true, and the absorbed markers keep their own colors, so which services took part is still readable from the lines. Surfaced by pointing one source's version expression at a coalesce over two attributes, which is how a heterogeneous fleet gets markers for services that carry `service.version` and services that only carry an image tag.
A fleet where some services carry the release on `service.version` and others only on a container image tag needs one expression covering both, which a coalesce already does. Nothing about that is guessable from an empty field, so say it in the help text and the API docs. Shared as a constant rather than duplicated across the log and trace forms, matching KNOWN_COLUMNS_EXPRESSION_HELP_TEXT.
The feature marks when a new service version is first observed in telemetry. That is not the same as a deployment, and the old name over-claimed in both directions. It marked non-deployments: a service idle past the lookback draws a marker when it scales back up, and a canary at 5% of pods looks like a full rollout. It also missed real deployments, which is the damaging half - config changes, secret rotations, and teams shipping from main with a static version all deploy without changing the version string, so a user could deploy twenty times and reasonably conclude the feature was broken. "Release markers" is true to the mechanism, stays true if we later accept pushed deploy events from CI, and pairs with the serviceVersionExpression source field. Its failure mode also reads correctly: no version change, no release to mark. "Deployment" stays in the help text and empty state, both for discoverability and because the empty state is now the right place to explain that deploying without changing the version marks nothing.
teeohhem
changed the title
Colour was the only thing tying a marker to a service, and that lookup breaks down exactly when it matters. The legend caps at 4 entries, so on a chart with more series than that a marker can be tinted to match a line whose legend entry is hidden behind "+N more" - there is nothing on screen to resolve the colour against. Collapsed clusters were worse still: "6 releases" named nobody at all. Hovering a marker now lists every release in its cluster with the service that shipped it, its version, and the time. Implementation notes: - The hit targets live in a ZIndexLayer above every other chart layer. Without that the series areas (zIndex 100) and the marker lines (400) receive the pointer first. The lines and labels are now pointer-transparent for the same reason; they are decoration and the hit layer owns the interaction. - Targets are confined to the label headroom rather than the full plot height. Covering the plot made the series tooltip fire alongside this one, and the label is the natural thing to aim at anyway. Events are never stopped, so drag-to-zoom still works underneath. - One target per cluster, spanning it, so the muted lines inside a cluster are covered by their anchor rather than intercepting it. - The tooltip is portaled and fixed-positioned, matching the series tooltip: a dashboard tile clips its overflow, so an absolutely positioned tooltip is cut off at the tile edge. Layout is now shared between the rendered lines and the hit layer via `layoutAnnotations`, so the hover bands can't drift from the markers they belong to.
# Conflicts: # packages/app/src/HDXMultiSeriesTimeChart.tsx
POST /sources now rejects a connection that does not exist for the team (added upstream alongside the same check on the external API). The suite registers a real Connection for MOCK_SOURCE.connection in its login helper, and every other test in this block inherits it by spreading MOCK_SOURCE. The trace fixture minted its own ObjectId instead, so it started 400ing once that check landed.
Contributor
Author
|
Superseded by a 3-PR stack, so the frontend work isn't gated behind the Tier 4 review that only the
The stack tip is byte-identical to this branch across every feature file; the only deliberate difference is the source field's help text, reworded so #2893 reads correctly on its own rather than forward-referencing markers. Marking this draft rather than closing it, in case the combined diff is easier to read in one place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Correlating a latency or error spike with a release currently means leaving HyperDX. Dashboard tiles can now overlay the moment each version of a service first appeared, so a deployment can be lined up against a change in the data without a CI integration. The markers reuse the annotation overlay built for alert firing/recovery lines, which was written source-agnostic for exactly this. They are off by default and toggled from the dashboard overflow menu, with the state carried in the URL as
releaseMarkersso a shared link keeps it.They are called release markers rather than deployment markers on purpose. What we detect is a new version value appearing in telemetry, which is not the same as a deployment: a service idle past the lookback draws a marker when it scales back up, and more importantly a deploy that does not change the version string (config changes, secret rotations, shipping from
mainwith a static version) produces no marker at all. "Release marker" is true to the mechanism and its failure mode reads correctly, where "deployment marker" would have users reasonably concluding the feature was broken.Where the version comes from is configurable per source. It defaults to the OpenTelemetry
service.versionresource attribute, so teams following resource semconv get markers with no setup. Teams whose release identifier lives elsewhere, which under GitOps is typically the container image tag arriving ascontainer.image.tag, setserviceVersionExpressionon the log or trace source instead of changing instrumentation. A fleet split across both can cover it with onecoalesce, which the field's help text spells out.Three rules keep the markers trustworthy rather than noisy. A marker only helps correlation if the reader can attribute it to something visible, so what a tile shows depends on what it charts:
The query runs against the tile's own source with the tile's own filters, which is what makes scoping possible. The version already running when the window opens is recognised and dropped rather than drawn as a release that never happened. Dense clusters collapse to "N releases", sized from the estimated label width; a cluster spanning several services goes neutral rather than wearing one of their colors and claiming the others' releases as its own.
Markers are available on log and trace sources. Metric sources resolve their table per metric type, so there is no single table to re-aggregate and no way to make a tile's filters meaningful against it; following the source correlation fields to a companion log source is the natural follow-up.
Screenshots or video
Link to chart: https://hyperdx-oss-git-tomdeployment-markers-hyperdx.vercel.app/dashboards/285e6f14040abce8?granularity=auto&from=1786393808084&to=1786566608084&alertAnnotations=true&releaseMarkers=true
How to test on Vercel preview
Preview routes: /dashboards
Steps:
data-testid="dashboard-menu-button").data-testid="toggle-release-annotations-menu-item").releaseMarkers=trueand the menu item now reads "Hide release markers".releaseMarkersis removed from the URL.Note: whether marker lines actually render depends on the preview's demo data carrying a version attribute. The steps above assert the toggle, URL state, and the new source field, which hold regardless.
References
serviceVersionExpressiondoes.