From fc3242dc39921eb7eb1b9650a3ea3ceab753c2fa Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Sun, 2 Aug 2026 16:13:38 +0600 Subject: [PATCH 1/9] Add initial implementation of the Analytics page with supporting documentation - Introduced the Analytics page to visualize agent performance metrics. - Created context, data contract, plan, research, and status documentation for the new feature. - Ensured the new page reuses existing data-fetching mechanisms without altering the backend. --- docs/design/agent-analytics/README.md | 43 +++++ docs/design/agent-analytics/context.md | 99 +++++++++++ docs/design/agent-analytics/data-contract.md | 104 ++++++++++++ docs/design/agent-analytics/plan.md | 164 +++++++++++++++++++ docs/design/agent-analytics/research.md | 152 +++++++++++++++++ docs/design/agent-analytics/status.md | 71 ++++++++ 6 files changed, 633 insertions(+) create mode 100644 docs/design/agent-analytics/README.md create mode 100644 docs/design/agent-analytics/context.md create mode 100644 docs/design/agent-analytics/data-contract.md create mode 100644 docs/design/agent-analytics/plan.md create mode 100644 docs/design/agent-analytics/research.md create mode 100644 docs/design/agent-analytics/status.md diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md new file mode 100644 index 0000000000..8f447a127c --- /dev/null +++ b/docs/design/agent-analytics/README.md @@ -0,0 +1,43 @@ +# Agent Analytics page + +A new project-scoped **Analytics** page for the Agenta web app. It charts how the +project's agents perform over a time window: run volume, success and failure, latency, +cost, and token usage. The page reads from the existing spec-driven analytics endpoint +(`POST /spans/analytics/query`) and reuses the frontend data-layer atoms that already exist +for it, so this work adds a page, not a new data layer. + +## Reading order + +1. **context.md** : why this page exists, what a user sees today, goals and non-goals, and + the three scope decisions that are already locked. +2. **research.md** : the parts of the codebase this feature reuses, with exact file paths: + the analytics fetch layer, the response-to-dashboard mapper, the sidebar and routing, + and the charting library. Read this before proposing any new file. +3. **data-contract.md** : the request the page sends (time window, filter, metric specs) + and the response fields it reads, including the fields the current mapper drops that + this page needs. +4. **plan.md** : the build broken into phases, with the file list per phase and the clean + boundary where the deferred model and tool views drop in once the backend supports them. +5. **status.md** : current state, open questions, and decisions. This is the source of + truth for progress; update it as work lands. + +## Glossary + +Terms used across these documents, defined once here. + +- **Run**: one agent invocation. On the backend it is one root span (a span with no + parent). Run count per time bucket equals the count of the `ag.type.trace` metric. +- **Span**: one unit of work inside a run (a model call, a tool call, or the agent step + itself). Model name and tool name live on child spans, not on the root span. +- **Root span**: the top span of a run. Today's analytics endpoint only reads root spans. +- **Bucket**: one time slice of the chart x-axis (for example one day, or one hour). The + endpoint returns one metrics object per bucket. +- **Metric spec**: a request instruction of the form `{type, path}` that tells the endpoint + which JSON path on the span to summarize and how. The endpoint has no fixed metric list; + it summarizes whatever path a spec names. +- **Focus**: a request field selecting whether the query aggregates over root spans + (`trace`) or all spans (`span`). Today only `trace` works; see context.md. +- **Project scope**: the web app always has exactly one project in context. This page lives + at that level and, by default, aggregates every agent in the project. +- **Health score**: a single 0 to 100 number this page computes in the browser from success + rate and average latency. It is a display aid, not a backend metric. diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md new file mode 100644 index 0000000000..393f789107 --- /dev/null +++ b/docs/design/agent-analytics/context.md @@ -0,0 +1,99 @@ +# Context + +## What a user sees today + +Today the web app has no page that shows agent performance across a project. A team running +several agents cannot open one screen and read how many runs happened, how many failed, how +fast they ran, and what they cost, narrowed to the agents they care about, over a window they +choose. + +The frontend already has the data plumbing to answer these questions. The atoms, the fetch +function, and the response mapper that talk to the analytics endpoint exist and are in use. +This feature reuses that plumbing; it does not rebuild the data layer. + +## What this feature adds + +A new page titled Analytics, reachable from the project sidebar, scoped to the whole project. +It shows: + +- A header with the page title, a one-line description, a time-range control that accepts any + window, and a Filters popover with an Agents multi-select that narrows the query to chosen + agents. The time-range control reuses the existing observability time windowing (the `Sort` + control and its `SortResult`), so it supports the standard presets and a custom + start-and-end range. It is not a fixed list of options. +- A summary panel: a health donut (0 to 100, with a Healthy / Watch / At risk band and a + one-line read-out) and four stat tiles (Total runs, Success rate, Avg latency, Total cost). + Each tile shows a change badge against the previous window of equal length and a small trend + line of the current window. +- Four charts in a grid, each with hover tooltips and a legend whose entries toggle series on + and off: + - Runs: stacked bars of successful and failed runs per bucket. + - Latency: bars of average latency per bucket, with a p95 marker line; the tooltip shows + average, p95, min, and max. + - Costs: stacked bars of prompt cost and completion cost per bucket. + - Tokens: stacked bars of prompt tokens and completion tokens per bucket. + +## Locked scope decisions + +Three decisions are settled and drive the plan. Do not reopen them without the requester. + +1. Frontend-first. Build the four charts above, the four stat tiles, the health donut, the + Agents filter, and the time-range control against today's endpoint. The endpoint already + returns every field these need, so this scope ships without any change under `api/`. + +2. Health donut computed in the browser. The page derives the health score from + `0.72 x successRate + 0.28 x latencyScore`, where `latencyScore` maps average latency onto + a 0 to 1 range (higher is faster). The bands are Healthy at 85 and above, Watch from 65 to + 84, At risk below 65. This is a display aid; it is not sent to or stored on the backend. + +3. New page at project scope. This is a net-new page named Analytics. It aggregates every + agent in the project by default. The Agents multi-select narrows the set, so the default + query carries no single-app reference filter. + +## Showing model usage and tool usage needs backend work + +Two views are out of scope for this plan because the backend cannot serve them yet. This is an +engineering constraint, not an open design question. + +Tool usage (calls per tool, tool error rate) and model usage (runs per model, cost per model) +read fields that live on child spans: the tool name, the model name, the span type, and the +per-span status. Today's analytics endpoint reads only root spans. It accepts a `focus` field +that would widen the scan to all spans, but the field is inert. So these fields are simply not +reachable through the endpoint today. + +The two backend prerequisites are not co-equal gates; they unlock different things. + +- Span-focus wiring unlocks both the tool view and the model-share view in their basic form. + `query.focus` is in scope in `dao.analytics()` but never reaches `build_base_cte`, which + unconditionally applies `WHERE parent_id IS NULL` + (`api/oss/src/dbs/postgres/tracing/utils.py`). Threading `focus` through so `focus = span` + drops that predicate lets the query read child spans. A `categorical/single` spec already + returns per-value frequencies, so counting calls per tool or runs per model needs no + group-by once span focus works. + - Correctness caveat, must ship with the fix: under `focus = span` the cumulative metric + paths double-count. `ag.metrics.*.cumulative` are rollups stored on the root span, so + scanning all spans sums the root total plus every child's. Cost and tokens must switch to + the `incremental` paths under span focus, or the figures come out inflated with no error. + The fix needs a guard that rejects or auto-maps a cumulative spec under `focus = span`. +- Group-by dimension unlocks only per-model cost and tokens. Crossing a numeric metric (cost, + tokens) with a categorical dimension (model name) is the one thing the frequency reducers + cannot do; the extract stage groups by `(timestamp, spec)` only, with no grouping key. This + is the harder, later change. Preferred shape: a per-query `group_by` dimension that splits + every spec by one path, which leaves `MetricSpec` untouched and confines the nested + `{group_value: stats}` output to one code path. + +## Goals + +- Reuse the existing analytics fetch layer, response mapper, and time-windowing control rather + than adding a second path to the same endpoint. +- Deliver the layout and interactions using the repo's charting library and theme tokens, in + both light and dark themes. +- Leave a clean boundary so the deferred model and tool views become an additive change once + the backend prerequisites land, not a rewrite. + +## Non-goals + +- No change to `api/` in this plan. +- No change to the existing Observability page. +- No real-time streaming; the page fetches per time-range selection and caches like the + existing dashboard atoms. diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md new file mode 100644 index 0000000000..d2b0b59c74 --- /dev/null +++ b/docs/design/agent-analytics/data-contract.md @@ -0,0 +1,104 @@ +# Data contract: request the page sends, response fields it reads + +This page talks to one endpoint, `POST /spans/analytics/query`, through the existing +`fetchSpansAnalytics`. Nothing here changes the endpoint. It documents the exact request the +page builds and the response fields the new mapper reads, and it flags the one frontend +extension: passing explicit metric specs. + +## Request + +The request carries four kinds of fields. Classified by what each field is, not by which +chart it feeds: + +- **Routing** (which tenant and window): `projectId`, `oldest`, `newest`. `projectId` comes + from `projectIdAtom`. `oldest` and `newest` are ISO bounds taken from the selected + `SortResult`, the same range object the observability windowing uses: a standard preset + resolves to a start with `newest` omitted (meaning "now"), and a custom range supplies both + `oldest` and `newest`. Any window is valid; there is no fixed set of options. +- **Policy** (how to slice and aggregate): `focus = "trace"` (root spans only, the only + value that works today), and `interval` (bucket size in minutes) from + `calculateIntervalFromDuration`. +- **Data selection** (what to measure): `specs`, a list of metric specs naming the JSON + paths to summarize. See below. +- **Data filter** (which spans qualify): `filter`, a `{conditions: [...]}` object. At + project scope with no agent selected, omit the reference conditions so the query spans the + whole project. For selected agents, add `references` conditions on the chosen agent ids. The + exact encoding, a single `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per + agent combined with `or`, follows the existing filter builder; see the plan.md open + questions. + +### The `specs` extension + +`fetchSpansAnalytics` omits `specs` today, so the backend applies its default set. The +defaults give totals for cost, tokens, duration, errors, and the trace and span type counts. +That is enough for run count, average latency, total cost, and total tokens, but not for the +prompt-and-completion split or for p95, min, and max latency. + +To get those, pass an explicit `specs` list. Add an optional `specs` field to +`SpansAnalyticsParams` and serialize it to a JSON-string query param exactly as `filter` is +serialized. The specs the page needs, by path and type: + +| Purpose | path | type | fields read | +| --- | --- | --- | --- | +| Run count | `attributes.ag.type.trace` | categorical/single | `count` | +| Failures | `attributes.ag.metrics.errors.cumulative` | numeric/continuous | `sum` | +| Latency | `attributes.ag.metrics.duration.cumulative` | numeric/continuous | `count`, `sum`, `min`, `max`, `pcts.p95` | +| Prompt cost | `attributes.ag.metrics.costs.cumulative.prompt` | numeric/continuous | `sum` | +| Completion cost | `attributes.ag.metrics.costs.cumulative.completion` | numeric/continuous | `sum` | +| Prompt tokens | `attributes.ag.metrics.tokens.cumulative.prompt` | numeric/continuous | `sum` | +| Completion tokens | `attributes.ag.metrics.tokens.cumulative.completion` | numeric/continuous | `sum` | + +The `type` strings are verified against the backend: `MetricType` +(`api/oss/src/core/tracing/dtos.py`) has only `numeric/continuous` and `numeric/discrete` +(there is no plain `numeric`), and the backend's own `DEFAULT_ANALYTICS_SPECS` +(`api/oss/src/core/tracing/service.py`) uses `numeric/continuous` for errors, costs, and +tokens. Use `numeric/continuous` for every number metric above; a bare `numeric` is silently +dropped. + +p95 is **nested**, not a flat field. The numeric/continuous reducer (`parse_pcts` in +`api/oss/src/dbs/postgres/tracing/utils.py`) emits percentiles under a `pcts` object, so the +value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and `max` are flat siblings and +read directly; only the percentiles sit one level down. Still confirm against one live +response in Phase 2, but expect the nested shape. + +## Response fields the mapper reads + +The response is `{buckets: [{timestamp, metrics: {: {: value}}}]}`. The new +mapper produces, per bucket: + +- `success` = `type.trace` count minus `errors` sum, floored at zero. +- `failed` = `errors` sum. Note this is an **error count**, not strictly a failed-run count: + `errors.cumulative` rolls up every failing span in a run, so a single run with two failing + child spans contributes `2`. The floor-at-zero on `success` hides the resulting negative. + This matches the existing observability mapper's convention (`analyticsToGeneration`), so + it stays internally consistent; just read the label as "errors," not "failed runs." +- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. +- `latencyMin`, `latencyMax` = the flat `min` / `max` duration fields. +- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95` (see the note above the + response section); `metricField`'s flat one-level read does not reach it, so the mapper + needs a small `pcts` accessor. +- `costPrompt`, `costCompletion` = the two cost sums. +- `tokensPrompt`, `tokensCompletion` = the two token sums. These split fields are computed + from per-part token counts, so providers that report only total tokens leave the + prompt/completion split at zero while `total` is correct; verify the split is non-zero on + live traffic in Phase 2, or the Costs/Tokens split bars render empty for those agents. + +And window totals for the four stat tiles: total runs, success rate, average latency, total +cost, plus the same figures for the previous window so the change badges have a baseline. + +## Health score (browser-side, not in the contract) + +Computed from the window totals, never sent to the backend: + +- `successRate` = successful runs / total runs. +- `latencyScore` = clamp01((3600 - avgLatencyMs) / 2800). Confirm these two constants before + implementing; they set where "fast" and "slow" fall. +- `health` = round(100 x (0.72 x successRate + 0.28 x latencyScore)). +- Band: Healthy at 85 and above, Watch from 65 to 84, At risk below 65. + +## Boundary for the deferred charts + +The deferred Tools and Models charts need `focus = "span"` and, for per-model cost, a +`group_by` field on a spec. Neither works on today's endpoint. Keep the request builder able +to take a different `focus` and extra specs without reshaping, so the follow-up change adds a +second query rather than rewriting this one. diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md new file mode 100644 index 0000000000..364b377498 --- /dev/null +++ b/docs/design/agent-analytics/plan.md @@ -0,0 +1,164 @@ +# Plan + +Build in four phases. Each phase is independently reviewable and leaves the app working. The +deferred Tools and Models charts are a fifth phase that lands after a backend change; the +first four phases finish the locked scope. + +File paths follow the existing observability feature so the new page reads as a sibling of +it, not a new pattern. + +## Phase 1: page shell, route, and navigation + +Goal: an empty Analytics page reachable from the sidebar, scoped to the project. + +- Add the route wrapper `web/oss/src/pages/w/[workspace_id]/p/[project_id]/analytics/index.tsx`, + a thin default export around a new page module, mirroring the observability page file. +- Add the page module `web/oss/src/components/pages/analytics/index.tsx` with the header + (title, description) and a placeholder body. +- Add the sidebar entry in + `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`: key `app-analytics-link`, + title `Analytics`, link `${projectURL}/analytics`, an icon from `@phosphor-icons/react`, + disabled when there is no project URL. Place it next to the Observability entry. +- Confirm the agents-list source for the later filter (app-management or workflow molecule + selectors) and note the chosen atom in status.md. + +Done when: the sidebar shows Analytics, the route renders the header, both themes look +correct. + +## Phase 2: data layer + +Goal: the page can fetch mapped analytics for the current and previous windows. + +- Extend `SpansAnalyticsParams` in `web/packages/agenta-entities/src/trace/api/api.ts` with an + optional `specs` field, serialized to a JSON-string query param like `filter`. Verify the + entities package still builds (`pnpm turbo run build --filter=@agenta/entities`). +- Add a new mapper next to the existing one in `web/oss/src/services/tracing/lib/` (do not + change `analyticsToGeneration`; the observability page depends on it). The new mapper reads + the split cost and token paths and the duration min, max, and p95 fields, and returns the + per-bucket and window-total shape in data-contract.md. Reuse `metricField` and + `calculateIntervalFromDuration`. +- Add a fetch function in `web/oss/src/services/tracing/api/` that builds the project-scope + conditions (no single-app reference by default; one reference condition per selected agent), + passes the explicit `specs`, and calls `fetchSpansAnalytics`. +- The spec `type` strings (`numeric/continuous`) and the nested `pcts.p95` shape are already + verified in the backend code (see data-contract.md); validate them against one live response + while building this phase, and check the prompt/completion split reads non-zero on real + traffic. +- Derive the previous comparison window as the equal-length window immediately before the + selected one: for `[oldest, newest]`, the previous window is + `[oldest - (newest - oldest), oldest]`. This works for both preset and custom ranges. + +Done when: a temporary log or test shows correct totals for a known window, and the previous +window is fetched for comparison. + +## Phase 3: state and page assembly + +Goal: the page is interactive; controls drive the data. + +- Add atoms under `web/oss/src/state/analytics/` following + `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default + matching the observability windowing default), an agents-filter atom, a current-window + query atom, and a previous-window query atom (or one query returning both). Key every atom + on project id, time range, and the agents filter. Set `staleTime` to one minute and + `refetchOnWindowFocus: false`. +- Build the header controls: the time-range control and the Filters popover with the Agents + multi-select. Reuse the observability time windowing (the `Sort` control and its + `SortResult`) for the time range, so any window, including a custom start-and-end range, + works; do not build a fixed list of range options. Source the Agents options from the + agents atom confirmed in Phase 1. +- Wire loading and empty states following `AnalyticsDashboard.tsx` (antd `Spin`, a "No data" + empty state per card). + +Done when: changing the time range or the agents filter refetches and the page reflects it, +with correct loading and empty states. + +## Phase 4: summary panel and the four charts + +Goal: the full locked-scope UI. + +- Summary panel: a health donut component (recharts `RadialBarChart` or a small SVG ring) + with the band label and prose, plus four stat tiles. Each tile shows the value, a change + badge versus the previous window (green when the change is good for that metric, red + otherwise; note that lower latency and lower cost are good), and a sparkline of the current + window. +- Chart components, each a card with a title, a one-line description, a recharts chart, and a + toggleable legend: + - Runs: stacked bars, successful and failed. + - Latency: bars of average with a p95 reference line; tooltip shows average, p95, min, max. + - Costs: stacked bars, prompt and completion. + - Tokens: stacked bars, prompt and completion. +- Lay the four charts out in a two-column grid. All colors come from theme tokens and the + theme scale; verify both light and dark themes and hover, empty, and loading states. + +Done when: the four in-scope charts and the summary panel render correctly in both themes, +and `pnpm lint-fix` passes. + +## Phase 5 (deferred, after a backend change): Tools and Models + +Not part of this plan's scope. Recorded so Phase 1 to 4 leave the right boundary. The backend +work splits into two prerequisites that are **not co-equal**; they unlock different views, so +Phase 5 itself splits into 5a and 5b. See context.md for the verified engine details. + +### Prerequisite 1: span-focus wiring (unlocks 5a) + +Thread `focus` through to the base query so `focus = "span"` reads child spans. +`query.focus` reaches `dao.analytics()` but is never passed to `build_base_cte`, which +hardcodes `WHERE parent_id IS NULL`; make that predicate conditional on `focus`. + +- **Ship a guard with it**: under `focus = "span"`, cumulative metrics double-count (the + rollup lives on the root span). Reject or auto-map a cumulative spec under `focus = "span"`, + and have per-model cost/tokens use the `incremental` paths, not the cumulative ones. + +### Prerequisite 2: group-by dimension (unlocks only per-model cost, in 5b) + +Add a grouping dimension so a numeric metric (cost, tokens) splits by a categorical path +(model name) in one call. Prefer a **per-query** dimension over a per-spec `group_by`: it +matches "one view, one breakdown," leaves `MetricSpec` untouched, and confines the nested +`{group_value: stats}` output shape to one code path. This is the harder change; defer it +until 5a has shipped. + +### 5a: Tools and Models-share (needs prerequisite 1 only) + +Once span-focus lands, these need **no** group-by, because a `categorical/single` spec +already returns per-value frequencies: + +- Tools horizontal-bar card: `categorical/single` on `span_name` (+ `status_code` for error + rate), `focus = "span"`. +- Models horizontal-bar card with a click-to-filter legend: `categorical/single` on + `attributes.ag.meta.request.model`, `focus = "span"`. +- Models multi-select in the Filters popover. + +### 5b: per-model cost/tokens (needs prerequisites 1 and 2) + +Adds the numeric-by-model breakdown once the group-by dimension exists, using the +`incremental` cost/token paths under `focus = "span"`. + +### Frontend seam to leave now + +The request builder in Phase 2 takes `focus` and an arbitrary `specs` list without reshaping, +and the page grid can accept two more chart cards. 5a and 5b add queries and cards without +touching the four existing charts. + +Recommendation for the boundary: ship Phase 1 to 4 as a four-chart grid. Do not mount empty +Tools and Models cards in this release; a visible "coming soon" card ages badly. Land the +chart-card component as a reusable shell so Phase 5 only supplies data and series config. + +## Testing and verification + +- Follow `docs/designs/testing/README.md`. Add unit tests for the new mapper (pure + function: buckets in, dashboard shape out) and for the health-score computation. These need + no live database. +- Verify the four charts against one live project by running the local stack per the root + `AGENTS.md` local dev loop. +- Run `pnpm lint-fix` in `web` before committing. Do not commit during the planning phase. + +## Risks and unknowns to resolve during the build + +- Live-response validation of the metric shape: the nested `pcts.p95` reads correctly and the + prompt/completion split is non-zero on real traffic (resolve in Phase 2). The `type` strings + and the `pcts.p95` nesting are already verified in the backend code. +- The correct agents-list atom for the filter options (resolve in Phase 1). +- Whether the multi-agent reference filter uses `or` grouping or a single `in` with multiple + values in this dialect (resolve in Phase 2 against the existing filter builder). +- The two latency-score constants that set where "fast" and "slow" fall (confirm before + Phase 4). diff --git a/docs/design/agent-analytics/research.md b/docs/design/agent-analytics/research.md new file mode 100644 index 0000000000..b12cc1cafa --- /dev/null +++ b/docs/design/agent-analytics/research.md @@ -0,0 +1,152 @@ +# Research: what this feature reuses + +Every path below was read directly. The takeaway: the data path from endpoint to dashboard +already exists for the Observability page. This feature adds a new page on top of that +spine and reads a few response fields the current mapper drops. + +## The analytics fetch layer (reuse, extend by one field) + +`web/packages/agenta-entities/src/trace/api/api.ts` + +- `fetchSpansAnalytics(params)` calls `POST /spans/analytics/query` through the Fern client + (`getTracesClient().querySpansAnalytics`) and validates the response with + `analyticsResponseSchema`. Returns `null` on a non-2xx response or a shape mismatch. +- `SpansAnalyticsParams` today: `projectId`, `appId`, `focus` (default `trace`), `interval` + (bucket minutes), `oldest` / `newest` (ISO bounds), `filter` (a `{conditions: [...]}` + object, serialized to a JSON-string query param), `abortSignal`. +- It intentionally omits `specs`, so the backend applies its default set. To get the prompt + and completion split this page needs, add an optional `specs` field to + `SpansAnalyticsParams` and pass it through as a JSON-string query param, the same way + `filter` is passed. The endpoint already accepts `specs`; this is a frontend-only change. + +## The response-to-dashboard mapper (reuse pattern, new mapper for richer fields) + +`web/oss/src/services/tracing/lib/helpers.ts` + +- `analyticsToGeneration(analytics, range)` reduces the bucket list into + `GenerationDashboardData`. It reads these dotted metric paths, which match the backend's + default specs: + - `attributes.ag.metrics.costs.cumulative.total` (field `sum`) + - `attributes.ag.metrics.tokens.cumulative.total` (field `sum`) + - `attributes.ag.metrics.duration.cumulative` (fields `sum`, `count`) + - `attributes.ag.metrics.errors.cumulative` (field `sum`) + - `attributes.ag.type.trace` (field `count`) +- It derives run count from the `ag.type.trace` count, failures from the errors sum, success + as `total - failures`, and average latency as `durationSum / durationCount` in + milliseconds. +- `metricField(metrics, path, field)` is the safe reader for one **flat** numeric field + (`count`, `sum`, `min`, `max`). Reuse it for those. It reads one level deep, so it does + **not** reach percentiles; p95 lives at `metrics[path].pcts.p95`, one level further down. + Add a small nested accessor (or extend `metricField` with a two-key form) for the p95 read; + do not assume p95 is a flat sibling of `sum`. +- What it does not read, and this page needs: + - `costs.cumulative.prompt` and `costs.cumulative.completion` for the Costs split. + - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split. + - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for + the Latency tooltip and marker. +- `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar + count reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for + the time-range-to-interval mapping. + +`web/oss/src/services/tracing/api/index.ts` + +- `fetchGenerationsDashboardData(appId, options)` builds the `conditions` array (it pushes a + `references in [{id: appId}]` condition when an app id is present), computes the interval, + and calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new + page's fetch function. For project scope, omit the single-app reference condition and add + reference conditions only for the agents the user selects in the filter. + +## The dashboard state atoms (reuse pattern, new atoms for this page) + +`web/oss/src/state/observability/dashboard.ts` + +- `observabilityDashboardQueryAtom` is an `atomWithQuery` keyed on app id, project id, and + the time-range atom; it calls `fetchGenerationsDashboardData` with a `staleTime` of one + minute and `refetchOnWindowFocus: false`. +- `observabilityDashboardTimeRangeAtom` holds the selected range as a `SortResult`. +- `useObservabilityDashboard()` unwraps loading and fetching flags. +- The new page follows this exact shape with its own atoms: a time-range atom, an + agents-filter atom, a main query atom for the current window, and a second query atom (or + a widened single query) for the previous window that the change badges compare against. + +## Existing dashboard UI (reference, not reused directly) + +`web/oss/src/components/pages/observability/dashboard/` + +- `AnalyticsDashboard.tsx` renders `WidgetCard`s and `CustomAreaChart`s from the mapped + data, with a `Sort` time-range selector and antd `Spin` for loading. +- `CustomAreaChart.tsx` wraps recharts area charts. +- These render **area** charts with total figures. This page needs **stacked bar** charts, a + **horizontal bar** chart (for the deferred Tools view), sparklines, and a donut, so it + brings its own chart components rather than bending the area chart. It still follows the + same card-plus-chart composition and the same loading and empty-state conventions. + +## Charting library + +`web/oss/package.json` depends on **recharts `^3.1.0`**. Existing recharts usage to copy +style and theming from: + +- `web/oss/src/components/pages/observability/dashboard/CustomAreaChart.tsx` +- `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/BarChart.tsx` +- `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/HistogramChart.tsx` + +Build the page's charts with recharts (`BarChart` stacked and horizontal, `LineChart` or +`AreaChart` for sparklines, `RadialBarChart` or a small SVG ring for the donut). Do not port +the reference implementation's hand-rolled SVG chart code; it hardcodes hex colors and +duplicates what recharts gives. + +## Routing and sidebar + +- Pages live under `web/oss/src/pages/w/[workspace_id]/p/[project_id]/`. Existing folders: + `observability`, `evaluations`, `annotations`, `apps`, `settings`, and others. The + observability page file is a thin wrapper: + + ```tsx + import ObservabilityTabs from "@/oss/components/pages/observability" + const GlobalObservability = () => + export default () => + ``` + + Add `analytics/index.tsx` as the same kind of thin wrapper around a new + `components/pages/analytics` module. + +- The sidebar project items are defined in + `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`. The Observability entry + is the shape to copy: + + ```tsx + { + key: "app-observability-link", + title: "Observability", + link: `${projectURL}/observability`, + icon: , + disabled: !hasProjectURL, + } + ``` + + Add an `Analytics` entry with its own key, a `${projectURL}/analytics` link, and an icon + from `@phosphor-icons/react`. + +## The agents list for the filter + +The Agents multi-select needs the project's agents as options. The filter narrows the query +by pushing `references in [{id: }]` conditions (the same field the observability +fetch uses for a single app). Source the option list from the existing apps or workflows +state rather than a new endpoint. Confirm the exact atom during Phase 1; candidates are the +app-management or workflow molecule selectors already used by the sidebar's agent switcher. + +## Conventions that constrain the build + +From `web/AGENTS.md`: + +- All new API calls go through the Fern client and the per-resource accessors in + `@agenta/sdk/resources`. Keep zod validation at the boundary with `safeParseWithLogging`. +- Data fetching uses Jotai `atomWithQuery`; never `useEffect` with manual state. Put every + reactive dependency in the `queryKey`; set a sensible `staleTime`. +- Exactly one project is ever in scope. Do not write multi-project-defensive code. +- Styling is Tailwind utility classes plus antd semantic tokens (`bg-colorBgContainer`, + `text-colorText`, and the `--ag-color*` variables). No raw hex, no inline `style`, no + CSS-in-JS except for antd overrides Tailwind cannot express. Implement and verify both + light and dark themes. The reference palette maps onto these tokens; series colors come + from the theme scale, not from literals. +- Keep in-code comments to one line. diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md new file mode 100644 index 0000000000..083453f81e --- /dev/null +++ b/docs/design/agent-analytics/status.md @@ -0,0 +1,71 @@ +# Status + +Source of truth for progress. Update as work lands. + +## Current state + +Planning complete. No code written. The workspace holds the plan; implementation has not +started and no branch exists yet. + +## Locked decisions + +1. Frontend-first scope: four charts, four stat tiles, health donut, Agents filter, + time-range control. No `api/` change in this plan. +2. Health donut computed in the browser (0.72 x success rate + 0.28 x latency score). +3. New page at project scope named Analytics; the default query aggregates all project + agents. + +## Key finding from research + +The data path from the analytics endpoint to a mapped dashboard shape already exists for the +Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability +dashboard atoms). This feature reuses that spine. The only data-layer gaps for the in-scope +charts are: pass explicit metric specs to get the prompt-and-completion split, and read the +duration min, max, and p95 fields the current mapper drops. The endpoint already returns all +of these, so the frontend-first scope needs no backend change. + +## Resolved by the 2026-08-02 code verification pass + +- **Spec `type` strings**: every number metric is `numeric/continuous` (there is no bare + `numeric` in `MetricType`; the backend defaults confirm it). data-contract.md corrected. +- **p95 field**: nested at `metrics[path].pcts.p95`, not a flat field; `metricField` does not + reach it. data-contract.md and research.md updated with the nested-accessor requirement. +- **`specs` plumbing**: the Fern `QuerySpansAnalyticsRequest` type already carries + `specs?: string`, and the backend parses it from the query param, so passing specs is a + two-line entities-layer change, not new plumbing. + +## Open questions to resolve during the build + +- Agents-list atom for the filter options (Phase 1). +- Multi-agent reference filter grouping in the filter dialect (Phase 2). +- The two latency-score constants that set where "fast" and "slow" fall (Phase 4). +- Data-quality check (Phase 2): confirm the prompt/completion cost and token split is + non-zero on live traffic, and that `errors.cumulative` sum reads as "errors" not "failed + runs" in the UI copy. + +## Deferred to a later backend change + +Tools chart, Models chart, Models filter, per-model cost. Split into two sub-phases because +the two backend prerequisites are not co-equal gates (verified against the engine 2026-08-02): + +- **5a, Tools and Models-share**: needs only span-focus wiring (thread `focus` into + `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by needed, because a + `categorical/single` spec already returns per-value frequencies. Ship a guard: under + `focus = "span"`, cumulative metrics double-count, so cost/tokens must use `incremental` + paths. +- **5b, per-model cost/tokens**: additionally needs a group-by dimension. Harder; defer + until 5a ships. + +Open design decision for 5b: group-by as a **per-query dimension** (preferred) vs a per-spec +`group_by` field. Phase 5 in plan.md; not scheduled here. + +## Source materials + +- Decoded mockup: the artifact was unpacked to plain source. The page logic (data model, + charts, KPIs, health score) is the `Component` class; the layout is the `x-dc` template. + Original artifact: + `https://claude.ai/code/artifact/75b4f14e-9c9b-407b-9d35-317927fb6772`. +- Backend capability notes: `docs/design/agent-analytics/Note.md`. +- Endpoint architecture review (the source of the root-span-only and dead-`focus` findings): + a read-only review generated 2026-08-01; its conclusions are captured in context.md and + data-contract.md, so the workspace does not depend on the review file. From 1d8758469b3d801da53c4f5da66adc4e1ffc5f4d Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Sun, 2 Aug 2026 17:20:24 +0600 Subject: [PATCH 2/9] Update documentation for Agent Analytics page: clarify scope decisions, refine glossary terms, and enhance data contract details --- docs/design/agent-analytics/README.md | 24 +++++-- docs/design/agent-analytics/context.md | 18 +++-- docs/design/agent-analytics/data-contract.md | 44 +++++++++--- docs/design/agent-analytics/plan.md | 38 +++++----- docs/design/agent-analytics/research.md | 7 +- docs/design/agent-analytics/status.md | 76 ++++++++++++-------- 6 files changed, 140 insertions(+), 67 deletions(-) diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md index 8f447a127c..8ee3bd58e5 100644 --- a/docs/design/agent-analytics/README.md +++ b/docs/design/agent-analytics/README.md @@ -9,7 +9,7 @@ for it, so this work adds a page, not a new data layer. ## Reading order 1. **context.md** : why this page exists, what a user sees today, goals and non-goals, and - the three scope decisions that are already locked. + the scope decisions that are locked. 2. **research.md** : the parts of the codebase this feature reuses, with exact file paths: the analytics fetch layer, the response-to-dashboard mapper, the sidebar and routing, and the charting library. Read this before proposing any new file. @@ -23,13 +23,25 @@ for it, so this work adds a page, not a new data layer. ## Glossary -Terms used across these documents, defined once here. +Terms used across these documents, defined once here. This section is the workspace glossary; +a separate `CONTEXT.md` cannot live here because the filesystem is case-insensitive and would +collide with `context.md`. +- **Agent**: a configured AI agent in the project, the top-level thing a user builds, runs, + and analyzes. It is the unit this page aggregates over and the unit the Agents filter + narrows to. Not called application, app, workflow, or variant in this page's copy. - **Run**: one agent invocation. On the backend it is one root span (a span with no - parent). Run count per time bucket equals the count of the `ag.type.trace` metric. + parent). Run count per time bucket equals the count of the `ag.type.trace` metric. This + page says "run"; the Observability dashboard says "request" for the same metric, and the two + are allowed to diverge until Observability is aligned later. Not called request here. - **Span**: one unit of work inside a run (a model call, a tool call, or the agent step itself). Model name and tool name live on child spans, not on the root span. - **Root span**: the top span of a run. Today's analytics endpoint only reads root spans. +- **Failed run**: a run whose root span status is `ERROR`, a run-level outcome. Counted from + the `status_code` column, not from the errors metric. +- **Error**: any errored step inside a run. A run can contain errors and still succeed, so an + error count is not a failed-run count. +- **Success rate**: successful runs over total runs, where a failed run is the one above. - **Bucket**: one time slice of the chart x-axis (for example one day, or one hour). The endpoint returns one metrics object per bucket. - **Metric spec**: a request instruction of the form `{type, path}` that tells the endpoint @@ -39,5 +51,7 @@ Terms used across these documents, defined once here. (`trace`) or all spans (`span`). Today only `trace` works; see context.md. - **Project scope**: the web app always has exactly one project in context. This page lives at that level and, by default, aggregates every agent in the project. -- **Health score**: a single 0 to 100 number this page computes in the browser from success - rate and average latency. It is a display aid, not a backend metric. +- **Health score**: a single 0 to 100 number this page computes in the browser from the + success rate alone (latency was dropped because a fixed latency band mislabels slow-but- + healthy agents). The bands map directly to success: Healthy at 85 and above, Watch from 65 + to 84, At risk below 65. It is a display aid, not a backend metric. diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md index 393f789107..ace290612f 100644 --- a/docs/design/agent-analytics/context.md +++ b/docs/design/agent-analytics/context.md @@ -20,7 +20,7 @@ It shows: window, and a Filters popover with an Agents multi-select that narrows the query to chosen agents. The time-range control reuses the existing observability time windowing (the `Sort` control and its `SortResult`), so it supports the standard presets and a custom - start-and-end range. It is not a fixed list of options. + start-and-end range. It is not a fixed list of options. It opens on the last 7 days. - A summary panel: a health donut (0 to 100, with a Healthy / Watch / At risk band and a one-line read-out) and four stat tiles (Total runs, Success rate, Avg latency, Total cost). Each tile shows a change badge against the previous window of equal length and a small trend @@ -35,21 +35,27 @@ It shows: ## Locked scope decisions -Three decisions are settled and drive the plan. Do not reopen them without the requester. +These decisions are settled and drive the plan. Do not reopen them without the requester. 1. Frontend-first. Build the four charts above, the four stat tiles, the health donut, the Agents filter, and the time-range control against today's endpoint. The endpoint already returns every field these need, so this scope ships without any change under `api/`. -2. Health donut computed in the browser. The page derives the health score from - `0.72 x successRate + 0.28 x latencyScore`, where `latencyScore` maps average latency onto - a 0 to 1 range (higher is faster). The bands are Healthy at 85 and above, Watch from 65 to - 84, At risk below 65. This is a display aid; it is not sent to or stored on the backend. +2. Health donut computed in the browser. The health score is the success rate: + `round(100 x successRate)`, banded Healthy at 85 and above, Watch from 65 to 84, At risk + below 65. Latency does not factor in, because a fixed latency band mislabels agents that + are legitimately slow. Below a minimum run count the donut shows a neutral "Not enough runs + yet" state instead of a band. This is a display aid; it is not sent to or stored on the + backend. 3. New page at project scope. This is a net-new page named Analytics. It aggregates every agent in the project by default. The Agents multi-select narrows the set, so the default query carries no single-app reference filter. +4. A failed run means a run whose root span status is `ERROR`, a run-level outcome. Success + rate and the health score build on that, not on a count of errored steps. data-contract.md + has the definition and the query it needs. + ## Showing model usage and tool usage needs backend work Two views are out of scope for this plan because the backend cannot serve them yet. This is an diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md index d2b0b59c74..fdf219832a 100644 --- a/docs/design/agent-analytics/data-contract.md +++ b/docs/design/agent-analytics/data-contract.md @@ -41,7 +41,6 @@ serialized. The specs the page needs, by path and type: | Purpose | path | type | fields read | | --- | --- | --- | --- | | Run count | `attributes.ag.type.trace` | categorical/single | `count` | -| Failures | `attributes.ag.metrics.errors.cumulative` | numeric/continuous | `sum` | | Latency | `attributes.ag.metrics.duration.cumulative` | numeric/continuous | `count`, `sum`, `min`, `max`, `pcts.p95` | | Prompt cost | `attributes.ag.metrics.costs.cumulative.prompt` | numeric/continuous | `sum` | | Completion cost | `attributes.ag.metrics.costs.cumulative.completion` | numeric/continuous | `sum` | @@ -61,17 +60,36 @@ value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and `max` are flat read directly; only the percentiles sit one level down. Still confirm against one live response in Phase 2, but expect the nested shape. +### Failed runs come from a second, filtered query + +A failed run is a run whose root span `status_code` is `ERROR`. `status_code` is a +table column, and metric specs read only the `attributes` JSON (`build_extract_cte` extracts +`attributes #> path`), so no spec can target it. Instead, the page runs a second analytics +query for the same window with an added filter condition on `status_code` and reads the run +count: + +- Failed-run query filter: the agent conditions above (if any), plus + `{field: "status_code", operator: "eq", value: "ERROR"}`. `status_code` is a first-class + filter field (`api/oss/src/core/tracing/utils/filtering.py`). +- It needs only the run-count spec (`attributes.ag.type.trace`), so it is a cheap query. +- Per bucket: `failed` = the filtered run count; `success` = the unfiltered run count minus + it. + +The page therefore issues two queries per window (unfiltered for totals and latency/cost/ +tokens, status-filtered for failed runs), and it fetches a current and a previous window, so +four analytics calls in total. + ## Response fields the mapper reads The response is `{buckets: [{timestamp, metrics: {: {: value}}}]}`. The new mapper produces, per bucket: -- `success` = `type.trace` count minus `errors` sum, floored at zero. -- `failed` = `errors` sum. Note this is an **error count**, not strictly a failed-run count: - `errors.cumulative` rolls up every failing span in a run, so a single run with two failing - child spans contributes `2`. The floor-at-zero on `success` hides the resulting negative. - This matches the existing observability mapper's convention (`analyticsToGeneration`), so - it stays internally consistent; just read the label as "errors," not "failed runs." +- `failed` = the `type.trace` count from the status-filtered query (runs whose root span is + `ERROR`). This is a true failed-run count, never larger than the total. +- `success` = the unfiltered `type.trace` count minus `failed`. It cannot go negative, so no + flooring is needed. This departs from the existing observability mapper, which subtracts the + `errors.cumulative` sum. That sum counts errored steps, not failed runs, and can exceed the + run count, so this page uses the run-level status instead. - `latencyAvg` = duration `sum` / duration `count`, in milliseconds. - `latencyMin`, `latencyMax` = the flat `min` / `max` duration fields. - `latencyP95` = the **nested** `metrics[durationPath].pcts.p95` (see the note above the @@ -91,10 +109,14 @@ cost, plus the same figures for the previous window so the change badges have a Computed from the window totals, never sent to the backend: - `successRate` = successful runs / total runs. -- `latencyScore` = clamp01((3600 - avgLatencyMs) / 2800). Confirm these two constants before - implementing; they set where "fast" and "slow" fall. -- `health` = round(100 x (0.72 x successRate + 0.28 x latencyScore)). -- Band: Healthy at 85 and above, Watch from 65 to 84, At risk below 65. +- `health` = round(100 x successRate). The score is the success rate; latency does not factor + in, because a fixed latency band mislabels agents that are legitimately slow. +- Band: Healthy at 85 and above, Watch from 65 to 84, At risk below 65. The bands read + directly as success percentages. +- Low-traffic guard: below a minimum run count in the window, do not band the score. Show a + neutral "Not enough runs yet" donut instead, so a single failure in a quiet window does not + read as At risk. The threshold is a build-time tuning value (start around 20 runs). The stat + tiles and the four charts still render whatever data exists. ## Boundary for the deferred charts diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md index 364b377498..422dc5080f 100644 --- a/docs/design/agent-analytics/plan.md +++ b/docs/design/agent-analytics/plan.md @@ -34,12 +34,16 @@ Goal: the page can fetch mapped analytics for the current and previous windows. entities package still builds (`pnpm turbo run build --filter=@agenta/entities`). - Add a new mapper next to the existing one in `web/oss/src/services/tracing/lib/` (do not change `analyticsToGeneration`; the observability page depends on it). The new mapper reads - the split cost and token paths and the duration min, max, and p95 fields, and returns the - per-bucket and window-total shape in data-contract.md. Reuse `metricField` and - `calculateIntervalFromDuration`. + the split cost and token paths and the duration min, max, and p95 fields, combines the + unfiltered and status-filtered run counts into success and failed, and returns the per-bucket + and window-total shape in data-contract.md. Reuse `metricField` and + `calculateIntervalFromDuration`, and add a small `pcts` accessor for p95. - Add a fetch function in `web/oss/src/services/tracing/api/` that builds the project-scope - conditions (no single-app reference by default; one reference condition per selected agent), - passes the explicit `specs`, and calls `fetchSpansAnalytics`. + conditions (no single-app reference by default; the selected agents' reference conditions), + passes the explicit `specs`, and calls `fetchSpansAnalytics`. It issues two queries per + window: the unfiltered one for totals and latency/cost/tokens, and a second one that adds + `{field: "status_code", operator: "eq", value: "ERROR"}` and reads the run count as failed + runs (see data-contract.md). The failed-run query needs only the run-count spec. - The spec `type` strings (`numeric/continuous`) and the nested `pcts.p95` shape are already verified in the backend code (see data-contract.md); validate them against one live response while building this phase, and check the prompt/completion split reads non-zero on real @@ -56,11 +60,11 @@ window is fetched for comparison. Goal: the page is interactive; controls drive the data. - Add atoms under `web/oss/src/state/analytics/` following - `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default - matching the observability windowing default), an agents-filter atom, a current-window - query atom, and a previous-window query atom (or one query returning both). Key every atom - on project id, time range, and the agents filter. Set `staleTime` to one minute and - `refetchOnWindowFocus: false`. + `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default: the + last 7 days), an agents-filter atom, a current-window query atom, and a previous-window + query atom (or one query returning both). Each query atom drives the two calls per window + (unfiltered and status-filtered). Key every atom on project id, time range, and the agents + filter. Set `staleTime` to one minute and `refetchOnWindowFocus: false`. - Build the header controls: the time-range control and the Filters popover with the Agents multi-select. Reuse the observability time windowing (the `Sort` control and its `SortResult`) for the time range, so any window, including a custom start-and-end range, @@ -77,10 +81,11 @@ with correct loading and empty states. Goal: the full locked-scope UI. - Summary panel: a health donut component (recharts `RadialBarChart` or a small SVG ring) - with the band label and prose, plus four stat tiles. Each tile shows the value, a change - badge versus the previous window (green when the change is good for that metric, red - otherwise; note that lower latency and lower cost are good), and a sparkline of the current - window. + showing `round(100 x successRate)` with the band label and prose. Below the run-count floor + it renders the neutral "Not enough runs yet" state instead of a band. Plus four stat tiles; + each shows the value, a change badge versus the previous window (green when the change is + good for that metric, red otherwise; note that lower latency and lower cost are good), and a + sparkline of the current window. - Chart components, each a card with a title, a one-line description, a recharts chart, and a toggleable legend: - Runs: stacked bars, successful and failed. @@ -160,5 +165,6 @@ chart-card component as a reusable shell so Phase 5 only supplies data and serie - The correct agents-list atom for the filter options (resolve in Phase 1). - Whether the multi-agent reference filter uses `or` grouping or a single `in` with multiple values in this dialect (resolve in Phase 2 against the existing filter builder). -- The two latency-score constants that set where "fast" and "slow" fall (confirm before - Phase 4). +- Whether the runner marks a failed run's root span `status_code = ERROR`. The failed-run + count and the health score depend on it; validate on live traffic in Phase 2. +- The run-count floor for the neutral health state (tune in Phase 4; start around 20). diff --git a/docs/design/agent-analytics/research.md b/docs/design/agent-analytics/research.md index b12cc1cafa..17efe6a2c5 100644 --- a/docs/design/agent-analytics/research.md +++ b/docs/design/agent-analytics/research.md @@ -44,6 +44,10 @@ spine and reads a few response fields the current mapper drops. - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split. - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for the Latency tooltip and marker. +- This page does not reuse the existing mapper's `errors.cumulative`-based failure count. A + failed run is a run whose root span status is `ERROR`, which a metric spec cannot + read, so the page gets the failed-run count from a separate status-filtered query. See + data-contract.md. - `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar count reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for the time-range-to-interval mapping. @@ -54,7 +58,8 @@ spine and reads a few response fields the current mapper drops. `references in [{id: appId}]` condition when an app id is present), computes the interval, and calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new page's fetch function. For project scope, omit the single-app reference condition and add - reference conditions only for the agents the user selects in the filter. + reference conditions only for the agents the user selects in the filter. The new page also + issues a second query per window with a `status_code = ERROR` filter to count failed runs. ## The dashboard state atoms (reuse pattern, new atoms for this page) diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index 083453f81e..dded1310f5 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -4,49 +4,69 @@ Source of truth for progress. Update as work lands. ## Current state -Planning complete. No code written. The workspace holds the plan; implementation has not -started and no branch exists yet. +Planning complete, and a grilling session on 2026-08-02 sharpened the design. No code written. +The workspace holds the plan; implementation has not started and no branch exists yet. ## Locked decisions -1. Frontend-first scope: four charts, four stat tiles, health donut, Agents filter, - time-range control. No `api/` change in this plan. -2. Health donut computed in the browser (0.72 x success rate + 0.28 x latency score). -3. New page at project scope named Analytics; the default query aggregates all project - agents. +1. Frontend-first scope: four charts (Runs, Latency, Costs, Tokens), four stat tiles, health + donut, Agents filter, time-range control. No `api/` change in this plan. +2. New page at project scope named Analytics; the default query aggregates all project agents, + and the Agents filter narrows the set. +3. An **agent** is an application/workflow artifact. The Agents filter lists the project's + agents and narrows by `references`; no variant or environment granularity in v1. +4. The count of agent invocations is called a **run**. The Observability dashboard calls the + same metric a "request"; the two are allowed to diverge until Observability is aligned in a + later, separate change. +5. A **failed run** is a run whose root span `status_code` is `ERROR` (a run-level outcome, + not a count of errored steps). It comes from a second, status-filtered analytics query, not + a metric spec. +6. The **health score** is the success rate: `round(100 x successRate)`, banded Healthy 85+, + Watch 65 to 84, At risk below 65. Latency was dropped from it. Below a run-count floor the + donut shows a neutral "Not enough runs yet" state instead of a band. +7. The time-range control opens on the **last 7 days** and accepts any window via the + observability `Sort` control and `SortResult`. +8. The deferred **Tools** and **Models** views are omitted entirely in this release, not shown + as placeholders. The chart-card shell is built reusably so they drop in later. ## Key finding from research The data path from the analytics endpoint to a mapped dashboard shape already exists for the -Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability -dashboard atoms). This feature reuses that spine. The only data-layer gaps for the in-scope -charts are: pass explicit metric specs to get the prompt-and-completion split, and read the -duration min, max, and p95 fields the current mapper drops. The endpoint already returns all -of these, so the frontend-first scope needs no backend change. +Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability dashboard +atoms). This feature reuses that spine. The data-layer work is: pass explicit metric specs for +the prompt/completion split, read the duration min/max/p95 the current mapper drops, and add +the status-filtered failed-run query. The endpoint returns all of these, so the frontend-first +scope needs no backend change. -## Resolved by the 2026-08-02 code verification pass +## Resolved by code verification - **Spec `type` strings**: every number metric is `numeric/continuous` (there is no bare - `numeric` in `MetricType`; the backend defaults confirm it). data-contract.md corrected. + `numeric` in `MetricType`; `DEFAULT_ANALYTICS_SPECS` confirms it). - **p95 field**: nested at `metrics[path].pcts.p95`, not a flat field; `metricField` does not - reach it. data-contract.md and research.md updated with the nested-accessor requirement. + reach it, so the mapper needs a small `pcts` accessor. - **`specs` plumbing**: the Fern `QuerySpansAnalyticsRequest` type already carries - `specs?: string`, and the backend parses it from the query param, so passing specs is a - two-line entities-layer change, not new plumbing. + `specs?: string` and forwards it, so passing specs is a small entities-layer change. +- **Failed-run mechanism**: `status_code` is a table column and metric specs read only the + `attributes` JSON (`build_extract_cte`), so a spec cannot target it. Failed runs come from a + second query with a `status_code = ERROR` filter (`status_code` is a first-class filter + field). Four analytics calls total: unfiltered and status-filtered, for the current and the + previous window. ## Open questions to resolve during the build - Agents-list atom for the filter options (Phase 1). -- Multi-agent reference filter grouping in the filter dialect (Phase 2). -- The two latency-score constants that set where "fast" and "slow" fall (Phase 4). -- Data-quality check (Phase 2): confirm the prompt/completion cost and token split is - non-zero on live traffic, and that `errors.cumulative` sum reads as "errors" not "failed - runs" in the UI copy. +- Multi-agent reference filter encoding: a single `in` with all ids, or one condition per + agent combined with `or` (Phase 2, against the existing filter builder). +- Whether the runner marks a failed run's root span `status_code = ERROR`. The failed-run + count and health score depend on it; validate on live traffic (Phase 2). +- Live-response validation: the nested `pcts.p95` reads correctly and the prompt/completion + split is non-zero on real traffic (Phase 2). +- The run-count floor for the neutral health state (Phase 4; start around 20). ## Deferred to a later backend change Tools chart, Models chart, Models filter, per-model cost. Split into two sub-phases because -the two backend prerequisites are not co-equal gates (verified against the engine 2026-08-02): +the two backend prerequisites are not co-equal gates: - **5a, Tools and Models-share**: needs only span-focus wiring (thread `focus` into `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by needed, because a @@ -56,16 +76,16 @@ the two backend prerequisites are not co-equal gates (verified against the engin - **5b, per-model cost/tokens**: additionally needs a group-by dimension. Harder; defer until 5a ships. -Open design decision for 5b: group-by as a **per-query dimension** (preferred) vs a per-spec +Open design decision for 5b: group-by as a per-query dimension (preferred) vs a per-spec `group_by` field. Phase 5 in plan.md; not scheduled here. ## Source materials -- Decoded mockup: the artifact was unpacked to plain source. The page logic (data model, - charts, KPIs, health score) is the `Component` class; the layout is the `x-dc` template. +- Decoded reference implementation: the artifact was unpacked to plain source. The page logic + (data model, charts, KPIs) is the `Component` class; the layout is the `x-dc` template. Original artifact: `https://claude.ai/code/artifact/75b4f14e-9c9b-407b-9d35-317927fb6772`. - Backend capability notes: `docs/design/agent-analytics/Note.md`. -- Endpoint architecture review (the source of the root-span-only and dead-`focus` findings): - a read-only review generated 2026-08-01; its conclusions are captured in context.md and +- Endpoint architecture review (the source of the root-span-only and dead-`focus` findings): a + read-only review generated 2026-08-01; its conclusions are captured in context.md and data-contract.md, so the workspace does not depend on the review file. From ea530f55fe943f6c25a4892e0a51b6e0e765eef2 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Sun, 2 Aug 2026 17:22:37 +0600 Subject: [PATCH 3/9] Clarify agent filter description in Analytics documentation --- docs/design/agent-analytics/status.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index dded1310f5..6c41a4071a 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -13,8 +13,8 @@ The workspace holds the plan; implementation has not started and no branch exist donut, Agents filter, time-range control. No `api/` change in this plan. 2. New page at project scope named Analytics; the default query aggregates all project agents, and the Agents filter narrows the set. -3. An **agent** is an application/workflow artifact. The Agents filter lists the project's - agents and narrows by `references`; no variant or environment granularity in v1. +3. An **agent** is an application/workflow artifact. The Agents multi-select is the only + filter; it lists the project's agents and narrows by `references`. 4. The count of agent invocations is called a **run**. The Observability dashboard calls the same metric a "request"; the two are allowed to diverge until Observability is aligned in a later, separate change. From 61b43e9b66e1d7a22c2da3d0643fc1ee820e7dba Mon Sep 17 00:00:00 2001 From: Mahmoud Mabrouk Date: Sun, 2 Aug 2026 18:21:36 +0200 Subject: [PATCH 4/9] docs(analytics): add an evidence-backed capability review of the analytics backend The design workspace in this PR plans the Analytics page on the assumption that the backend already answers everything the first release needs, and that no backend change is required. This adds an independent review that tests that assumption against the running code. The review answers the two framing questions (how analytics works today, and how the three mounted routes and two implementations differ), then takes each wanted capability one at a time with a verdict, the exact request that produces it, and the live probe that proved or disproved it. It ends with measured latency, a scale projection, the frontend requirements, and a v1 and v2 proposal. Findings that change the plan: - Six of the fourteen wanted capabilities work today and mean what the wish list assumes. Four work only under a narrower label, such as "configured model" rather than the model that answered. Four are not available today at any price. - The query engine is more capable than the plan assumes. It summarizes any JSON path, returns 27 percentiles on any numeric metric, and returns a per-value frequency table per bucket on any categorical metric. Three capabilities the plan defers work today. - The plan's failed-run filter is rejected by the backend and returns an empty success, so the page would report perfect health on a project with a 9.5% failure rate. - The cost chart reads JSON paths that hold no data on either dataset measured. A different path does hold cost, and its coverage fell from about 70% of runs to near zero in mid-July on both datasets. The cause is unknown and is the top open item. - A 30-day query takes 1.7 seconds at today's volume, and a query killed by timeout returns HTTP 200 with an empty result rather than an error. Seven days costs 0.26 seconds and should be the default window. Evidence: every claim carries a file:line citation or a live probe result. Probes ran against two local development stacks; no production data was queried, so coverage percentages are unverified on production traffic. Raw probe payloads and query plans are not committed; Appendix A indexes them. Docs only. No code changes. Claude-Session: https://claude.ai/code/session_01RkWWQUNNzRbaB5jnCAdjYA --- docs/design/agent-analytics/README.md | 6 + .../agent-analytics/capability-review.md | 1853 +++++++++++++++++ 2 files changed, 1859 insertions(+) create mode 100644 docs/design/agent-analytics/capability-review.md diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md index 8ee3bd58e5..83000fb791 100644 --- a/docs/design/agent-analytics/README.md +++ b/docs/design/agent-analytics/README.md @@ -20,6 +20,12 @@ for it, so this work adds a page, not a new data layer. boundary where the deferred model and tool views drop in once the backend supports them. 5. **status.md** : current state, open questions, and decisions. This is the source of truth for progress; update it as work lands. +6. **capability-review.md** : an independent, evidence-backed review of what the analytics + backend can actually answer today, written after the four documents above. It tests each + wanted capability against the running code with live queries, measures how long those + queries take, and proposes a v1 and a v2. It contradicts the four documents above in + several places, so read it before building anything. Where the two disagree, the review + carries the evidence. ## Glossary diff --git a/docs/design/agent-analytics/capability-review.md b/docs/design/agent-analytics/capability-review.md new file mode 100644 index 0000000000..44da46f72b --- /dev/null +++ b/docs/design/agent-analytics/capability-review.md @@ -0,0 +1,1853 @@ +# What the analytics backend can answer today + +An evidence-backed capability review for the project Analytics page proposed in PR #5648. + +Written against commit `31c0781d42`. Every claim below carries either a `path/to/file:LINE` +citation or the result of a live HTTP call made on 2026-08-02. + +The raw evidence, meaning one saved request and response pair per probe plus the `EXPLAIN +(ANALYZE, BUFFERS)` query plans, came to 119 files and stayed out of the repository. The numbers +those files support are quoted inline wherever a claim depends on them. Appendix A lists every +probe by number and says what it tested, so anyone can reproduce a probe from the request shown +next to the claim it supports. + +--- + +## 1. The decision + +### 1.1 What we recommend + +**Do not build the six-chart page as PR #5648 specifies it.** Two of its six charts read JSON +paths that hold no data on either dataset we measured, and its failed-run filter is rejected by +the backend and returns an empty success, so the page would report 100% health forever on a +project with a 9.5% failure rate. + +**Do build a narrower page.** Six of the fourteen capabilities the repo owner listed work today +and mean what a reader would assume they mean. Four more work today but only under a narrower +label than the wish list uses, such as "configured model" rather than "model". Four are not +available at any price today. The narrower page needs no new backend capability, but it does +need three things PR #5648 does not have: a corrected failure filter, a default window of seven +days rather than thirty, and an explicit answer for what the page shows when a metric has no +data. + +**Settle two things before writing code.** First, what each metric means, because several of +today's queryable values are proxies rather than the thing itself. Section 7 lists the choices. +Second, why cost and the prompt/completion token split stopped being recorded in mid-July on both +datasets we measured. Until someone answers that, the cost tile has nothing to show. + +### 1.2 The four findings that drive that recommendation + +1. **The query engine is more capable than the PR assumes.** It summarizes any JSON path you + name, returns 27 percentiles on every numeric metric, and returns a per-value frequency table + per time bucket on every categorical metric. Three of the wish-list items the PR defers to a + later release work today, and three items the PR never mentions work today. +2. **The data pipeline is less healthy than the PR assumes.** The canonical cost paths hold no + data on any agent run. A different, unmapped path does hold cost, and its coverage fell from + about 70% of runs to near zero within a week in mid-July on both datasets. We measured the + collapse. We did not find its cause. +3. **Several queryable values are proxies, not the thing named.** The endpoint can return the + first *configured* tool of each run. It cannot return which tools ran. It can return the + author's model *alias*. It cannot return which model answered. Counting those as working + capabilities would flatter the plan. +4. **Performance, not capability, sets the limits.** The thirty-day view the shipped dashboard + already issues takes 1.7 seconds at today's volume, and a killed query returns HTTP 200 with + an empty result rather than an error. Seven days costs 0.26 seconds. Make seven days the + default. + +### 1.3 Where the numbers come from, and what that limits + +Every measurement here was taken on one of two local development stacks. + +| | Stack A | Stack B | +|---|---|---| +| Edition and image | EE, dev | OSS, dev | +| Tracing database | `agenta_ee_tracing` | `agenta_oss_tracing` | +| Root spans in the probe window | 7,529 | 9,720 | +| All spans in the probe window | 37,316 | 38,874 | +| Mean root-span attribute size | 10,011 bytes | 6,482 bytes | + +The probe window everywhere is `2026-07-01T00:00:00Z` to `2026-08-03T00:00:00Z`. + +**No production or cloud data was probed.** A read-only check against a hosted deployment +confirmed only that all three analytics routes are mounted; the credential available resolved to +a project with zero spans, so no data query was possible there. **Every coverage percentage in +this report therefore describes two local dev stacks and is unverified on production traffic.** + +Coverage claims are the ones most likely to differ elsewhere. Capability claims, meaning what the +query engine can and cannot compute, come from code and from live calls and hold on any dataset. +Performance claims come from single-user runs against a warm cache on one machine, so they +establish the shape of the cost curve, not a capacity number you can plan against. + +--- + +## 2. Vocabulary + +The rest of the report leans on these. Read them once. + +**Span.** One unit of recorded work, stored as one database row. A model call is a span. A tool +call is a span. The agent invocation itself is a span. + +**Trace.** All the spans of one run, linked by a shared trace id. + +**Root span.** The span in a trace with no parent (`parent_id IS NULL`). One agent run produces +exactly one root span, so counting root spans counts runs. **Child span** means any other span in +the trace. This distinction decides most of the report: the analytics endpoint reads root spans +only. + +**Configured, resolved, invoked.** Three different things that are easy to confuse. *Configured* +is what the agent's author wrote down, for example the model alias `haiku` and the list of tools +the agent is allowed to use. *Resolved* is what the system turned that into at run time, for +example the model id the provider actually served. *Invoked* is what the run actually did, for +example the three tools it called. Today's analytics endpoint can read configured values. It +cannot read resolved or invoked values. + +**Incremental and cumulative.** A metric written on one span for that span alone is +*incremental*. The same metric summed over a span and all its descendants is *cumulative*. +Ingest computes cumulative values by walking up the tree, so a cumulative value appears on every +ancestor. Summing a cumulative metric across all spans double-counts it. + +**Window.** The absolute from-and-to time range of the whole query, sent as `oldest` and +`newest`. + +**Period.** The bucket size the window is sliced into, sent as `interval`. The unit is +**minutes**, not seconds. The router docstring at +`api/oss/src/apis/fastapi/tracing/router.py:450-451` says seconds and is wrong. + +**Bucket.** One time slice of the result. The response returns one metrics object per bucket. + +**Fixed-duration bucket versus calendar bucket.** The backend buckets by fixed strides, for +example 1,440 minutes. A calendar day is not always 1,440 minutes long, because of daylight +saving transitions, and a calendar month is never a fixed number of minutes. The backend cannot +express calendar buckets. + +**Metric spec.** A request instruction of the form `{type, path}` that names a JSON path to +summarize and how to summarize it. The endpoint has no fixed metric list. It summarizes whatever +path you name. **Default specs** are the six the backend applies when a request sends none. + +**Coverage.** The share of rows on which a given path actually holds a value. A metric can be +perfectly expressible and still return nothing, because coverage is zero. + +**Cardinality.** The number of distinct values a categorical breakdown can return. Nothing in the +current code caps it. + +**Detoasting.** PostgreSQL stores large column values out of line, in a side table called TOAST. +Reading any part of such a value requires fetching and reassembling the whole thing. Every span's +`attributes` column is large enough for this to apply, and it is the single largest cost in every +analytics query. + +**Semantic convention.** The agreed naming scheme for span attributes. Agenta's namespace is +`ag.*`, for example `ag.metrics.duration.cumulative`. Attributes arriving under a foreign +namespace, such as the OpenTelemetry GenAI convention `gen_ai.*`, stay verbatim as sibling keys +unless an adapter maps them into `ag.*`. + +--- + +## 3. How analytics works today + +### 3.1 Storage: two kinds of column, and why it matters + +Analytics reads one table, `spans`, in a **separate analytics database** +(`AnalyticsEngine`, `api/oss/src/dbs/postgres/shared/engine.py:110-124`, wired at +`api/entrypoints/routers.py:534-543`). The schema is at +`api/oss/src/dbs/postgres/tracing/dbas.py:9-97` and splits into two kinds of storage: + +- **Columns.** `project_id`, `trace_id`, `span_id`, `parent_id`, the enums `trace_type`, + `span_type`, `span_kind`, `status_code`, the timestamps, `created_by_id`, and the root-only + promoted `session_id` / `user_id` / `agent_id`. +- **JSONB.** `attributes`, `references`, `links`, `hashes`, `events`. + +**Metric specs read the `attributes` JSONB only. Every column is filter-only.** The extract stage +projects `attributes #> path` and nothing else +(`api/oss/src/dbs/postgres/tracing/utils.py:1106-1126`). Verified live: specs on `span_name`, +`status_code`, `attributes.span_name` and `attributes.status_code` all returned zero buckets +(probe 37). + +That one fact explains two later findings. Success and failure need a second filtered query +rather than one grouped query, and per-user numbers are unreachable even though the user id sits +in a column on every row. + +### 3.2 How one run becomes rows + +One agent run produces **one trace, two OTLP batches, one root span**. The batch boundary is +where cost goes missing, so it is worth a picture. + +``` + one trace, one trace_id + ┌───────────────────────────────────────────────────────────────────┐ + │ │ + │ BATCH 1 (exported by the SDK) │ + │ ┌─────────────────────────────────────────────┐ │ + │ │ ROOT SPAN type=workflow │ <- the only row │ + │ │ ag.data.parameters.* (the agent config) │ analytics │ + │ │ ag.metrics.duration.cumulative │ can read │ + │ │ gen_ai.usage.* (stamped by record_usage) │ │ + │ └─────────────────────────────────────────────┘ │ + │ │ + │ BATCH 2 (exported by the runner, linked by traceparent) │ + │ ┌─────────────────────────────────────────────┐ │ + │ │ invoke_agent ag.meta.skills.loaded │ │ + │ │ turn 1 │ invisible to │ + │ │ chat ag.meta.request.model │ the analytics │ + │ │ cache-token counts │ endpoint │ + │ │ execute_tool ag.meta.tool.name │ │ + │ └─────────────────────────────────────────────┘ │ + └───────────────────────────────────────────────────────────────────┘ +``` + +**Batch one is the SDK.** The `/invoke` handler `_agent` +(`sdks/python/agenta/sdk/agents/handler.py:252`) runs inside the tracing decorator. As the +outermost recording span it is forced to type `workflow` +(`sdks/python/agenta/sdk/decorators/tracing.py:337-339`). That is the root span. + +**Batch two is the runner.** The root span's `traceparent` is injected into the harness config +(`sdks/python/agenta/sdk/agents/tracing.py:41-77`), and the runner exports its own tree in a +separate request. + +Two consequences from that shape decide half the wish list. + +**First, the agent's configuration is queryable on the root span.** The decorator writes the +resolved agent config as `ag.meta.configuration` +(`sdks/python/agenta/sdk/decorators/tracing.py:548-556`), and ingest relocates the whole blob to +`ag.data.parameters` (`api/oss/src/core/tracing/utils/attributes.py:250-256`). Harness, +configured model, provider, configured tools and configured skills are therefore a JSON subtree +on the run row. None of PR #5648's documents mention this, and it is the single most useful fact +in the report. + +**Second, the cost roll-up never reaches the root span.** Ingest computes the roll-up per OTLP +request (`parse_span_idx_to_span_id_tree`, `api/oss/src/core/tracing/utils/trees.py:174-193`). +The runner's spans have a remote parent, so they never join the SDK batch's tree. Confirmed by +data: `ag.metrics.costs.cumulative` is present on **zero of 48,005 spans** on stack A. Section +4.4 item 4 covers what does work instead. + +The child spans in batch two carry exactly what the wish list wants for tools and models: +`ag.meta.tool.name` on each `execute_tool` span (`logfire_adapter.py:172`, emitted at +`services/runner/src/tracing/otel.ts:787`), and `ag.meta.request.model` on each `chat` span +(`logfire_adapter.py:150`). Both are invisible to the analytics endpoint, for the reason in +section 3.5. + +### 3.3 The three endpoints and the two implementations + +The repo owner remembered two analytics endpoints. There are three mounted routes plus two hidden +aliases, served by two implementations. + +| Mounted path | operation_id | OpenAPI tag | Deprecated | Registered at | +|---|---|---|---|---| +| `POST /tracing/spans/analytics` | `fetch_legacy_analytics` | Legacy | no | `router.py:108-116` | +| `POST /tracing/analytics/query` | `query_analytics` | Deprecated | **yes** | `router.py:118-127` | +| `POST /spans/analytics/query` | `query_spans_analytics` | Traces | no | `router.py:955-962` | + +All three are in `api/oss/src/apis/fastapi/tracing/router.py`. Two hidden `preview` aliases also +exist (`api/entrypoints/routers.py:1146-1153` and `:1194-1200`). + +**Routes 2 and 3 are the same implementation.** Both call `dao.analytics` with the same parsing. +Verified live: the same request body to both returned an identical payload (`count: 7529`, +probe 0b). Route 2 is marked `deprecated=True` and has no caller in `web/`. + +**Route 1 is a different implementation** (`dao.legacy_analytics`, +`api/oss/src/dbs/postgres/tracing/dao.py:529-736`). It computes four fixed metrics with two +hand-written aggregate queries. + +Side by side: + +| | Legacy `/tracing/spans/analytics` | Specs-driven `/spans/analytics/query` | +|---|---|---| +| Metrics | Fixed four (count, duration, costs, tokens), each split into total and errors (`dao.py:554-607`) | Arbitrary, driven by `specs` | +| `focus=span` | **Works** (`dao.py:566-570`, `:680-688`) | **Accepted, echoed, ignored** (`utils.py:1073`) | +| "Errors" means | Spans carrying an `exception` event, hard-coded (`dao.py:659-676`) | Whatever you ask for; `status_code` is unreachable as a metric | +| Empty buckets | Zero-filled across the window (`dao.py:706-711`) | Omitted entirely | +| Percentiles, min, max, histograms | None | Yes, on every numeric spec | +| Category breakdowns | None | Yes, per value per bucket | +| Speed on the same data | 0.29 s for a 30-day daily query | 0.52 s for one spec, 1.92 s for eight | +| Rate limit on EE cloud | 180-request burst, then 1 request per minute per organization | Falls to the general bucket: 480/min free, 1,440/min paid | +| Callers | None in `web/`; still in both generated clients | The only path the product uses | + +### 3.4 Do both work? Yes, with one caveat each + +**Both execute and both return correct numbers.** Live calls to both succeeded on both stacks. + +The legacy endpoint has a latent defect worth one sentence and no bug report. The condition that +defines its `errors` half is added inside `if query.filtering:` +(`api/oss/src/dbs/postgres/tracing/dao.py:645-676`), so a call with `filtering=None` would leave +`errors` un-narrowed and equal to `total`. The HTTP surface cannot reach that state in practice: +`merge_queries` injects an empty `Filtering()` object whenever either the query params or the +body parse (`api/oss/src/core/tracing/service.py:343-355`), and an empty `Filtering()` is truthy, +so the branch always runs. Verified live: an unfiltered legacy call returned `total.count 11` and +`errors.count 2` for the same day, and adding a filter changed nothing (probe 18). One corner +remains untested: a completely bare POST with no query params and no body, which is the only +shape that could produce `filtering=None` (`service.py:333-341`). One curl would settle it. + +The specs-driven endpoint has a bigger caveat, and it is the one that bites: **every failure +returns HTTP 200 with an empty result.** Section 3.8 covers all four failure modes. + +### 3.5 The query, and its three hard constraints + +`TracingDAO.analytics` (`api/oss/src/dbs/postgres/tracing/dao.py:305-527`) composes one +statement: + +1. Strip the `attributes.` prefix from each spec path (`dao.py:346-352`). A spec path is a JSONB + path relative to the `attributes` column. +2. `build_specs_values` (`utils.py:1008-1039`) turns the spec list into a SQL `VALUES` relation + with `path = spec.path.split(".")`. +3. `build_base_cte` (`utils.py:1042-1103`) selects the rows: project scope, the `created_at` + window, `date_bin(stride, created_at, oldest)` bucketing, external filters, and + `.where(SpanDBE.parent_id.is_(None))`. +4. `build_extract_cte` (`utils.py:1106-1126`) cross-joins those rows against the spec relation + and projects `attributes #> path AS jv`. +5. `build_statistics_stmt` (`utils.py:1129-1164`) unions the per-type reducer families. +6. Python assembles the metric blob (`dao.py:426-484`). + +**Constraint one: root spans only.** The `parent_id IS NULL` predicate at +`api/oss/src/dbs/postgres/tracing/utils.py:1073` is unconditional, and `dao.analytics` never +passes `query.formatting` to `build_base_cte` (`dao.py:378-387`). The endpoint accepts `focus`, +echoes it back, and ignores it. Verified live with hard proof: two calls identical except for +`focus=trace` versus `focus=span`, on a duration spec, returned **byte-identical** payloads +(count 7,530, sum 88,193,670.749 both times), while the echoed `query.formatting.focus` reported +whichever value was sent (probe 13b). PR #5648 identifies this correctly and names the right fix +location. + +**Constraint two: no grouping key but time.** Every reducer groups by `(timestamp, idx)` and +nothing else (`utils.py:1205, 1228, 1259, 1285, 1671, 1701, 1723`). You can get counts per +category, and you can get statistics per time bucket. In one request you cannot cross them. + +**Constraint three: specs read `attributes` only** (`utils.py:1120`), as covered in 3.1. + +### 3.6 What each spec type returns + +`MetricSpec` is `{type, path, bins, vmin, vmax, edge}` +(`api/oss/src/core/tracing/dtos.py:252-259`). The `type` picks the reducer family: + +| `type` | What comes back | +|---|---| +| `numeric/continuous` | `count`, `sum`, `mean`, `min`, `max`, `range`, `pcts` (**27** percentile levels, p00.05 to p99.95, including p95 and p99), `iqrs`, `pscs`, `hist` | +| `numeric/discrete` | The above plus a `freq` table and `uniq` | +| `categorical/single` | `count` plus `freq` (an array of `{value, count, density}`) plus `uniq` | +| `categorical/multiple` | The same, over a JSON array **of strings** | +| `binary` | `count` plus a true/false `freq` | +| `string`, `json` | `count` only | +| `none`, `*` | Nothing. No reducer family exists | + +The percentile levels are at `api/oss/src/dbs/postgres/tracing/utils.py:924-955`. There are +**27** of them, confirmed by counting the dict keys in a live response (probe 16a). + +The caller cannot ask for a subset. A `numeric/continuous` spec always computes the count, the +basic statistics, all 27 percentiles and a histogram, whatever the page needs +(`utils.py:1211-1215`, `:1263-1287`). + +**Every family type-gates its rows.** `numeric/continuous` counts only rows where +`jsonb_typeof(jv) = 'number'` (`utils.py:1187`); `categorical/single` only `'string'` +(`utils.py:1660`); `categorical/multiple` requires an array whose elements are strings +(`utils.py:1746-1749`). A spec pointed at the wrong JSON type contributes zero rows and reports +no error. This is why `…agent.harness.kind`, a string, works and `…agent.harness`, an object, +does not. + +### 3.7 Bucketing + +Rules, from `_get_stride` and `_get_interval` (`utils.py:830-876`): + +- `interval` is in minutes. 1440 is a day, 10080 is a week. +- **Omitting `interval` collapses the window into one exact bucket** (`utils.py:838-841`). This + is the cheapest way to get a true window-level total or percentile, and PR #5648 never uses it. +- Bucket edges are fixed-width offsets from `oldest`, computed by `date_bin` + (`utils.py:1051-1055`), always in UTC, keyed on `created_at`, which is ingest time rather than + the span's start time (`parse_windowing`, `utils.py:892-921`). +- `_MAX_ALLOWED_BUCKETS = 1024` (`utils.py:808`). Above it the stride is **silently widened** + (`utils.py:848-859`). Verified live: a 7-day window with `interval=1` returned 627 buckets each + reporting `"interval": 15` (probe 15). +- Buckets with no rows are **omitted** by the specs-driven endpoint. The frontend must build its + own x-axis. + +### 3.8 Four ways the endpoint fails quietly + +All four verified live, all producing plausible wrong numbers rather than errors. + +1. **An unknown filter field is logged and dropped, not rejected** + (`api/oss/src/core/tracing/utils/filtering.py:542-546`). A typo **widens** the result. + Verified: a filter on `{"field": "environment", "operator": "is", "value": "production"}` + returned 7,529, identical to unfiltered (probe 11b). +2. **An invalid operator or value raises `FilteringException`, which becomes an empty 200.** The + handler is decorated `@suppress_exceptions(default=AnalyticsResponse(), exclude=[HTTPException])` + (`router.py:1276`), and `FilteringException` is not an `HTTPException`. Verified: the filter + PR #5648 specifies returned HTTP 200 with `buckets: []` in 0.016 seconds (probe 5). +3. **A malformed `oldest` or `newest` silently substitutes the default 30-day window.** + `parse_query_from_body_request` catches everything + (`api/oss/src/apis/fastapi/tracing/utils.py:136-158`) and `parse_windowing` then applies + `_DEFAULT_TIME_DELTA` (`utils.py:807`, applied at `:909-915`). Verified: `"oldest": + "not-a-date"` returned 7,519 instead of 7,529, with `query.windowing` echoing `{}` (probe 39). +4. **A query killed by the statement timeout also becomes an empty 200.** Section 5.5. + +**Do not trust the echoed query.** The response echoes back the resolved `query` and `specs` +(`router.py:1311-1315`), and both earlier research passes recommended reading it as the standard +debugging move. It reports **what you asked for, not what ran**. With `interval=1` the echo says +1 while the buckets ran at a coarsened stride, and `query.formatting.focus` echoes `span` on an +endpoint that ignores focus. The only field that reports what actually ran is +`buckets[].interval`. + +### 3.9 A worked example + +Two real calls, taken from the saved artifacts. Project and account identifiers are redacted. + +**Call one: run count, latency and percentiles for a whole window, in one bucket.** Omitting +`interval` gives exact window-level numbers that cannot be derived from per-bucket data. + +```json +POST /api/spans/analytics/query?project_id= +{ + "oldest": "2026-07-01T00:00:00+00:00", + "newest": "2026-08-03T00:00:00+00:00", + "specs": [ + {"type": "numeric/continuous", "path": "attributes.ag.metrics.duration.cumulative"} + ] +} +``` + +Response, abridged; `pcts` has 27 keys, and `iqrs`, `pscs` and `hist` are omitted here: + +```json +{ + "count": 1, + "buckets": [{ + "timestamp": "2026-07-01T00:00:00Z", + "interval": 47520, + "metrics": { + "attributes.ag.metrics.duration.cumulative": { + "type": "numeric/continuous", + "count": 7529, + "sum": 88185838.233, + "mean": 11712.822, + "min": 0.77, + "max": 465914.479, + "range": 465913.709, + "pcts": {"p50": 4090.6, "p95": 25777.31, "p99": 60184.84} + } + } + }], + "query": {...}, "specs": [...] +} +``` + +Read that as 7,529 runs, 11.7 seconds average, 25.8 seconds at p95. Duration is in milliseconds. +`count` at the top of the response is the **number of buckets**, not the number of runs +(`router.py:1310`). Anything reading the top-level `count` as a run total is wrong. + +**Call two: harness usage per day.** A `categorical/single` spec returns a full frequency table +per bucket, which is a per-value breakdown in one request. + +```json +POST /api/spans/analytics/query?project_id= +{ + "oldest": "2026-07-01T00:00:00+00:00", + "newest": "2026-08-03T00:00:00+00:00", + "interval": 1440, + "specs": [ + {"type": "categorical/single", + "path": "attributes.ag.data.parameters.agent.harness.kind"} + ] +} +``` + +Response, first bucket of 30: + +```json +{ + "count": 30, + "buckets": [{ + "timestamp": "2026-07-03T00:00:00Z", + "interval": 1440, + "metrics": { + "attributes.ag.data.parameters.agent.harness.kind": { + "type": "categorical/single", + "count": 11, + "freq": [ + {"value": "claude", "count": 10, "density": 0.90909}, + {"value": "pi_core", "count": 1, "density": 0.09091} + ], + "uniq": ["claude", "pi_core"] + } + } + }] +} +``` + +Artifacts: `p16a-no-interval-duration.*` and `p08a-harness-kind.*`. + +### 3.10 The chart layer today + +The web app has one analytics call path, four files deep: + +``` +AnalyticsDashboard.tsx + └── useObservabilityDashboard() web/oss/src/state/observability/dashboard.ts:57 + └── observabilityDashboardQueryAtom dashboard.ts:22 + └── fetchGenerationsDashboardData web/oss/src/services/tracing/api/index.ts:19 + ├── analyticsToGeneration web/oss/src/services/tracing/lib/helpers.ts:106 + └── fetchSpansAnalytics + web/packages/agenta-entities/src/trace/api/api.ts:323 +``` + +It calls route 3 with everything in the query string and **never sends `specs`** (`api.ts:343`), +so the backend applies its six defaults (`DEFAULT_ANALYTICS_SPECS`, +`api/oss/src/core/tracing/service.py:91-98`). The mapper reads five dotted paths and one flat +field through `metricField` (`helpers.ts:88-92`), which reads **one level deep and numeric +only**. Everything nested, meaning `pcts`, `freq`, `hist` and `iqrs`, is invisible to the app +today, not because the backend withholds it but because nobody reads it. + +--- + +## 4. The wish list, capability by capability + +The repo owner asked about fourteen capabilities: + +1. Number of runs, per window, grouped by period (day, week, month). +2. Average latency, per window, grouped by period. +3. Latency maximum, minimum and 95th percentile. +4. Cost, split into prompt, completion and cache, with the same window and period. +5. Tokens, split into prompt, completion and cache, with the same window and period. +6. Run success and failure counts, with the same window and period. +7. Tool usage aggregated per period: which tools ran, how often. +8. Model usage aggregated per period. +9. Harness usage aggregated per period. +10. Filtering by agent. +11. Filtering by model. +12. Filtering by harness. +13. Filtering by the user who made the call, and numbers per user. +14. Skills used. + +### 4.1 How to read the table + +A single "does it work" column would mislead, because four different questions hide inside it. +The table below answers all four separately. + +- **Expressible?** Can today's query engine produce this number at all, from any request? +- **What it means.** Is the number the thing the wish list names, or a proxy standing in for it? + A proxy can still be worth charting, but it must be labelled honestly in the UI. +- **Coverage.** What share of runs actually carry the data, on the two datasets we measured. +- **Ready to ship?** Would a user reading this chart draw a correct conclusion. + +### 4.2 The summary table + +| # | Capability | Expressible? | What the number means | Coverage measured | Ready to ship? | +|---|---|---|---|---|---| +| 1 | Runs per window, per period | Yes, in fixed-duration buckets. Calendar months are not expressible | Root spans in the window. Includes annotation traces unless filtered | 100% | **Yes**, with a `trace_type` filter and an honest bucket label | +| 2 | Average latency per period | Yes | Mean root-span wall clock, milliseconds | 8,817 of 8,820 roots (A) | **Yes** | +| 3 | Latency min, max, p95 | Yes. 27 percentiles ship on every numeric spec | Exact percentile over root durations. Window-level p95 needs a separate no-interval call | Same as 2 | **Yes** | +| 4 | Cost total | Yes, but only at `attributes.gen_ai.usage.cost` | The harness's own reported total for the run. The canonical `ag.metrics.costs.*` paths are empty on agent roots | 2.0% (A), 68.6% (B), both near zero after mid-July | **No.** Blocked on the coverage investigation | +| 4b | Cost split prompt / completion | Yes in principle. Ingest computes it | Modelled at ingest on LLM child spans and rolled up within one batch. For agent runs it never reaches the run's root span | 0 of 48,005 spans (A) | **No** | +| 4c | Cost split for cache | No | Not modelled anywhere in the cost shape | n/a | **No** | +| 5 | Tokens total | Yes | Harness-reported total tokens for the run | 5,842 of 7,529 (A) | **Yes**, if the tile states its coverage | +| 5b | Tokens split prompt / completion | Yes | Same field family as the total | Zero on A, real on B, collapsed alongside cost | **No.** Gate on coverage | +| 5c | Cache tokens | No. Child spans only | Cache reads and writes per model call | 444 spans, 0 roots (A) | **No** | +| 6 | Success versus failure | Yes, with a corrected filter | Root span status. Blind to failures inside an otherwise clean run, measured at 1.2% of runs | 100% | **Yes**, if you write down that failure means root-span error | +| 7 | Tool usage per period | Only the **first configured** tool, by array index | Configuration order, not usage. Which tools ran is on child spans | index 0 present on 7,221 roots; invoked tool names on 7,587 non-root spans | **No** as "tool usage" | +| 8 | Model usage per period | Yes, for the configured alias | The author's alias, not the model that answered. A run can call several models | 7,345 of 7,529 (A) | **Proxy only.** Label it "configured model" | +| 9 | Harness usage per period | Yes | The configured harness kind, one per run, so this is exact | 8,765 of 8,809 roots (A) | **Yes.** Absent from PR #5648 | +| 10 | Filter by agent, and per-agent breakdown | Yes, both | Agent identity from `references`. Two naming families must be unioned | 7,241 of 7,529; 288 roots carry none | **Yes** | +| 11 | Filter by model | Yes, for the configured alias | Same proxy as item 8 | Same as 8 | **Yes**, with the alias label | +| 12 | Filter by harness | Yes | Configured harness kind | 7,056 of 7,529 | **Yes.** Absent from PR #5648 | +| 13 | Filter by user, and numbers per user | Filter yes, breakdown no. Specs cannot read columns | `created_by_id` is the credential owner, not an end user | Exactly one distinct value per project on both stacks | **No** | +| 14 | Skills used | Only the **first configured** skill, by array index | Configuration order. Invoked skills are recorded nowhere | index 0 present on 491 roots | **No** | + +**Counting honestly:** six capabilities are ready and mean what the wish list says (1, 2, 3, 9, +10, 12). Four more are ready under a narrower label (5, 6, 8, 11). Four are not available (4, 7, +13, 14), and within item 5 the split and the cache lines are not available either. + +Three dimensions nobody asked for also work today and cost one spec each: runs per connection +mode, runs per default permission, and runs by streaming flag. Appendix B has the numbers. + +### 4.3 The recurring wall, stated precisely + +Six verdicts above reduce to one limitation. + +**Works today in one call:** a count breakdown per attribute value, per time bucket. Runs per +harness per day, per model, per agent, per skill, per connection mode. This is what +`categorical/single` returns, and it needs no backend change. + +**Works today at one call per value:** a numeric statistic *for* a given value, through a deep +JSON filter. Average or p95 latency for `haiku`, then for `sonnet`, and so on. The value list +comes from one no-interval call, whose `uniq` array is the complete distinct-value list for the +window. So the pattern is one call to learn the values plus N calls to measure them. Six models +issued in parallel measured 0.68 seconds (section 5.3). + +**Does not work in a single request:** a numeric statistic split by a category. "Sum of cost by +model", "p95 latency by tool". Verified live: a request with a duration spec and a harness spec +returns both, but the duration statistics cover all 7,529 rows with no split by harness (probe +27). There is no grouping dimension anywhere in `api/oss/src/core/tracing/` today. Section 8.4 +covers what adding one should look like, and why the obvious shape is not the right one. + +### 4.4 The detail, item by item + +Each item names the probe that tested it. Appendix A indexes every probe by number. + +--- + +#### 1. Number of runs per window, grouped by period + +**Ready to ship, in fixed-duration buckets. Calendar months are not expressible.** + +**How.** One `categorical/single` spec on `attributes.ag.type.trace`, with `interval` set to 1440 +for days or 10080 for weeks, plus a `trace_type is invocation` filter. Read +`buckets[].metrics["attributes.ag.type.trace"].count`. + +**Live test.** Probe 0 returned `count 7529` for the probe window on stack A, and a direct SQL +count of root spans over the same window returned **exactly 7,529**. So the analytics run count +provably equals the root-span count. Probes 4a to 4c then showed the failure split summing +exactly: 7,529 unfiltered, 715 errored, 6,814 not errored. + +**Semantic conventions.** `ag.type.trace` is stamped on every span of every trace by +`api/oss/src/core/tracing/utils/trees.py:114-140`, with values `invocation` or `annotation`. +Nothing new needs recording. + +**Three caveats.** + +- **The run count includes annotation traces** unless you filter. Annotations are evaluator runs + and human annotations, not agent runs. The magnitude is small on our datasets, 3 annotation + roots of 8,820 on stack A, but PR #5648's own glossary defines a run as an invocation and then + specifies a metric that counts both. Either add the `trace_type` filter, or read the `freq` + array, which already carries the split. +- **Calendar months are not expressible.** `_get_interval` (`utils.py:862-876`) maps minutes, + hours, days and weeks only. A 30-day stride is not a month. The workaround is one no-interval + call per calendar month, so twelve calls for a year. +- **A "day" is 1,440 minutes, not a calendar day.** The frontend can align `oldest` to the + viewer's local midnight, and that is worth doing. Verified live: moving `oldest` from + `00:00+00:00` to `00:00+02:00` moved every bucket boundary from `T00:00:00Z` to `T22:00:00Z` + and re-shuffled the counts (probe 17). But `date_bin` steps by a fixed duration + (`utils.py:1051-1055`), so across a daylight saving transition the buckets drift off local + midnight by an hour for the rest of the window. Label the axis as 24-hour periods, or add + timezone-aware boundaries in the backend. Do not call fixed-stride buckets calendar days. + +--- + +#### 2. Average latency per window, grouped by period + +**Ready to ship. No backend work.** + +**How.** Add `{"type": "numeric/continuous", "path": "attributes.ag.metrics.duration.cumulative"}` +to the same request. Per bucket, `mean` comes back directly. For a window average, do **not** +average the per-bucket means. Sum the `sum` fields and divide by the summed `count` fields, or +issue the no-interval call and read `mean` exactly. + +**Live test.** Probe 16a, whole probe window, one bucket: `count 7529`, `sum 88185838.233`, +`mean 11712.82`, `min 0.77`, `max 465914.479`. + +**Semantic conventions.** `ag.metrics.duration.cumulative` is not a roll-up despite the name. +Ingest overwrites it with the span's own wall-clock duration in milliseconds +(`api/oss/src/core/tracing/utils/parsing.py:296-306`). Coverage is effectively total, 8,817 of +8,820 roots on stack A. Units are milliseconds, and the existing frontend mapper documents a past +bug where they were divided by 1000 (`helpers.ts:132-135`). + +--- + +#### 3. Latency maximum, minimum and 95th percentile + +**Ready to ship. No backend work of any kind. PR #5648 treats this as new work.** + +**How.** The same `numeric/continuous` spec as item 2. `min` and `max` are flat fields. +Percentiles are **nested**: `metrics[path].pcts.p95`. All 27 levels ship on every +`numeric/continuous` spec, always, computed by +`percentile_cont(ARRAY[...]) WITHIN GROUP (ORDER BY value)` (`utils.py:1266-1287`, levels at +`:924-955`). + +**For a window-level p95, omit `interval`.** Percentiles do not compose across buckets. Averaging +or taking the maximum of per-bucket p95s is wrong for any non-uniform distribution. + +**Live test.** Probe 16a versus 16b, same window, one with no interval and one with +`interval=1440`: window p95 was **25,777 ms**, while the July 3 bucket's p95 was **251,838 ms** +and July 7's was 103,313 ms. You cannot derive the first from the second. Probe 31 additionally +confirmed that a spec with `bins: 10, vmin: 0, vmax: 60000` returns a ten-bin histogram with +per-bin density, for free. + +**Frontend note.** `metricField` (`helpers.ts:88-92`) reads one level deep and numeric only, so +p95 is unreachable through it. The zod boundary is not the obstacle: `metricsBucketSchema` +(`web/packages/agenta-entities/src/trace/core/schema.ts:310-317`) types metrics as +`z.record(z.string(), z.record(z.string(), z.unknown()).nullable())`, so nested objects survive +validation. The mapper needs a small nested accessor, nothing more. + +--- + +#### 4. Cost, split prompt / completion / cache + +**Not ready. Total cost is computable through a path PR #5648 does not name, but its coverage +collapsed in mid-July on both datasets and nobody has found out why. The prompt and completion +split is modelled but never reaches an analytics-visible row. A cache line item does not exist.** + +This is the most consequential correction to PR #5648. + +**What does not work.** `ag.metrics.costs.cumulative.total`, `.prompt` and `.completion`, which +are what PR #5648's data contract reads and what the backend's own default spec reads +(`service.py:94`). Live, a request naming all three returned HTTP 200 with `buckets: []` (probe +1). SQL: `ag.metrics.costs.cumulative` is present on **0 of 48,005 spans** on stack A, root or +child. **Built as specified, PR #5648's Total-cost tile and its entire Costs chart render +nothing.** + +**Those paths are modelled, not missing.** `calculate_costs` +(`api/oss/src/core/tracing/utils/trees.py:579-650`) prices every chat-family span through litellm +and writes `ag.metrics.costs.incremental.{prompt, completion, total}`. `cumulate_costs` +(`trees.py:231-350`) then walks the tree and writes the same three keys under `cumulative` on +every ancestor. The prompt-and-completion split of cost therefore exists as a first-class +concept. What it does not do is cross the batch boundary from section 3.2: the tree is built per +OTLP request (`trees.py:174-193`), the run's root span is a `workflow` span with no cost of its +own, and its LLM children arrive in a different request. The roll-up fires inside batch two and +lands on `invoke_agent`, which is a child span and therefore invisible to analytics. + +**What does work.** `attributes.gen_ai.usage.cost`. `record_usage` +(`sdks/python/agenta/sdk/agents/tracing.py:213-236`) stamps the harness's own cost figure on the +root span, and no OTLP adapter maps it into `ag.*` +(`grep -rn "usage.cost" --include=*.py api/` returns nothing), so it stays at a raw top-level +JSONB path. Specs take arbitrary paths, so it aggregates: + +```json +{"type": "numeric/continuous", "path": "attributes.gen_ai.usage.cost"} +``` + +**Live test.** Probe 2. Stack A: `count 151` of 7,529 runs (2.0%), `sum $34.03`, `p50 $0.096`, +`p95 $0.97`. Stack B: **6,666 of 9,720 runs (68.6%), sum $436.06**. + +**The coverage collapse.** The two stacks disagree by a factor of thirty, and the difference is +not configuration. Both projects are about 99% `harness=claude`, `provider=anthropic`, +`connection.mode=self_managed`. The difference is time. Daily coverage of `gen_ai.usage.cost` on +the root span: + +``` +stack B Jul-06 381/395 Jul-07 1101/1145 Jul-10 1153/1442 Jul-12 1308/1459 + Jul-13 81/1455 Jul-14 0/629 +stack A Jul-03 6/11 Jul-07 55/74 Jul-08 28/184 Jul-10 12/378 + Jul-12 0/364 Jul-20 to Aug-02: 0 to 2 per ~290 runs per day +``` + +Cost was populated on 70% to 95% of runs in early July on **both** stacks and fell to roughly zero +within a week. The wiring exists end to end: harness, runner, root span, analytics spec. We proved +the pipeline works by aggregating $436 over 6,666 runs on stack B. + +**We did not find the cause, so we do not call it a regression.** We measured coverage, not code +history. Nobody ran `git log` over `record_usage`, the runner's usage reporting, or the harness +adapters. The mechanism is easy to state and hard to attribute: `record_usage` returns early when +the harness reports a falsy `usage.total` (`tracing.py:221`) and writes cost only when the +harness returns a truthy `cost` (`:232-234`). So the collapse could be a code change, a harness +version change, a change to the shape of the usage payload, or a change in what kind of traffic +these stacks carry. **This is the single most important open item in this report.** In order: + +1. Correlate the mid-July date with runner, SDK, harness and deployment versions. +2. Compare terminal `run.result().usage` payloads by harness and by streaming versus batch mode. +3. Check whether `total`, `input`, `output` and `cost` changed names or types. +4. Separate successful, failed, cancelled and manually instrumented traffic before comparing. + +Both stacks are local dev stacks. Whether production shows the same collapse is unverified. + +**Where litellm undercounts, separately.** Where litellm does price a call, it undercounts cached +prompts, because `cost_per_token` receives the non-cached prompt count only. One measured +example: a span with 1 uncached prompt token, 25,182 cache-read tokens and 20 completion tokens +was priced at $0.000303 while the harness reported $0.0082, a 27x undercount. + +**A partial fallback, with a hard caveat.** The legacy endpoint under `focus=span` sums +`costs.incremental` over all spans and returns a real number (`dao.py:566-570`). Live: a July 4 +bucket returned `costs: 0.0066` where the specs endpoint returns nothing (probe 3). **Do not +build a product on it.** On EE cloud that route is throttled to a 180-request burst and then one +request per minute per organization (section 5.4). It is a debugging tool. + +**The cheapest fix, if one is wanted.** Map `gen_ai.usage.cost` to +`ag.metrics.costs.cumulative.total` in `GENAI_SEMCONV_ATTRIBUTES_EXACT` +(`api/oss/src/apis/fastapi/otlp/extractors/adapters/logfire_adapter.py:148-196`, verified +absent), so the canonical path holds the run total and no default spec has to change. Map it to +`cumulative`, not `incremental`: `record_usage`'s own docstring says the value is the run's +aggregate total (`tracing.py:213-219`), so labelling it incremental would misname it and would +double-count against the runner's own child spans once `focus=span` works. One thing to verify +with a test before shipping that map: ingest's roll-up writes `cumulative` only when it computes +a non-zero total (`trees.py:319-341`), so an adapter-written value should survive on a zero-cost +batch, but nothing pins that behaviour today. + +A prompt, completion and cache split of cost needs the harnesses to report split costs, which +most do not. + +--- + +#### 5. Tokens, split prompt / completion / cache + +**Total is ready to ship with a coverage label. The prompt and completion split moves with cost +and shares the same collapse. Cache tokens are instrumented on child spans only.** + +**How.** + +```json +"specs": [ + {"type": "numeric/continuous", "path": "attributes.ag.metrics.tokens.cumulative.total"}, + {"type": "numeric/continuous", "path": "attributes.ag.metrics.tokens.cumulative.prompt"}, + {"type": "numeric/continuous", "path": "attributes.ag.metrics.tokens.cumulative.completion"} +] +``` + +**Live test.** Probe 7. Stack A: total `count 5842`, `sum 309M`, but prompt and completion at +`p50 0` and `p95 0`. Stack B: **prompt `sum 4.47M`, `p95 2877`; completion `sum 1.62M`, +`p95 473`.** + +So the split is not structurally zero. It moves in lockstep with the cost field and shares the +same mid-July collapse. On a dataset where the pipeline works, a stacked prompt and completion +chart renders real data. On a dataset where it does not, it renders a flat zero band, which is +worse than an empty chart because it looks like data. `record_usage` +(`sdks/python/agenta/sdk/agents/tracing.py:213-236`) stamps whatever the harness reports for +input and output, and returns early when the harness reports a falsy total (`:221`). + +**Cache tokens need no new instrumentation, but they need a way to be read.** The real path is +`gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.cache_creation.input_tokens`, note the +dotted spelling rather than `cache_read_input_tokens`. Live: **444 spans** carry it, holding 8.98M +cached input tokens, and **zero of them are root spans**; 446 are `CHAT` children. Verified by +requesting both paths and getting no entry at all in the metrics dict (probe 32). Cache tokens +therefore need exactly the same unblocking as tool usage: either roll them up to the root at +ingest, or make `focus=span` work. + +--- + +#### 6. Run success versus failure + +**Ready to ship, with a required correction to PR #5648's filter and an explicit definition of +failure.** + +**How.** Two calls with the same window and interval. Unfiltered gives totals. Filtered gives +failures. + +```json +"filter": {"conditions": [ + {"field": "status_code", "operator": "is", "value": "STATUS_CODE_ERROR"} +]} +``` + +**Live test.** Probes 4a to 4c on stack A: unfiltered 7,529; `is STATUS_CODE_ERROR` 715; +`is_not STATUS_CODE_ERROR` 6,814. The halves sum exactly. + +**Three traps, in order of cost.** + +1. **PR #5648's filter is wrong and fails silently.** Its data contract specifies + `{field: "status_code", operator: "eq", value: "ERROR"}`. Both halves fail. `status_code` is + dispatched to `_parse_enum_field_condition` + (`api/oss/src/core/tracing/utils/filtering.py:508-509`), which raises for any operator outside + comparison and list (`:366-371`); `"eq"` is `NumericOperator.EQ` + (`api/oss/src/core/tracing/dtos.py:114`). The value fails too, because `OTelStatusCode` accepts + only `STATUS_CODE_OK`, `STATUS_CODE_ERROR` and `STATUS_CODE_UNSET` + (`api/oss/src/core/otel/dtos.py:113-116`). `FilteringException` then becomes an empty 200. + Verified live: HTTP 200 in 0.016 seconds with `buckets: []` (probe 5). Because the PR's data + contract computes `success = total - failed` (`pr5648-docs/data-contract.md:71`), **the page + as specified reports zero failures and 100% health forever, on a project with a 9.5% failure + rate.** +2. **There is no `STATUS_CODE_OK` on root spans.** Success is `STATUS_CODE_UNSET`. A "success" + filter must be `is_not STATUS_CODE_ERROR`. PR #5648 gets this right by accident, by defining + success as the complement of failure. Write it down explicitly, because the obvious + implementation returns zero. +3. **Do not use `errors.cumulative`,** which is what the shipped dashboard does + (`helpers.ts:124`). It counts errored steps rolled up the tree and can exceed the run count. + For agent traffic the two agree exactly today, 715 rows both ways on stack A, but the hazard + is real for multi-span SDK workflows. PR #5648 correctly rejects it. + +**A blind spot that needs a product decision.** Root status does not see failures inside a run. +Live on stack A over the July window: 569 traces have an errored child span, and **94 of them +have a clean root**, which is **1.2%** of 7,530 runs (probe 6). Those runs count as successes. An +all-time measurement on an earlier snapshot of the same stack put it at 2.7%; use 1.2% as the +current figure. Section 7 lists this as a semantics decision, not a bug. + +--- + +#### 7. Tool usage aggregation per period + +**Not ready as "tool usage". The endpoint can chart the first configured tool of each run, which +is configuration order rather than usage. Which tools actually ran needs `focus=span`.** + +**What the engine can do today.** A spec path can index into a JSON array. The DAO splits the +spec path on `.` and hands it to the JSONB `#>` operator, which reads an integer segment as an +array index (`build_specs_values`, `utils.py:1008-1039`). So this returns a frequency table +today: + +```json +{"type": "categorical/single", + "path": "attributes.ag.data.parameters.agent.tools.0.name"} +``` + +**Live test.** Probe 34: **7,221 rows**, with `get_pr` 6,777, `read` 297, `list_open_issues` 103, +`bash` 23 and a tail. Configured tools ship on the root span at +`ag.data.parameters.agent.tools`, on 8,685 of 8,792 roots. + +**Read that result honestly.** The path names element **zero**. It charts the first *configured* +tool of each run. It does not chart the tools the run used, and it does not even chart the whole +configured set. A chart titled "Tool usage" fed by that path would be wrong, and a chart titled +"Most common first configured tool" is not a chart anyone asked for. Treat the array-index +mechanism as proof that the extraction works, not as a shippable metric. + +Charting the whole configured array needs either one spec per index, with no way to know the +array length, or a `jsonb_array_elements` expansion in the backend. Separately, a +`categorical/multiple` spec on `…agent.tools` returns nothing, because the array holds +**objects** and that family requires an array of strings (`utils.py:1746-1749`). Verified across +four spec types (probe 25). + +**What actual tool usage would need.** Each tool call is its own span, named `execute_tool +`, with `span_type = TOOL` and the name at `ag.meta.tool.name`, emitted at +`services/runner/src/tracing/otel.ts:778-787`. SQL: **7,587 spans carry that path, zero of them +roots.** A `categorical/single` spec on it returns zero buckets under any `focus` value (probe +13). This is the `focus=span` blocker. + +**Three things must ship with a `focus=span` fix, or it produces wrong numbers.** + +1. **A cumulative-versus-incremental guard.** Cumulative paths are written onto every ancestor + (`trees.py:311-350`), so scanning all spans double-counts them. Cost and tokens must switch to + `incremental` under span focus. PR #5648 identifies this correctly and it is the sharpest + thing in its plan. +2. **A run-count dedupe rule.** `ag.type.trace` is stamped on every span in a trace, so a run + count under `focus=span` counts spans, not runs. +3. **An index for non-root rows.** Today's index is partial: + `ix_spans_root_project_created_trace ... WHERE parent_id IS NULL`. Measured fan-out is + **4.96x**, 37,316 spans against 7,529 roots, and the measured cost of scanning all spans + instead of roots for one spec over 30 days is **2.60 s against 0.68 s** (section 5). + +--- + +#### 8. Model usage aggregation per period + +**A proxy is ready to ship: the configured model alias. The model that actually answered needs +`focus=span`. Cost or latency broken down by model needs a grouping dimension.** + +**How, for the alias, today.** + +```json +"specs": [ + {"type": "categorical/single", "path": "attributes.ag.data.parameters.agent.llm.model"}, + {"type": "categorical/single", "path": "attributes.ag.data.parameters.agent.llm.provider"} +] +``` + +**Live test.** Probe 9: model returned 7,345 rows across **26 distinct values** (`haiku` 6,808, +`gpt-5.6-luna` 158, `sonnet` 114 and a long tail); provider returned 7,317 rows across 5 values. +`runner.kind` and `sandbox.kind` also return frequency tables. + +**What the UI must say.** This is the **author's alias**, not the model that served the request, +and one run can call several models. The resolved id lives at `ag.meta.request.model` on each +child `chat` span, on 10,003 child spans and zero roots, unreachable until `focus=span` lands. A +small number of root spans store `…llm.model` as an object rather than a string, and the +`jsonb_typeof(jv) = 'string'` gate skips them, so the frequency counts can sit slightly below the +run count. + +**Per-model latency is cheaper than it looks.** One filtered call per model: + +```json +"filter": {"conditions": [ + {"field": "attributes", "key": "ag.data.parameters.agent.llm.model", + "operator": "is", "value": "haiku"} +]} +``` + +Verified live (probe 26): 27 buckets with full statistics, and six models issued in parallel +finished in **0.68 seconds** (section 5.3). For a dimension with ten or fewer values this is a +workable v1 pattern. Section 5.7 states its costs, because it is not free. + +**Per-model cost needs a grouping dimension** (section 4.3). PR #5648 correctly names group-by as +the harder, later change, but its plan text elsewhere implies span focus is the blocker for both. +It is not. + +--- + +#### 9. Harness usage aggregation per period + +**Ready to ship. Absent from all six of PR #5648's documents.** + +**How.** One `categorical/single` spec on `attributes.ag.data.parameters.agent.harness.kind`. The +full request and response are the worked example in section 3.9. + +**Live test.** Probe 8a: 30 daily buckets, each with a frequency array. SQL: the path is present +on 8,765 of 8,809 roots (99.5%) and is always a JSON string. + +**Two caveats.** + +- **The path must end in `.kind`.** `…agent.harness` is a JSON object. A `categorical/single` + spec pointed at it returns exactly the handful of legacy rows that store `harness` as a bare + string: **6 rows** in the probe window, `pi_agenta` 3 and `pi_core` 3 (probe 8b). A `json` spec + on the same object path returns `count: 7523`, and 7,523 plus 6 equals 7,529, which proves the + type gate exactly (probe 8c). +- **The runner's own harness id is unmapped.** `gen_ai.agent.name` on the `invoke_agent` child + span has no `ag.*` mapping. `logfire_adapter.py:186-187` maps `gen_ai.agent.id` and + `gen_ai.agent.description` but not `.name`, and it is on a child span anyway. The config-blob + path is the usable one. + +--- + +#### 10. Filtering by agent, and per-agent breakdown + +**Both ready to ship. The breakdown is not in PR #5648's plan.** + +**Filter.** + +```json +"filter": {"conditions": [ + {"field": "references", "operator": "in", "value": [{"id": ""}]} +]} +``` + +This is already the mechanism the shipped frontend uses +(`web/oss/src/services/tracing/api/index.ts:41-47`). `references` accepts list, dictionary and +existence operators only (`filtering.py:159-163`), and the `in` value must be a list of +dictionaries. + +**Breakdown, one call, no filter.** + +```json +"specs": [ + {"type": "categorical/single", "path": "attributes.ag.references.workflow_variant.id"}, + {"type": "categorical/single", "path": "attributes.ag.references.application_variant.id"} +] +``` + +**Live test.** Probe 10: 6,900 plus 341 equals 7,241 of 7,529 runs. **288 roots carry no agent +reference at all.** + +**You must union two naming families.** Agent identity is recorded under +`workflow` / `workflow_variant` / `workflow_revision` and under +`application` / `application_variant` / `application_revision`. A page that queries only one +silently loses a chunk of traffic. + +**One thing that does nothing.** The frontend sends `application_id` on every analytics call +(`projectScopedRequest`, `client.ts:40`). The analytics handler never reads it +(`router.py:1277-1315` uses `request.state.project_id` only). App scoping comes entirely from the +`references` condition. + +--- + +#### 11. Filtering by model + +**Ready to ship, for the configured alias.** + +Shown under item 8. Verified live (probe 23): a filter on `…agent.llm.model is haiku` combined +with a harness-kind spec returned 6,808 rows, all `claude`. Arbitrary-depth attribute filters +work through `_to_jsonb_path` (`utils.py:72-87`), and for comparison operators they compile to a +containment predicate that the GIN index on `attributes` can assist. + +**One trap.** `key` is required on an `attributes` condition (`filtering.py:89-93`). Omitting it +raises, which becomes an empty 200. + +--- + +#### 12. Filtering by harness + +**Ready to ship. Absent from PR #5648.** + +```json +"filter": {"conditions": [ + {"field": "attributes", "key": "ag.data.parameters.agent.harness.kind", + "operator": "is", "value": "claude"} +]} +``` + +**Live test.** Probe 22: 7,056 of 7,529 runs. The only loss is the small number of legacy rows +that store `harness` as a bare string. + +--- + +#### 13. Filtering by the calling user, and numbers per user + +**Not ready. The filter mechanism works, but the dimension holds one value per project on every +dataset we checked, and per-user numbers need a backend change on top of that.** + +**The filter.** + +```json +"filter": {"conditions": [ + {"field": "created_by_id", "operator": "in", "value": [""]} +]} +``` + +`created_by_id` is the Agenta account whose credential ingested the span. Ingest writes it on +every row (`api/oss/src/dbs/postgres/tracing/mappings.py:210`, column at +`api/oss/src/dbs/postgres/shared/dbas.py:106-109`), and the filter accepts comparison, list and +existence operators (`filtering.py:428-432`). + +**Live test, and why it proves less than it looks.** Probe 11a returned **all 7,529 runs**, +because probe 12 found `created_by_id` has **exactly one distinct value per project** on both +stacks. A fabricated id returns zero buckets, so the filter is mechanically valid. But the demo +is evidentially empty: there is no second user to narrow to. + +**Numbers per user do not work, twice over.** + +- **Mechanically.** `created_by_id` is a **column**, and specs read `attributes #> path` only. No + spec can produce a per-user frequency table (probe 37). +- **Evidentially.** There is a second user concept, `ag.user.id`, an end-user identity that the + instrumenting caller must set. It survives ingest as an attribute and is also copied to a + `user_id` column (`trees.py:143-172`). A `categorical/single` spec on `attributes.ag.user.id` + is a working mechanism with zero data: SQL shows the promoted `user_id`, `session_id` and + `agent_id` columns are **NULL on 100% of root spans on both stacks**, and no root span carries + the attribute. The agent runtime never sets it. + +**Options.** The cheapest hack is to copy `created_by_id` into `attributes` at ingest, so one +`categorical/single` spec gives the breakdown. It is a hack: it duplicates a typed column into +JSON to route around the query engine's inability to group by a column, and it doubles the +storage of a value that already exists. The better fix is to let the analytics contract name +`created_by_id` as a dimension directly (section 8.4). Either way, decide first whether "who +called" means the API credential or an end user set through `ag.user.id`; that is a product +question, and the answer changes which field matters. + +Whether cloud projects have more than one distinct `created_by_id` is unverified. + +--- + +#### 14. Skills used + +**Not ready. The endpoint can chart the first configured skill of each run. Which skills the +agent actually used is recorded nowhere.** + +**What the engine can do today.** The same array-index mechanism as item 7: + +```json +{"type": "categorical/single", + "path": "attributes.ag.data.parameters.agent.skills.0.name"} +``` + +**Live test.** Probe 34: **491 rows**, `build-an-agent` 422, `style-editing` 30, +`agenta-getting-started` 23, `composio-github-pr` 16. A separate existence filter on +`…agent.skills` found **518 root spans** carrying a configured skill list, split `pi_core` 266, +`claude` 248, `pi_agenta` 4 (probe 35). The entries carry `name`, `description`, `body`, `files`, +`allow_executable_files` and `disable_model_invocation`. + +The same honesty applies as for tools: index zero is configuration order. + +**What is missing.** Nothing records a skill invocation anywhere. Two adjacent things exist and +neither answers the question: + +- **Configured skills** on the root span, covered above. Configured is not used. +- **Loaded skills** on the `invoke_agent` child span at `ag.meta.skills.loaded`, an array of + strings on 1,067 spans, written at `services/runner/src/tracing/otel.ts:703-705`. It would work + with a `categorical/multiple` spec under `focus=span`. But the runner sets it from the skills + surfaced to the model, so "loaded into context" is not "invoked", and it is on a child span + (probe 13 confirms child paths return nothing). + +**What it would take.** A runner-side signal for skill invocation, as a span or an event carrying +the skill's identity, then promotion to the root span if you want it without `focus=span`. This +is a product and instrumentation question before it is an analytics one. + +### 4.5 One more hazard: response size + +`freq` and `uniq` arrays are **uncapped**. A `TOP_K = 3` constant is defined at +`api/oss/src/dbs/postgres/tracing/utils.py:988` and never applied. One measured response with a +single metric across 590 buckets was **897 KB**, because every bucket carries 27 percentiles, an +IQR set and a histogram. A breakdown over an open-ended dimension, such as tool name or user id, +has no size bound at all. Cap the bucket count on the frontend, and treat any high-cardinality +categorical spec as a size risk until the backend caps it. + +--- + +## 5. Scale and performance + +### 5.1 The conclusion first + +**Today's query shape is not a durable contract for a product page.** Three measured facts say +so, and they hold regardless of how the numbers extrapolate: + +1. **Cost scales with bytes read, not with rows returned.** Every analytics query de-TOASTs the + whole `attributes` JSONB of every matched span to read a handful of values out of it. A + 30-day, one-metric query touched about 915 MB of buffers to produce 29 averages. +2. **A killed query is indistinguishable from an empty window.** The statement timeout returns + HTTP 200 with `buckets: []` after 15 seconds of spinner. +3. **The 30-day view the shipped dashboard already issues costs 1.7 seconds** at 7,500 runs, on a + warm cache, with one user. The 7-day view costs 0.26 seconds. + +None of that blocks a seven-day beta. All of it blocks treating "any window, any metric" as a +supported API. + +### 5.2 What was measured, and over how much data + +Wall-clock timings against both local stacks, 7 repetitions per shape on stack A and 5 on stack +B, plus `EXPLAIN (ANALYZE, BUFFERS)` runs against the tracing database directly. All windows end +at a fixed anchor so they are reproducible. Every run was single-user against a warm cache on one +machine. **Nothing here was measured under concurrency, and nothing was measured on +production-shaped data.** + +Volume behind each window on stack A, counted rather than estimated: + +| Window | Root spans | All spans | Root attribute bytes | +|---|---|---|---| +| 24 hours | 295 | 1,497 | 0.91 MB | +| 7 days | 2,051 | 10,571 | 7.9 MB | +| 21 days | 6,320 | | 24 MB | +| 30 days | 7,523 | 37,207 | **72 MB** | + +Note the jump from 24 MB at 21 days to 72 MB at 30 days for only 19% more rows. Early-July runs +carry much larger attribute blobs; the largest single root span is 2.1 MB. + +### 5.3 The measurements + +**A. Window size dominates. Bucket period barely matters.** One numeric spec, median seconds: + +| Window | Hourly | Daily | Weekly | +|---|---|---|---| +| 24 hours | 0.040 | 0.038 | 0.038 | +| 7 days | 0.169 | 0.090 | 0.097 | +| 30 days | 0.77 to 1.54 | 0.724 | 0.688 | + +The one exception is high bucket counts, and that cost sits in the API process, not the database. +The 590-bucket 30-day hourly query measured 700 ms at the database against 679 ms for the +29-bucket daily version, a 3% difference. The extra time is Python bucket assembly and JSON +serialization of an 897 KB response. + +**B. Each additional metric spec costs about 0.2 seconds.** 30 days, daily buckets, median +seconds: 1 spec 0.725, 4 specs 1.087, 8 specs 2.318, 16 specs 3.726. Specs are not amortized. The +query cross-joins every matched span against the spec list (`utils.py:1106-1126`), so eight specs +means 60,184 intermediate rows for 7,523 spans. + +**C. Filtering can make a query faster than the row count suggests.** A filter on +`…agent.llm.model is haiku` keeps 91% of the rows and runs **2.1x faster**, 0.329 s against +0.700 s. It drops 710 rows and **49 of the 72 MB**, because the excluded runs average 127 KB of +attributes each against haiku's 3.5 KB. Latency follows the bytes. + +**D. The shapes an analytics page would actually issue,** median seconds on stack A: + +| Shape | Buckets | Median | +|---|---|---| +| The shipped dashboard's own request, 24 hours | 48 | **0.075** | +| Same, 7 days | 56 | **0.260** | +| Same, **30 days** | 56 | **1.714** | +| One call, 8 specs, 30 days daily | 29 | **2.486** | +| Window-level statistics, 8 specs, no interval | 1 | **1.821** | +| Per-model latency, one model | 26 | **0.302** | + +**E. Six small parallel calls beat one big call, on an idle database.** Six tiles issued as six +parallel single-spec calls finished in **0.974 s**. The identical coverage in one six-spec request +took **1.700 s**. Postgres parallelises across connections, while the multi-spec query's internal +joins are serial nested loops. Five per-model calls issued sequentially took 2.659 s, so +parallelism is doing the work, not call count. Section 5.7 explains why this is a useful +measurement and a poor contract. + +### 5.4 Rate limits: two routes are unusable for a dashboard on cloud + +While benchmarking, the legacy endpoint began returning **HTTP 429** with +`X-Ratelimit-Limit: 180` and `Retry-After: 41`. + +`api/ee/src/core/access/entitlements/types.py:170-175` puts both `/tracing/...` analytics routes +in the `TRACING_SLOW` category: + +```python +Category.TRACING_SLOW: [ + (Method.POST, "/tracing/*/query"), + (Method.POST, "/tracing/spans/analytics"), # LEGACY +], +``` + +The bucket for that category on the paid tier is `capacity=180, rate=1` (`types.py:486-493`), and +`rate` is **tokens added per minute** (`types.py:117`; the Lua refill at +`api/oss/src/utils/throttling.py:108-110` divides elapsed milliseconds by 60,000). So it is a +180-request burst that refills at one request per minute. The bucket key includes the +organization (`api/ee/src/middlewares/throttling.py:242-247`), so the budget is shared by every +user in the org. + +The documentation states this correctly. `docs/docs/misc/faq/platform/api-rate-limits.mdx:11` +defines the two numbers as burst and per-minute refill, and the table lists trace queries and +analytics at 180 / 1 on the Pro plan. An earlier draft of this report claimed the FAQ described +the bucket as a per-minute rate. That claim was wrong; the FAQ is right. + +Measured live, three requests back to back: + +| Route | Result | +|---|---| +| `POST /spans/analytics/query` | 200, `X-Ratelimit-Remaining: 1439` | +| `POST /tracing/analytics/query` (deprecated) | 200, `X-Ratelimit-Remaining: 0` | +| `POST /tracing/spans/analytics` (legacy) | **429**, `Retry-After: 27` | + +An earlier probe fired 130 sequential requests at both routes and saw 130 HTTP 200s each (probe +19). That does not contradict the 429: 130 is below the 180-request burst, so the probe never +drained the bucket. The three-request table above was taken after the benchmark run had drained +it, which is why the deprecated route reports zero remaining. + +Three consequences: + +- **A page firing six analytics calls exhausts the `TRACING_SLOW` burst in 30 page loads** and is + then limited to one chart refresh per minute for the whole organization. Those routes are not + usable for a dashboard. +- **`/spans/analytics/query` is not in the category map at all,** so it falls to the `STANDARD` + catch-all: 480 requests per minute on the free tier and 1,440 on the paid tier + (`types.py:376-379` and `:466-469`). That is what makes a dashboard viable. It also means + analytics shares a bucket with every other ordinary API call, and 480/min is only 80 page loads + per minute at six calls each. We could not determine whether the omission was deliberate; the + code shows the classification, not the intent. +- **OSS self-hosted has no throttling at all.** The middleware exists only under `api/ee`. + +**The product rule: the new page must use `POST /spans/analytics/query`.** The shipped frontend +already does. + +### 5.5 Where the time actually goes + +**Latency tracks attribute bytes, not rows.** Measured from `EXPLAIN (ANALYZE)` of the exact SQL +the DAO builds: + +| Window | Roots | Attribute MB | Database time | ms per MB | +|---|---|---|---|---| +| 24 hours | 295 | 0.91 | 7.5 ms | 8.2 | +| 7 days | 2,051 | 7.9 | 69.8 ms | 8.8 | +| 21 days | 6,320 | 24 | 212.1 ms | 8.8 | +| 30 days | 7,523 | 72 | 678.9 ms | 9.4 | + +That is a straight line in megabytes and visibly not a straight line in rows: 19% more rows from +21 to 30 days, 220% more time. + +**The plan shows two separate costs, and only one of them is about bytes.** + +*One spec, 30 days* (query plan `exp-1.out`). PostgreSQL **inlined** the base and extract CTEs and +evaluated `attributes #> '{ag,metrics,duration,cumulative}'` directly in the heap scan's filter +(`exp-1.out:5-8`). That scan alone took 660 ms of the 679 ms total and touched 117,085 buffers, +about 915 MB, to produce 7,523 numbers. + +So the SQL is not asking for more than it needs; the planner already pushes the extraction down +to the scan. The cost is that reading any path out of a JSONB value requires fetching and +de-TOASTing the entire value first. A root span's `attributes` averages 10 KB on this dataset, so +reading one 8-byte number costs a 10 KB detoast. **Rewriting the query to project `#>` earlier +would change nothing.** An earlier draft of this report recommended exactly that. It was wrong. + +*Eight specs, 30 days* (query plan `exp-8.out`) exposes a second cost that has nothing to do with +bytes. With eight specs, several downstream CTEs reference `extract_cte`, so PostgreSQL +materializes it: 60,184 extracted rows in 1,614 ms (`exp-8.out:6`). Then the planner's row +estimate for every CTE scan collapses to 1, and it chooses nested loops. Four separate joins each +discard **3,123,312** rows to keep 25,188 (`exp-8.out:43-45`, `:55-57`, `:67-69`, `:90-92`). +Total 2,688 ms. That is why the eighth spec costs so much more than the first, and no amount of +byte reduction fixes it. + +Two problems, two different fixes: + +1. **Byte cost.** Only typed storage removes it. Promote the handful of hot metric paths into real + columns, or write a per-run facts table at ingest, so a chart never reads a JSONB blob. +2. **Plan cost.** The multi-spec query joins CTEs the planner cannot estimate. Reshaping that + query, or letting a request name which aggregations it wants, cuts work directly: today a + `numeric/continuous` spec always computes the count, the basics, all 27 percentiles and a + histogram, even for a tile that shows one sum (`utils.py:1211-1215`, `:1263-1287`). + +**Row selection is healthy and will stay healthy.** The index +`ix_spans_root_project_created_trace btree (project_id, created_at, trace_id) WHERE parent_id IS +NULL` matches the generated predicate and is 984 kB. At 24 hours the planner uses it with a real +range condition. At 30 days on a project whose entire history is 30 days it flips to a different +index and post-filters the time predicate, which is correct there. The cost is entirely in what +happens after the rows are found. + +**The timeout returns HTTP 200 with an empty body.** `TIMEOUT_STMT` sets +`statement_timeout = '15000'` (`utils.py:48`, applied at `dao.py:420`). The DAO method is +decorated `@suppress_exceptions(default=[])` (`dao.py:305`), which catches everything and returns +an empty list, and the router then wraps that empty list in a normal `AnalyticsResponse`. + +Reproduced live on stack A, 30-day window, 80 specs: + +``` +HTTP 200 in 15.033 s -> {"count": 0, "buckets": []} +``` + +and in the API log: + +``` +asyncpg.exceptions.QueryCanceledError: canceling statement due to statement timeout +[SUPPRESSED] +``` + +The user sees an empty chart after a 15-second spinner, with no error code and nothing that lets +the frontend tell "no data in this window" apart from "the query was killed". Spec counts at 30 +days approach that cliff steadily: 24 specs 5.4 s, 32 specs 7.3 s, 48 specs 10.6 s, 64 specs 13.3 +to 14.1 s, 80 specs timeout. + +### 5.6 What happens as data grows + +**This subsection is a risk range, not a capacity forecast.** Read the caveats before the table, +because they govern it. + +- The extrapolation rests on the measured 9 ms/MB/spec figure. That figure is an observation on + one dataset with one attribute-size distribution, not a constant. A project whose runs carry + small attributes will sit far below it; one with 2 MB blobs will sit above it. +- The synthetic 10x check duplicated existing rows with `generate_series(1,10)` in the benchmark + script `exp-1-10x.sql`. That reuses the same heap pages, the same TOAST entries, the + same cache locality, the same value distribution and the same table statistics. It is not a + database with ten times as many independently stored spans, and it flatters the result. +- Every measurement is single-user with a warm cache. Concurrency was not tested at all. +- Working against all of the above: the TOAST relation is already about 5x `shared_buffers` at + today's volume, so cache hit rates fall as data grows and the real curve steepens. + +| Project size | 30-day root attributes | 1 spec | 6 specs (shipped shape) | 8 specs | +|---|---|---|---|---| +| **1x** (today, 7.5k runs) | 72 MB | 0.7 s | **1.7 s** | 2.4 s | +| **3x** (~23k runs) | 216 MB | ~2 s | ~5 s | ~7 s | +| **10x** (~75k runs) | 720 MB | 3 to 6.5 s | 10 to 17 s | 13 to 23 s | +| **100x** (~750k runs) | 7.2 GB | 24 to 65 s | beyond the timeout | beyond the timeout | + +What to take from it: + +- **The 30-day view is the shape at risk.** It is the slowest thing the product issues today, and + it is the first thing that will cross the 15-second statement timeout as projects grow. Whether + that happens at 5x or at 20x depends on attribute sizes and concurrency, which we did not + measure. Treat "30 days will break before 100x" as the claim; treat any tighter number as + unproven. +- **Short windows are much safer,** because a 24-hour window is bounded by ingest rate rather + than by history. A 100x project's 24-hour window would hold roughly 29,500 roots and 290 MB. + That is well inside today's limits on these measurements, but it has not been tested under + concurrent load. +- **A blank chart, not a slow chart, is the failure mode,** until section 8.4's first item lands. + +### 5.7 What is safe behind a page load + +**Safe today on these measurements:** + +- Any window up to **7 days**, at any bucket period, with up to about 8 specs: 0.09 s to 1.3 s. +- **24-hour** windows at any shape: under 0.2 s. +- The choice of metric does not matter. Counts, latency, percentiles, histograms and categorical + breakdowns all cost about the same. Only the number of specs and the volume of data matter. + +**Needs a narrower default or caching first:** + +- **The 30-day window.** 1.7 s for the shipped six-spec shape at today's volume. Make the page's + default **7 days**, and treat 30 and 90 days as an explicit user choice with a loading state. +- **Any request with more than about 8 specs.** +- **Bucket counts above about 100.** The database does not care; the JSON does. + +**On fanning out into many small calls.** Six parallel one-spec calls measured 1.7x faster than +one six-spec call, and per-value latency through N filtered calls is the only way to get +per-model or per-harness statistics today. Both are legitimate ways to build v1. Neither is a +contract to design around, and the report should not pretend otherwise: + +- Six calls do not do less work. They do six scans instead of one, and they win only by borrowing + six database connections from a pool that other users share. The win shrinks or reverses under + concurrency, which we did not test. +- Six calls consume six times the rate-limit budget. At 480 requests per minute on the free tier + that is 80 page loads per minute for the whole organization (section 5.4). +- Six calls can partly fail, partly time out, and disagree with each other if data lands between + them. The page needs a per-call error state, not one spinner. + +Use the fan-out for v1 because it works and it is measurable. Put "one bounded request per page +section" in the target contract (section 8.4). + +--- + +## 6. Frontend integration + +### 6.1 Which endpoint, and what the client already supports + +Use `POST /spans/analytics/query`. Nothing needs regenerating. + +- The Fern request type already carries `specs`: `QuerySpansAnalyticsRequest` has `focus`, + `format`, `oldest`, `newest`, `interval`, `rate`, `filter` and `specs` + (`web/packages/agenta-api-client/src/generated/api/resources/traces/client/requests/QuerySpansAnalyticsRequest.ts:9-18`), + and the client forwards `specs` straight into the query params + (`.../traces/client/Client.ts:709-719`). +- `filter` and `specs` are typed as plain strings, so the generated client imposes no shape. + Whatever JSON you stringify goes through. The only real constraint is `MetricSpec` on the + backend. +- The zod boundary does not strip nested metric fields. `metricsBucketSchema` + (`web/packages/agenta-entities/src/trace/core/schema.ts:310-317`) types metrics as + `z.record(z.string(), z.record(z.string(), z.unknown()).nullable())`, so `pcts`, `freq` and + `hist` survive validation as `unknown`. TypeScript forces a cast at the read site. No data is + lost. + +### 6.2 The client changes needed + +1. **Add an optional `specs` field to `SpansAnalyticsParams`** + (`web/packages/agenta-entities/src/trace/api/api.ts:293-310`) and serialize it to a JSON string + query param beside the existing `filter` line at `api.ts:344`. One field, one line. PR #5648 + identifies this correctly, and the package layer is the right home for it: it is transport, + not page logic. +2. **Add a nested accessor for percentiles** in a new page-level mapper. `metricField` + (`helpers.ts:88-92`) reads one level deep and finite numbers only; p95 lives at + `metrics[path].pcts.p95`. +3. **Add a reader for `freq` arrays,** which unlocks every category breakdown in section 4. + Nothing reads them today. +4. **Keep the new mapper in the app layer, beside `analyticsToGeneration`, not inside it.** The + Observability page depends on the existing mapper's shape. +5. **Do not reuse `observabilityDashboardTimeRangeAtom`.** A second consumer already shares it + (`web/oss/src/components/pages/agent-home/components/UsageSummary/index.tsx:11,29`), and a + third surface would fight over the range. Give the new page its own atoms. +6. **Do not copy `CustomAreaChart.tsx`.** It hardcodes hex colours at `:28-32`, which + `web/AGENTS.md` forbids. + +### 6.3 What the browser can compute, and what it cannot + +**Simple arithmetic, needs nothing new:** window totals (sum the per-bucket sums), window average +latency (sum of duration sums divided by the sum of duration counts, never the mean of per-bucket +means), success rate, ratios, percentage-change badges against a previous window, sparklines, +stacked series, and window-level min and max, which do compose. + +**Impossible from bucketed data, no matter how clever the mapper:** + +- **A window-level percentile.** p95 does not compose across buckets. Verified: window p95 + 25,777 ms against a single day's 251,838 ms. The fix is not backend work. It is the no-interval + second call. +- **A window-level distinct count.** The same value appears in many buckets, so `uniq` arrays + cannot be summed. Same fix. +- **Any grouping the request did not ask for.** No numeric metric can be split by a category + after the fact. +- **Anything on a child span.** Those rows never leave the database. + +### 6.4 What the response shape forces on the UI + +- **Empty buckets are omitted.** The frontend must build its own x-axis, or gaps will read as + missing days rather than as zero days. +- **The top-level `count` is the number of buckets,** not the number of runs. +- **The requested interval may have been coarsened silently.** Only `buckets[].interval` reports + what ran. If the UI lets the user pick a period, read the effective interval back and label the + chart with it. +- **A wrong filter renders as zero, not as an error.** Verify every filter the page sends once + against a known-good window during development, because there is no runtime signal. +- **Buckets are 24-hour periods aligned to `oldest`,** not calendar days. Align `oldest` to the + viewer's local midnight and label the axis honestly. + +### 6.5 Two defects already shipped on this spine + +Neither was introduced by PR #5648, and both are inherited by any page built beside it. + +- **`failure_rate` is a fraction rendered with a percent sign.** `helpers.ts:160` returns + `errorCount / totalCount`, and `AnalyticsDashboard.tsx:96` renders it with a `%` suffix through + `formatNumber`, which does not multiply by 100. A 50% failure rate displays as "0.5%". +- **Two dead filter conditions.** `web/oss/src/services/tracing/api/index.ts:48-61` pushes + `environment` and `variant` conditions. Neither is a real field, so the backend logs a warning + and drops them, which **widens** the query. No current caller sets those options, so it is + latent rather than live, but it is the same silent-drop failure mode that breaks PR #5648's + failed-run filter. + +--- + +## 7. The semantics you must decide before you build + +Several numbers on this page can be computed today but mean more than one thing. Pick an answer +for each, write it in the UI, and do not let the implementation pick by default. + +| Question | The options | What we would pick, and why | +|---|---|---| +| What is a **failed run**? | (a) The root span's status is `STATUS_CODE_ERROR`. (b) Any span in the trace errored. (c) The run produced no usable result. | (a) for v1, and say so in a tooltip. It is the only one computable in one call. It misses 1.2% of runs that failed inside and recovered a clean root (probe 6). (b) needs `focus=span`. (c) is not recorded anywhere. | +| What is a **run**? | (a) Every root span. (b) Root spans of `trace_type = invocation`. | (b). Annotations are evaluator and human-annotation traces, not agent runs, and the PR's own glossary says so before its metric counts both. | +| What is **the model**? | (a) The alias the author configured. (b) Every model the run actually called. | Chart (a) and label it "configured model". (b) needs `focus=span` and a decision about runs that call several models. Do not print (a) under the word "model". | +| What is **tool usage**? | (a) The first tool in the configured list. (b) The whole configured list. (c) The tools the run actually called. | Only (c) is worth charting, and it needs `focus=span`. Ship nothing here in v1 rather than shipping (a). | +| What is **skill usage**? | (a) Configured skills. (b) Skills loaded into context. (c) Skills the agent invoked. | (c), and nothing records it. This needs a runner change before it is an analytics question. | +| Who is **the user**? | (a) The Agenta account whose API credential wrote the span. (b) An end user the caller declares through `ag.user.id`. | Decide by what the page is for. (a) answers "which teammate ran this"; today it holds one value per project. (b) answers "which of my customers", and nothing sets it. | +| What does a **cost number** include? | (a) Whatever the harness reported. (b) Agenta's own litellm pricing of the LLM spans. | (a) is what the working path holds, and it is the run total. (b) exists but never reaches the run's root span, and it undercounts cached prompts by up to 27x. Whichever you pick, the tile must state its coverage. | +| What is a **day**? | (a) A fixed 1,440-minute period aligned to the window start. (b) A calendar day in the viewer's timezone. | (a) is what the backend does. Offer it and label it. (b) needs timezone-aware bucketing in the backend, because a fixed stride drifts across daylight saving transitions. | + +--- + +## 8. Proposal + +### 8.1 What PR #5648 gets right + +Build on these. They are correct and they were not obvious. + +- **The endpoint choice.** `POST /spans/analytics/query` is the right route, for capability + reasons and for the rate-limit reason the PR did not know about. +- **Reusing the existing fetch layer and adding `specs` as one optional field.** That is exactly + the right seam. +- **Rejecting `errors.cumulative` as a failed-run count.** It counts errored steps, not failed + runs. The PR is right to define a failed run as a run whose root span status is `ERROR`, and + right that this needs a second filtered query because specs cannot read columns. +- **Identifying `focus` as dead and naming the exact fix location** (`utils.py:1073`). +- **Naming the double-count guard that must ship with a `focus=span` fix.** Cumulative paths are + written onto every ancestor, so scanning all spans inflates cost and tokens with no error. This + is the sharpest observation in the PR's plan. +- **Splitting the deferred work into two unequal prerequisites,** span focus and group-by, and + noting that only per-model cost needs the second. +- **Deciding not to mount empty "coming soon" cards.** + +### 8.2 The corrections that block the plan as written + +1. **The failed-run filter is rejected and returns an empty 200.** Correct form: + `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}`. Success is + `is_not STATUS_CODE_ERROR`; there is no `STATUS_CODE_OK` on root spans. Verified live. +2. **The Costs chart and the Total-cost tile read dead paths.** + `ag.metrics.costs.cumulative.prompt` and `.completion` are on zero spans on both stacks. The + only populated path is `attributes.gen_ai.usage.cost`, which the PR never names, and its + coverage collapsed in mid-July for reasons nobody has established. That is a data question, so + it is neither `api/` work nor frontend work. +3. **The Tokens split chart depends on the same collapsed coverage.** It works where the pipeline + works and renders a flat zero band where it does not. +4. **Window-level p95 is absent from the plan.** Omitting `interval` returns one exact bucket for + the whole window. The plan's four-call shape spends calls re-fetching per-bucket data to + compute totals the backend would compute exactly. +5. **`focus=span` is accepted, echoed and ignored,** which means the plan's own debugging advice, + reading the echoed query, will confirm a parameter that did nothing. +6. **The run count includes annotation traces** without a `trace_type` filter, which contradicts + the PR's own glossary definition of a run. +7. **Calendar months are not expressible,** and period semantics are never defined in the PR. + Buckets are fixed-width offsets from `oldest`, in UTC, keyed on ingest time. +8. **The 1024-bucket ceiling is a silent override, not a limit to stay under.** A long window with + a small period returns a different granularity than the user chose, with nothing in the UI + saying so. +9. **Harness, per-user, cache tokens and skills never appear in the PR's six documents,** not even + as out of scope. Two of those four work today. + +### 8.3 v1: a coverage-gated beta + +**Principle.** The page is a beta behind a flag. It ships only metrics whose meaning we can state +in one sentence and whose coverage we can show. Nothing on it depends on a backend capability +that does not exist. Two backend items are prerequisites anyway, and they are listed as such +rather than hidden inside "no backend changes required". + +**Scope.** + +- **Default window of 7 days,** not 30. This is the single most important change to the plan. + 30 days costs 1.7 s today and is the shape that degrades first. 7 days costs 0.26 s. +- **Summary tiles from one no-interval call:** total runs, average latency, p95 latency, total + tokens, and the distinct-value lists for harness, configured model and agent. One call, one + bucket, exact numbers. +- **Charts from bucketed calls:** runs stacked by success and failure, average latency with a p95 + line, total tokens. +- **Three category charts nobody planned and that cost one spec each:** runs per harness, runs per + configured model, runs per agent, each per period. These are the most differentiated content on + the page and the honest ones. +- **Filters:** by agent (`references`), by harness, and by configured model. +- **Cost and the token split are coverage-gated, not omitted.** Query + `attributes.gen_ai.usage.cost`, compare its `count` against the run count for the same window, + and render the tile only when coverage clears a threshold. Below it, say "cost data is not + available for this window" rather than showing a zero. The same rule governs the + prompt/completion token split. +- **Not on the page:** tool usage, skill usage, resolved-model usage, per-user numbers, cache + tokens. Section 4.2 says why for each. + +**Query shape.** Fan out into several small parallel calls for now, with a per-call error state, +and keep any single request at or under 8 specs. Section 5.7 states what that costs. + +**Work items, in order.** + +Backend, and both are prerequisites: + +1. **Make a killed or rejected query say so.** Section 8.4 item 1. Without it, every empty chart + on the page is ambiguous, and the coverage gate above cannot tell "no cost data" from "the + query died". +2. **Investigate the cost and token-split coverage collapse.** File it. Section 4.4 item 4 lists + the first four checks. + +Frontend: + +3. Add the optional `specs` field to `SpansAnalyticsParams` and serialize it + (`web/packages/agenta-entities/src/trace/api/api.ts`). +4. Add a page-level mapper with a nested reader for `pcts` and a reader for `freq` arrays. Do not + change the existing mapper; the Observability page depends on it. +5. Build the page's own atoms. Do not reuse `observabilityDashboardTimeRangeAtom`. +6. Issue the no-interval window call for the tiles and the bucketed calls for the charts, in + parallel. +7. Use the corrected failure filter, and add a `trace_type is invocation` filter to the run count. +8. Align `oldest` to the viewer's local midnight. Offer day, week and "whole window" as periods. + Do not offer calendar months. Label the axis as 24-hour periods. +9. Read `buckets[].interval` and label the chart with the effective period. +10. Implement the four page states: data, no data in window, metric unavailable (coverage below + threshold), and request failed. +11. Fix `failure_rate`, which is rendered without multiplying by 100. + +**Named semantic-convention attributes v1 adds: none.** That is deliberate, and it is also the +reason cost has to be coverage-gated rather than fixed here. + +### 8.4 v2: the backend work, in order + +**1. Make failures visible. Small. Do this first.** + +A killed query, a rejected filter and an empty window are three different things that all return +`{"count": 0, "buckets": []}` today. Removing `@suppress_exceptions(default=[])` from +`TracingDAO.analytics` (`dao.py:305`) is **not** enough, and the sibling `query` method is not the +model to copy: it raises a bare `Exception` (`dao.py:288-298`), and the route is itself wrapped in +`@suppress_exceptions(default=AnalyticsResponse(), exclude=[HTTPException])` (`router.py:1276`), +which swallows any non-`HTTPException` (`api/oss/src/utils/exceptions.py:97-113`). The fix has to +cross both layers: + +- Raise a typed analytics-timeout error in core, and translate it at the router into an + `HTTPException` with status 504 and a "narrow your window" message. `HTTPException` passes + through both the route's `suppress_exceptions` and the outer `intercept_exceptions` + (`exceptions.py:129-130`), so it reaches the client. +- Do the same for `FilteringException` and for a malformed window, as 4xx. +- Add a per-metric `sample_count` to the response so the frontend can tell "no data" from + "unavailable". + +**2. Decide the contract. Medium, and it gates everything below.** + +Today's request shape is an arbitrary JSON path plus a type hint, validated by silence: an unknown +filter field is dropped (`filtering.py:542-546`), a wrong-typed metric contributes zero rows +(`utils.py:1660`), and cardinality is uncapped. That is a reasonable internal exploration engine. +It is a poor public contract for a product page, because every UI built on it depends on the exact +JSON shape ingest happens to write today. + +The alternative is a typed run-analytics request: named metrics and dimensions from an enum, an +explicit list of requested aggregations, a filter grammar over known fields, granularity, a +timezone, and documented caps on window, bucket count and cardinality. Core validates it and +returns 4xx on unsupported combinations; a query compiler turns it into SQL. + +The trade is real: the typed contract is more work now and cannot answer a question nobody +anticipated. Given that the page is the first product surface on this engine and that its metrics +are the ones in section 7, the typed contract is the better bet. Decide it before v2 code, because +items 3 to 6 look different under each answer. + +**3. Make `focus=span` work. Medium. Unlocks tool usage, resolved-model usage, cache tokens.** + +Thread `query.formatting` into `build_base_cte` and make the `parent_id IS NULL` predicate +conditional (`utils.py:1073`, `dao.py:378-387`). Three things must ship with it or it produces +wrong numbers, all named in section 4.4 item 7: a cumulative-versus-incremental guard, a run-count +dedupe rule, and an index for non-root rows. Measured cost of the wider scan: 2.60 s against +0.68 s for one spec over 30 days, a 4.96x row fan-out. Do not ship it before item 4, or a +span-focus query on a busy project will hit the statement timeout. + +**4. Stop reading JSONB on the chart path. Medium to large. Improves everything, permanently.** + +Section 5.5 shows why: reading one 8-byte number out of a 10 KB JSONB value costs a 10 KB +detoast, and the planner already does everything it can. The fix is storage, not SQL. Two shapes, +in increasing cost: + +- **Hot columns.** Promote duration, status, token counts, cost, harness, agent reference and + configured model into typed columns on `spans` at ingest. Charts stop touching TOAST entirely. +- **A per-run facts table.** One row per run with those fields plus provenance, and optional child + facts for invoked tools and models. This also gives invoked tools and resolved models a home + that does not require copying one value onto the root span, and it is what item 3 would + otherwise force. + +Both need a plan for late-arriving spans: a run's two OTLP batches can arrive seconds apart, so +any fact row must be updated idempotently and the page needs a stated freshness policy. + +**5. Add a dimension to the query. Medium. Unlocks every "metric by category" chart.** + +Today no request can split a numeric statistic by a category (section 4.3). The obvious fix, one +`group_by` JSON path per query, is the wrong shape: it forces every metric in the request to share +one dimension, and it extends the arbitrary-path protocol that item 2 is trying to replace. Under +a typed contract, each requested series should name its own optional dimension from the enum, +which also makes a cardinality cap natural to express and enforce. Either way it needs that cap, +because `freq` and `uniq` are uncapped today and one metric at 590 buckets is already 897 KB. + +**6. Per-user attribution. Small, once item 2 or item 4 lands.** + +`created_by_id` is already a typed column on every row +(`api/oss/src/dbs/postgres/tracing/mappings.py:210`). It needs to become a nameable dimension, not +a value copied into JSON. It is worth nothing until projects have more than one writer, and it +needs the "who is the user" decision from section 7 first. + +**7. Pre-aggregation. Large, and not automatically the answer.** + +A rollup table keyed by `(project_id, bucket, harness, model, variant)` would turn every query in +this report into an index-only scan of a few hundred rows. Three reasons to sequence it last. +Ordinary count and sum rollups cannot reproduce today's exact `percentile_cont` p95 +(`utils.py:1263-1287`); you would be trading exactness for sketches, which is a product decision. +That key omits status, user, tool, skill, trace type and timezone, so it answers fewer questions +than it looks like it does. And it has the same late-arriving-batch problem as item 4, with less +room to fix it. Normalize typed facts first, measure production, then roll up the specific queries +that justify it. + +**Semantic-convention attributes v2 would add or change:** + +- Map `gen_ai.usage.cost` to `ag.metrics.costs.cumulative.total` in + `GENAI_SEMCONV_ATTRIBUTES_EXACT` (`logfire_adapter.py:148-196`). Cumulative, not incremental: + the value is the run's aggregate total, and calling it incremental would double-count against + the runner's own spans under `focus=span`. Mapping it at ingest is better than changing the + tracing endpoint's default specs, for two reasons. It fixes the path rather than routing around + it, and it also reaches the second consumer of this engine: the evaluations service builds its + own metric list, which includes `attributes.ag.metrics.costs.cumulative.total` + (`api/oss/src/core/evaluations/service.py:141-158`), and passes its own specs + (`service.py:1565-1571`). It never reads `DEFAULT_ANALYTICS_SPECS`, so a default-spec change + would leave evaluation cost metrics as empty as they are now. +- Do not add an `ag.meta.model`. `ag.meta.request.model` and `ag.meta.response.model` already + exist (`logfire_adapter.py:150,160`) and already mean the two different things people want. A + run can call several models, so copying one of them onto the root span loses information; the + child facts in item 4 are the right home. +- Map `gen_ai.agent.name` to `ag.meta.agent.name`, beside the existing + `gen_ai.agent.description` mapping. It is the only `gen_ai.agent.*` key with no mapping. +- Add a skill-invocation signal from the runner, as a span or event carrying the skill's identity. + Neither the configured array nor `ag.meta.skills.loaded` means invoked. + +### 8.5 What still has to be decided outside this report + +These are not analytics questions, but the page cannot ship without answers. + +- **Observability and the deprecated routes.** The sidebar already exposes an Observability page + (`web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx:143`) built on the same + endpoint. Does Analytics replace it, or sit beside it? If it replaces it, plan the URL + migration and the removal of `analyticsToGeneration`. Separately, `POST + /tracing/analytics/query` is marked deprecated and has no caller in `web/`, and `POST + /tracing/spans/analytics` has none either; both are still in the generated clients, so removing + them needs a client-compatibility decision. +- **Rollout.** Put the page behind a flag until the cost coverage question is answered and the + metric semantics in section 7 are written down. The existing per-user exploration switches in + `web/oss/src/state/settings/featureFlags.ts:6-20` are a local precedent, but a beta of this kind + wants an org or project level flag the team can set, not a per-browser toggle. +- **Authorization and tenancy tests.** The route checks `Permission.VIEW_SPANS` and scopes the + query to `request.state.project_id` (`router.py:1284-1290`). Nothing tests that a filter cannot + reach another project's rows, that reference filters respect app scope, or that per-user + filters cannot enumerate other accounts. +- **Test coverage.** One unit test touches analytics + (`api/oss/tests/pytest/unit/tracing/test_analytics_bucket_order.py`), and it asserts bucket + ordering. Nothing pins the 1024-bucket cap, the interval widening, `focus` semantics, the + percentile output shape, the `freq` shape, filter-operator validation, the empty-200 behaviour, + bucket boundaries across a daylight saving transition, or partial coverage. Every one of those + is a behaviour the page would depend on. +- **A performance gate.** Before the page leaves beta, measure it under concurrent load on + production-shaped data, and set a documented maximum window and bucket count. + +### 8.6 OSS and EE + +Both editions run the same analytics SQL. The application entrypoint imports the OSS DAO +(`from oss.src.dbs.postgres.tracing.dao import TracingDAO`, `api/entrypoints/routers.py:70`), and +`api/ee/src/dbs/postgres/tracing/dao.py` defines only `TracingRetentionDAO` (`:77`), which deletes +old traces. There is no EE analytics implementation to diverge. + +What does differ in EE: the throttling middleware and its entitlement categories +(`api/ee/src/middlewares/throttling.py`, `api/ee/src/core/access/entitlements/types.py`), and +retention, which bounds how much history a query can reach. Every capability finding in this +report therefore holds on both editions. Every rate-limit finding is EE-only. Performance findings +were taken on both, on stack A (EE) and stack B (OSS), and agree in shape. + +### 8.7 Open items + +Things this report could not settle, listed so nobody assumes they were checked. + +1. **The cause of the cost and token-split coverage collapse.** We measured it on two stacks and + did not investigate why. This blocks the cost tile and it is the most important open item here. +2. **Whether production shows the same coverage.** Every percentage here comes from local dev + stacks. +3. **How the query behaves under concurrency and on production-shaped data.** Every timing here is + single-user and warm-cache. The 10x extrapolation used duplicated rows, which understates the + real cost. +4. **Whether cloud projects have more than one distinct `created_by_id`.** If they do not, + per-user analytics has no meaning regardless of backend work. +5. **The bare-POST corner of the legacy `errors` defect.** A POST with no query params and no body + is the only shape that could reach `filtering=None`. One curl settles it. +6. **Whether an adapter-written `ag.metrics.costs.cumulative.total` survives ingest's roll-up.** + The roll-up writes only on a non-zero total (`trees.py:319-341`), which suggests it would, but + nothing tests it. + +--- + +## Appendix A: probe index + +Each probe was one live HTTP call whose request and response were saved. Those files are not in +the repository. The table below says what each probe tested and which section uses its result, so +a reader who wants to re-run one can rebuild the request from the section that cites it. + +| Probe | What it tested | Result used in | +|---|---|---| +| 00 | Baseline run count | 4.4 item 1 | +| 01 | `ag.metrics.costs.cumulative.*` specs | 4.4 item 4 | +| 02 | `gen_ai.usage.cost` coverage, both stacks | 4.4 item 4 | +| 03 | Legacy endpoint under `focus=span` | 4.4 item 4 | +| 04a-c | Unfiltered / errored / non-errored counts | 4.4 item 6 | +| 05 | PR #5648's `eq` / `ERROR` filter | 4.4 item 6 | +| 06 | Traces with errored children and clean roots | 4.4 item 6 | +| 07 | Token totals and splits, both stacks | 4.4 item 5 | +| 08a-c | Harness kind, harness object, json spec | 4.4 item 9 | +| 09 | Configured model and provider frequencies | 4.4 item 8 | +| 10 | Agent reference breakdown | 4.4 item 10 | +| 11a-c | `created_by_id` filters, unknown-field widening | 3.8, 4.4 item 13 | +| 12 | Distinct `created_by_id` per project | 4.4 item 13 | +| 13a-c | Tool name under each `focus` value | 3.5, 4.4 item 7 | +| 15 | `interval=1` over 7 days, silent coarsening | 3.7 | +| 16a-b | No-interval versus daily percentiles | 4.4 items 2, 3 | +| 17a-b | UTC versus offset midnight bucket edges | 4.4 item 1 | +| 18a-b | Legacy endpoint with and without a filter | 3.4 | +| 19 | 130 sequential requests to both routes | 5.4 | +| 20 | Wide window, many specs | 5.3 | +| 22, 23 | Deep JSON filters on harness and model | 4.4 items 11, 12 | +| 24 | Deprecated route parity | 3.3 | +| 25 | Four spec types against the tools array | 4.4 item 7 | +| 26 | Per-model filtered latency | 4.4 item 8 | +| 27 | Duration spec plus harness spec, no split | 4.3 | +| 29, 36 | Streaming flag, connection mode, permissions | Appendix B | +| 31 | Explicit histogram bins | 4.4 item 3 | +| 32 | Cache-token paths | 4.4 item 5 | +| 34 | Array index 0 on tools and skills | 4.4 items 7, 14 | +| 35 | Existence filter on configured skills | 4.4 item 14 | +| 37 | Column names as metric specs | 3.1 | +| 39 | Malformed `oldest` | 3.8 | + +Section 5 also cites four benchmark artifacts, likewise kept outside the repository: `exp-1.out` +(the query plan for one spec over 30 days), `exp-8.out` (eight specs over 30 days), +`exp-1-10x.sql` (the synthetic row multiplier), and a `results-*.json` timing series. Every number +this report draws from them is quoted at the point of use. + +## Appendix B: three dimensions nobody asked for + +Found while probing, all working today with one `categorical/single` spec each, all on stack A +over the probe window: + +- **Connection mode**, `attributes.ag.data.parameters.agent.llm.connection.mode`: `self_managed` + 7,019, `agenta` 172 (probe 36). +- **Default runner permission**, `attributes.ag.data.parameters.runner.permissions.default`: + `allow_reads` 7,357, `allow` 90, `ask` 27 (probe 36). +- **Streaming flag**, `attributes.ag.flags.stream`: true 409, false 6,946 (probe 29). + +None is on the wish list. Each costs one spec and answers a real operational question. From d1c55ad8aa5560ed88f1c7c549059e01dd5c51d5 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Mon, 3 Aug 2026 14:01:15 +0600 Subject: [PATCH 5/9] Refine agent analytics plan and documentation - Updated the plan to clarify backend prerequisites for Phase 0, including distinguishing between empty results and failures in API responses. - Enhanced Phase 1 details to specify the structure of analytics queries and the validation process for metrics. - Revised Phase 3 to include specific state management for the analytics page and ensure proper handling of loading and empty states. - Expanded Phase 4 to detail the summary panel and chart requirements, including coverage-gated metrics and responsive design considerations. - Clarified the definitions of failed runs and health scores in the status documentation, ensuring consistency with backend expectations. - Added details on the necessary data paths and metrics for the new analytics page, emphasizing the need for coverage-gated data due to previous collapses. - Documented open questions and risks to address during the build process, focusing on validation of metrics and response handling. --- docs/design/agent-analytics/README.md | 5 +- docs/design/agent-analytics/context.md | 38 ++- docs/design/agent-analytics/data-contract.md | 227 +++++++++++++----- docs/design/agent-analytics/plan.md | 240 ++++++++++++------- docs/design/agent-analytics/research.md | 21 +- docs/design/agent-analytics/status.md | 40 ++-- 6 files changed, 388 insertions(+), 183 deletions(-) diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md index 83000fb791..aa2ede2966 100644 --- a/docs/design/agent-analytics/README.md +++ b/docs/design/agent-analytics/README.md @@ -43,8 +43,9 @@ collide with `context.md`. - **Span**: one unit of work inside a run (a model call, a tool call, or the agent step itself). Model name and tool name live on child spans, not on the root span. - **Root span**: the top span of a run. Today's analytics endpoint only reads root spans. -- **Failed run**: a run whose root span status is `ERROR`, a run-level outcome. Counted from - the `status_code` column, not from the errors metric. +- **Failed run**: a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome + (there is no `STATUS_CODE_OK` on root spans; success is the complement). Counted from the + `status_code` column, not from the errors metric. - **Error**: any errored step inside a run. A run can contain errors and still succeed, so an error count is not a failed-run count. - **Success rate**: successful runs over total runs, where a failed run is the one above. diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md index ace290612f..fa79e0d1cf 100644 --- a/docs/design/agent-analytics/context.md +++ b/docs/design/agent-analytics/context.md @@ -22,24 +22,35 @@ It shows: control and its `SortResult`), so it supports the standard presets and a custom start-and-end range. It is not a fixed list of options. It opens on the last 7 days. - A summary panel: a health donut (0 to 100, with a Healthy / Watch / At risk band and a - one-line read-out) and four stat tiles (Total runs, Success rate, Avg latency, Total cost). - Each tile shows a change badge against the previous window of equal length and a small trend - line of the current window. -- Four charts in a grid, each with hover tooltips and a legend whose entries toggle series on - and off: + one-line read-out) and stat tiles (Total runs, Avg latency, p95 latency, Total tokens, and a + coverage-gated Total cost that shows an explicit "not available for this window" instead of a + zero when cost coverage is low). Each tile shows a change badge against the previous window of + equal length and a small trend line of the current window. +- Charts in a grid, each with hover tooltips and a legend whose entries toggle series on and + off: - Runs: stacked bars of successful and failed runs per bucket. - Latency: bars of average latency per bucket, with a p95 marker line; the tooltip shows average, p95, min, and max. - - Costs: stacked bars of prompt cost and completion cost per bucket. - - Tokens: stacked bars of prompt tokens and completion tokens per bucket. + - Tokens: total tokens per bucket, with a coverage label; the prompt/completion split renders + as stacked bars only when its coverage gate passes. + - Runs per harness, runs per configured model, and runs per agent: category breakdowns, one + `categorical/single` spec each. + + There is no Costs prompt/completion chart: those cost paths hold no data on agent runs, and + the one working cost path (`gen_ai.usage.cost`) is a coverage-gated total shown as the tile + above. See data-contract.md and capability-review.md. ## Locked scope decisions These decisions are settled and drive the plan. Do not reopen them without the requester. -1. Frontend-first. Build the four charts above, the four stat tiles, the health donut, the - Agents filter, and the time-range control against today's endpoint. The endpoint already - returns every field these need, so this scope ships without any change under `api/`. +1. Frontend-first, with two backend prerequisites. Build the charts above, the stat tiles, the + health donut, the Agents / Harness / configured-Model filters, and the time-range control + against today's endpoint. The endpoint returns most fields these need directly, but two + backend items gate a trustworthy release and are tracked as Phase 0 in plan.md: making a + killed or rejected query distinguishable from a genuinely empty one, and investigating the + mid-July collapse in cost and token-split coverage. The cost tile and the token split stay + coverage-gated until the second is understood. 2. Health donut computed in the browser. The health score is the success rate: `round(100 x successRate)`, banded Healthy at 85 and above, Watch from 65 to 84, At risk @@ -52,9 +63,10 @@ These decisions are settled and drive the plan. Do not reopen them without the r agent in the project by default. The Agents multi-select narrows the set, so the default query carries no single-app reference filter. -4. A failed run means a run whose root span status is `ERROR`, a run-level outcome. Success - rate and the health score build on that, not on a count of errored steps. data-contract.md - has the definition and the query it needs. +4. A failed run means a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome + (there is no `STATUS_CODE_OK` on root spans; success is the complement). Success rate and the + health score build on that, not on a count of errored steps. data-contract.md has the + definition and the query it needs. ## Showing model usage and tool usage needs backend work diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md index fdf219832a..5876cfe4bb 100644 --- a/docs/design/agent-analytics/data-contract.md +++ b/docs/design/agent-analytics/data-contract.md @@ -5,6 +5,10 @@ This page talks to one endpoint, `POST /spans/analytics/query`, through the exis page builds and the response fields the new mapper reads, and it flags the one frontend extension: passing explicit metric specs. +Every request and response shape below is taken from the capability review +(`capability-review.md`), which verified each one against live calls on two stacks. Where a +value still has to be confirmed against a live response during the build, the text says so. + ## Request The request carries four kinds of fields. Classified by what each field is, not by which @@ -14,95 +18,194 @@ chart it feeds: from `projectIdAtom`. `oldest` and `newest` are ISO bounds taken from the selected `SortResult`, the same range object the observability windowing uses: a standard preset resolves to a start with `newest` omitted (meaning "now"), and a custom range supplies both - `oldest` and `newest`. Any window is valid; there is no fixed set of options. -- **Policy** (how to slice and aggregate): `focus = "trace"` (root spans only, the only - value that works today), and `interval` (bucket size in minutes) from - `calculateIntervalFromDuration`. + `oldest` and `newest`. Any window is valid; there is no fixed set of options. Align `oldest` + to the viewer's local midnight, because buckets are fixed-width offsets from `oldest`, not + calendar days (see "Response quirks" below). +- **Policy** (how to slice and aggregate): `focus = "trace"` and `interval` (bucket size in + minutes) from `calculateIntervalFromDuration`. The endpoint ignores `focus` entirely and + always reads root spans, so `focus = "trace"` is the honest label rather than a switch; + never rely on the echoed `focus` to confirm behaviour. Omit `interval` to collapse the + window into one exact bucket (used for the window-level summary numbers below). - **Data selection** (what to measure): `specs`, a list of metric specs naming the JSON paths to summarize. See below. -- **Data filter** (which spans qualify): `filter`, a `{conditions: [...]}` object. At - project scope with no agent selected, omit the reference conditions so the query spans the - whole project. For selected agents, add `references` conditions on the chosen agent ids. The - exact encoding, a single `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per - agent combined with `or`, follows the existing filter builder; see the plan.md open - questions. +- **Data filter** (which spans qualify): `filter`, a `{conditions: [...]}` object. Every + query carries a base `{field: "trace_type", operator: "is", value: "invocation"}` condition + so annotation traces (evaluator and human-annotation runs) do not inflate the run count and + the latency, cost, and token numbers. At project scope with no agent selected, add nothing + else so the query spans the whole project. For selected agents, add `references` conditions + on the chosen agent ids. Two things to validate against the existing filter builder in + Phase 2: the exact enum literal for `trace_type` (mirror the `status_code` lesson below — + the accepted value may be prefixed), and whether multiple agent ids encode as a single + `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per agent combined with + `or`. A fallback for the run count that needs no `trace_type` literal: read the `freq` array + of the `attributes.ag.type.trace` categorical spec, which already carries the + invocation-versus-annotation split. + + A filter typo does not error. An unknown field is logged and dropped, which **widens** the + result rather than narrowing it, and an invalid operator or value returns HTTP 200 with an + empty result. Verify every filter the page sends once against a known-good window during + development; there is no runtime signal. ### The `specs` extension `fetchSpansAnalytics` omits `specs` today, so the backend applies its default set. The -defaults give totals for cost, tokens, duration, errors, and the trace and span type counts. -That is enough for run count, average latency, total cost, and total tokens, but not for the -prompt-and-completion split or for p95, min, and max latency. +defaults give totals for cost, tokens, duration, errors, and the trace and span type counts, +but they read the canonical `ag.metrics.costs.cumulative.*` cost paths, which hold no data on +agent runs (see the cost note below). The page must pass an explicit `specs` list. Add an +optional `specs` field to `SpansAnalyticsParams` and serialize it to a JSON-string query param +exactly as `filter` is serialized. -To get those, pass an explicit `specs` list. Add an optional `specs` field to -`SpansAnalyticsParams` and serialize it to a JSON-string query param exactly as `filter` is -serialized. The specs the page needs, by path and type: +The specs the page sends, by path and type: | Purpose | path | type | fields read | | --- | --- | --- | --- | -| Run count | `attributes.ag.type.trace` | categorical/single | `count` | +| Run count | `attributes.ag.type.trace` | categorical/single | `count` (or the `freq` array) | | Latency | `attributes.ag.metrics.duration.cumulative` | numeric/continuous | `count`, `sum`, `min`, `max`, `pcts.p95` | -| Prompt cost | `attributes.ag.metrics.costs.cumulative.prompt` | numeric/continuous | `sum` | -| Completion cost | `attributes.ag.metrics.costs.cumulative.completion` | numeric/continuous | `sum` | -| Prompt tokens | `attributes.ag.metrics.tokens.cumulative.prompt` | numeric/continuous | `sum` | -| Completion tokens | `attributes.ag.metrics.tokens.cumulative.completion` | numeric/continuous | `sum` | - -The `type` strings are verified against the backend: `MetricType` -(`api/oss/src/core/tracing/dtos.py`) has only `numeric/continuous` and `numeric/discrete` -(there is no plain `numeric`), and the backend's own `DEFAULT_ANALYTICS_SPECS` -(`api/oss/src/core/tracing/service.py`) uses `numeric/continuous` for errors, costs, and -tokens. Use `numeric/continuous` for every number metric above; a bare `numeric` is silently -dropped. - -p95 is **nested**, not a flat field. The numeric/continuous reducer (`parse_pcts` in -`api/oss/src/dbs/postgres/tracing/utils.py`) emits percentiles under a `pcts` object, so the -value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and `max` are flat siblings and -read directly; only the percentiles sit one level down. Still confirm against one live -response in Phase 2, but expect the nested shape. +| Total cost (coverage-gated) | `attributes.gen_ai.usage.cost` | numeric/continuous | `sum`, `count` | +| Total tokens | `attributes.ag.metrics.tokens.cumulative.total` | numeric/continuous | `sum`, `count` | +| Prompt tokens (coverage-gated split) | `attributes.ag.metrics.tokens.cumulative.prompt` | numeric/continuous | `sum` | +| Completion tokens (coverage-gated split) | `attributes.ag.metrics.tokens.cumulative.completion` | numeric/continuous | `sum` | + +The category breakdown charts (runs per harness, per configured model, per agent) each send +one extra `categorical/single` spec and read its `freq` array: + +| Purpose | path | type | fields read | +| --- | --- | --- | --- | +| Runs per harness | `attributes.ag.data.parameters.agent.harness.kind` | categorical/single | `freq` | +| Runs per configured model | `attributes.ag.data.parameters.agent.llm.model` | categorical/single | `freq` | +| Runs per agent | `attributes.ag.references.workflow_variant.id` and `attributes.ag.references.application_variant.id` | categorical/single | `freq` (union both families) | + +Notes on the specs: + +- **The `type` strings are verified against the backend.** `MetricType` + (`api/oss/src/core/tracing/dtos.py`) has only `numeric/continuous` and `numeric/discrete` + (there is no plain `numeric`). Use `numeric/continuous` for every number metric; a bare + `numeric` is silently dropped. +- **p95 is nested, not a flat field.** The numeric/continuous reducer emits percentiles under + a `pcts` object, so the value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and + `max` are flat siblings and read directly; only the percentiles sit one level down. All 27 + percentile levels ship on every numeric/continuous spec, so min/max/p95 latency need no + backend work. Confirm the nested shape against one live response in Phase 2. +- **Cost has no prompt/completion split, and its one working path is coverage-gated.** The + canonical `ag.metrics.costs.cumulative.total`, `.prompt`, and `.completion` paths hold no + data on agent root spans (the cost roll-up never crosses the run's OTLP batch boundary to + reach the root span). The only populated cost path is `attributes.gen_ai.usage.cost`, the + harness's own reported run total, and its coverage collapsed to near zero in mid-July on + both measured stacks for a cause nobody has established yet. The cost tile therefore renders + only when coverage clears a threshold (see "Coverage gating" below), and there is no cost + split to chart. Do not add a Costs prompt/completion chart. +- **Total tokens works with a coverage label; the split moves with cost.** The + prompt/completion token split shares the same mid-July collapse as cost: it is real where + the pipeline works and a flat zero band where it does not, which reads as data and is worse + than an empty chart. Coverage-gate the split the same way as cost. ### Failed runs come from a second, filtered query -A failed run is a run whose root span `status_code` is `ERROR`. `status_code` is a +A failed run is a run whose root span `status_code` is `STATUS_CODE_ERROR`. `status_code` is a table column, and metric specs read only the `attributes` JSON (`build_extract_cte` extracts `attributes #> path`), so no spec can target it. Instead, the page runs a second analytics query for the same window with an added filter condition on `status_code` and reads the run count: -- Failed-run query filter: the agent conditions above (if any), plus - `{field: "status_code", operator: "eq", value: "ERROR"}`. `status_code` is a first-class - filter field (`api/oss/src/core/tracing/utils/filtering.py`). +- Failed-run query filter: the base `trace_type` condition and the agent conditions (if any), + plus `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}`. `status_code` is + a first-class filter field (`api/oss/src/core/tracing/utils/filtering.py`). The operator + must be `is` and the value must be the full enum literal `STATUS_CODE_ERROR`. An `eq` + operator or a bare `ERROR` value both raise inside the backend and come back as HTTP 200 + with an empty result, which would silently report zero failures forever. +- There is **no** `STATUS_CODE_OK` on root spans; a clean run's status is `STATUS_CODE_UNSET`. + So success is the complement of failure, never a positive `STATUS_CODE_OK` filter. - It needs only the run-count spec (`attributes.ag.type.trace`), so it is a cheap query. - Per bucket: `failed` = the filtered run count; `success` = the unfiltered run count minus it. -The page therefore issues two queries per window (unfiltered for totals and latency/cost/ -tokens, status-filtered for failed runs), and it fetches a current and a previous window, so -four analytics calls in total. +**Blind spot to state in the UI.** Root-span status does not see failures inside a run that +recovered a clean root. On the measured data that is about 1.2% of runs. The v1 definition of +a failed run is "the root span errored"; say so in a tooltip. Catching in-run failures needs +`focus = "span"`, which does not work today (see the deferred boundary). + +### The queries per window + +Two shapes are needed because window-level percentiles cannot be composed from per-bucket +percentiles (averaging or maxing per-bucket p95s is wrong for any non-uniform distribution). +Counts and sums do compose, so they can be summed from the bucketed calls; only p95 forces the +no-interval call. + +For the **current** window: + +1. **Bucketed, unfiltered** (`interval` set): drives the Runs, Latency, Cost, and Tokens + charts. Also carries the per-bucket failed count once combined with query 2. +2. **Bucketed, status-filtered**: the per-bucket failed run count for the stacked Runs chart. +3. **No-interval, unfiltered** (`interval` omitted, one exact bucket): the exact summary-tile + numbers, including the window-level p95 latency that the bucketed call cannot produce. + +For the **previous** comparison window (change badges only): + +4. **No-interval, unfiltered**: exact previous-window totals and p95. +5. **No-interval, status-filtered**: the previous-window failed count for the previous success + rate. + +Derive the previous window as the equal-length window immediately before the selected one: for +`[oldest, newest]`, the previous window is `[oldest - (newest - oldest), oldest]`. This works +for both preset and custom ranges. + +Keep any single request at or under about eight specs, and fan the calls out in parallel with a +per-call error state. On the measured data a 7-day window costs about 0.26 s and a 30-day +window about 1.7 s for the shipped six-spec shape, so 7 days is the default and 30 or 90 days +is an explicit user choice with a loading state. ## Response fields the mapper reads -The response is `{buckets: [{timestamp, metrics: {: {: value}}}]}`. The new -mapper produces, per bucket: +The response is `{buckets: [{timestamp, interval, metrics: {: {: value}}}]}`. The +new mapper produces, per bucket: - `failed` = the `type.trace` count from the status-filtered query (runs whose root span is - `ERROR`). This is a true failed-run count, never larger than the total. + `STATUS_CODE_ERROR`). This is a true failed-run count, never larger than the total. - `success` = the unfiltered `type.trace` count minus `failed`. It cannot go negative, so no flooring is needed. This departs from the existing observability mapper, which subtracts the `errors.cumulative` sum. That sum counts errored steps, not failed runs, and can exceed the run count, so this page uses the run-level status instead. -- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. +- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. For a window average, sum + the duration `sum` fields and divide by the summed `count` fields, or read `mean` from the + no-interval call. Never average per-bucket means. - `latencyMin`, `latencyMax` = the flat `min` / `max` duration fields. -- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95` (see the note above the - response section); `metricField`'s flat one-level read does not reach it, so the mapper - needs a small `pcts` accessor. -- `costPrompt`, `costCompletion` = the two cost sums. -- `tokensPrompt`, `tokensCompletion` = the two token sums. These split fields are computed - from per-part token counts, so providers that report only total tokens leave the - prompt/completion split at zero while `total` is correct; verify the split is non-zero on - live traffic in Phase 2, or the Costs/Tokens split bars render empty for those agents. - -And window totals for the four stat tiles: total runs, success rate, average latency, total -cost, plus the same figures for the previous window so the change badges have a baseline. +- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95`. `metricField`'s flat + one-level read does not reach it, so the mapper needs a small `pcts` accessor. The + window-level p95 tile reads this from the no-interval call, not from the buckets. +- `costTotal` = the `gen_ai.usage.cost` `sum`. Rendered only when its coverage gate passes. +- `tokensTotal` = the `tokens.cumulative.total` `sum`. +- `tokensPrompt`, `tokensCompletion` = the two token split sums. Rendered only when their + coverage gate passes; below it they are suppressed, not shown as zero. + +And window totals for the summary tiles: total runs, success rate, average latency, p95 +latency, total tokens, and coverage-gated total cost, plus the same figures for the previous +window so the change badges have a baseline. Lower latency and lower cost are the "good" +direction for the badge colour. + +### Coverage gating + +Cost and the token split can be perfectly expressible and still hold no data, because coverage +is zero on the window. For each gated metric, compare its spec `count` against the run count +for the same window. Render the tile or chart only when coverage clears a threshold; below it, +show "cost data is not available for this window" (or the token equivalent) rather than a zero. +A zero reads as a real measurement and is worse than an explicit unavailable state. Total +tokens is shippable with a coverage label rather than a hard gate, because its coverage is +partial rather than collapsed. + +### Response quirks the mapper must handle + +- **Empty buckets are omitted.** The mapper must build its own x-axis, or gaps read as missing + days rather than as zero days. +- **The top-level `count` is the number of buckets, not the number of runs.** Read run counts + from the per-bucket `type.trace` metric. +- **The requested `interval` may have been coarsened silently** above 1024 buckets. Only + `buckets[].interval` reports what actually ran. If the UI lets the user pick a period, read + the effective interval back and label the chart with it. +- **A wrong filter renders as zero, not as an error**, as noted under the request filter. +- **Buckets are 24-hour periods aligned to `oldest`**, not calendar days, and `date_bin` steps + by a fixed duration, so buckets drift off local midnight across a daylight-saving transition. + Calendar months are not expressible at all. Label the axis as 24-hour periods; do not offer + a month period. ## Health score (browser-side, not in the contract) @@ -116,11 +219,13 @@ Computed from the window totals, never sent to the backend: - Low-traffic guard: below a minimum run count in the window, do not band the score. Show a neutral "Not enough runs yet" donut instead, so a single failure in a quiet window does not read as At risk. The threshold is a build-time tuning value (start around 20 runs). The stat - tiles and the four charts still render whatever data exists. + tiles and the charts still render whatever data exists. ## Boundary for the deferred charts -The deferred Tools and Models charts need `focus = "span"` and, for per-model cost, a -`group_by` field on a spec. Neither works on today's endpoint. Keep the request builder able -to take a different `focus` and extra specs without reshaping, so the follow-up change adds a -second query rather than rewriting this one. +The deferred Tools and resolved-Models charts need `focus = "span"`, which is accepted, echoed, +and ignored today, so it always reads root spans. Actual tool names live at +`attributes.ag.meta.tool.name` and resolved model ids at `attributes.ag.meta.request.model`, +both on child spans only. Per-model cost additionally needs a group-by dimension the endpoint +does not have. Keep the request builder able to take a different `focus` and extra specs +without reshaping, so the follow-up change adds a second query rather than rewriting this one. diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md index 422dc5080f..56ebe55ade 100644 --- a/docs/design/agent-analytics/plan.md +++ b/docs/design/agent-analytics/plan.md @@ -1,11 +1,34 @@ # Plan Build in four phases. Each phase is independently reviewable and leaves the app working. The -deferred Tools and Models charts are a fifth phase that lands after a backend change; the -first four phases finish the locked scope. +deferred Tools and resolved-Models charts are a fifth phase that lands after a backend change; +the first four phases finish the locked scope. File paths follow the existing observability feature so the new page reads as a sibling of -it, not a new pattern. +it, not a new pattern. Every request and response shape referenced here is specified in +data-contract.md and verified in capability-review.md. + +## Phase 0: backend prerequisites + +Two backend items gate a trustworthy v1. Neither is frontend work; list and track them before +the page ships behind its flag. + +- **Make a killed or rejected query say so.** Today a statement-timeout, a rejected filter, a + malformed window, and a genuinely empty window all return `{"count": 0, "buckets": []}` with + HTTP 200. The page cannot tell "no data" from "the query died", and the cost coverage gate + in Phase 4 depends on that distinction. The fix crosses two layers: raise a typed + timeout/filtering error in core and translate it at the router into an `HTTPException` + (504 for timeout, 4xx for a bad filter or window), and add a per-metric `sample_count` to + the response. See capability-review.md section 8.4 item 1. +- **Investigate the cost and token-split coverage collapse.** `attributes.gen_ai.usage.cost` + and the `tokens.cumulative.prompt`/`.completion` split were populated on 70–95% of runs in + early July on both measured stacks and fell to roughly zero within a week, cause unknown. + Until this is understood, the cost tile has nothing dependable to show. File it; the first + four diagnostic checks are in capability-review.md section 4.4 item 4. Both measured stacks + were local dev; whether production shows the same collapse is unverified. + +Done when: an empty result is distinguishable from a failure at the API surface, and the cost +coverage question has an owner and a tracking issue. ## Phase 1: page shell, route, and navigation @@ -34,26 +57,33 @@ Goal: the page can fetch mapped analytics for the current and previous windows. entities package still builds (`pnpm turbo run build --filter=@agenta/entities`). - Add a new mapper next to the existing one in `web/oss/src/services/tracing/lib/` (do not change `analyticsToGeneration`; the observability page depends on it). The new mapper reads - the split cost and token paths and the duration min, max, and p95 fields, combines the - unfiltered and status-filtered run counts into success and failed, and returns the per-bucket - and window-total shape in data-contract.md. Reuse `metricField` and - `calculateIntervalFromDuration`, and add a small `pcts` accessor for p95. -- Add a fetch function in `web/oss/src/services/tracing/api/` that builds the project-scope - conditions (no single-app reference by default; the selected agents' reference conditions), - passes the explicit `specs`, and calls `fetchSpansAnalytics`. It issues two queries per - window: the unfiltered one for totals and latency/cost/tokens, and a second one that adds - `{field: "status_code", operator: "eq", value: "ERROR"}` and reads the run count as failed - runs (see data-contract.md). The failed-run query needs only the run-count spec. -- The spec `type` strings (`numeric/continuous`) and the nested `pcts.p95` shape are already - verified in the backend code (see data-contract.md); validate them against one live response - while building this phase, and check the prompt/completion split reads non-zero on real - traffic. + the duration min, max, sum, count, and nested `pcts.p95` fields, the total cost and total + token sums with their `count` for the coverage gate, the coverage-gated prompt/completion + token split, and the category `freq` arrays for the harness, configured-model, and agent + breakdowns. It combines the unfiltered and status-filtered run counts into success and + failed, and returns the per-bucket and window-total shape in data-contract.md. Reuse + `metricField` and `calculateIntervalFromDuration`, and add a small `pcts` accessor for p95 + and a `freq` reader for the breakdowns. +- Add a fetch function in `web/oss/src/services/tracing/api/` that builds the base conditions + (the `trace_type is invocation` condition, plus the selected agents' `references` + conditions), passes the explicit `specs`, and calls `fetchSpansAnalytics`. Per window it + issues the queries in data-contract.md: a bucketed unfiltered query for the charts, a + bucketed status-filtered query for failed runs, and a no-interval unfiltered query for the + exact summary-tile numbers including window-level p95. The status-filtered query adds + `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` and reads the run count + as failed runs; it needs only the run-count spec. The previous window needs only the + no-interval unfiltered and status-filtered calls for the change badges. +- Validate three things against one live response while building this phase, because the + backend fails silently on all three: the nested `pcts.p95` shape reads correctly; the exact + enum literal for the `trace_type` filter and the `STATUS_CODE_ERROR` filter both narrow + rather than returning an empty 200; and the coverage of `gen_ai.usage.cost` and the token + split on real traffic (they may be near zero — this is the Phase 0 investigation surfacing). - Derive the previous comparison window as the equal-length window immediately before the selected one: for `[oldest, newest]`, the previous window is `[oldest - (newest - oldest), oldest]`. This works for both preset and custom ranges. -Done when: a temporary log or test shows correct totals for a known window, and the previous -window is fetched for comparison. +Done when: a temporary log or test shows correct totals for a known window, the failed-run +filter is confirmed to narrow the result, and the previous window is fetched for comparison. ## Phase 3: state and page assembly @@ -62,109 +92,147 @@ Goal: the page is interactive; controls drive the data. - Add atoms under `web/oss/src/state/analytics/` following `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default: the last 7 days), an agents-filter atom, a current-window query atom, and a previous-window - query atom (or one query returning both). Each query atom drives the two calls per window - (unfiltered and status-filtered). Key every atom on project id, time range, and the agents - filter. Set `staleTime` to one minute and `refetchOnWindowFocus: false`. + query atom (or one query returning both). Each query atom drives the calls per window + described in data-contract.md. Key every atom on project id, time range, and the agents + filter. Set `staleTime` to one minute and `refetchOnWindowFocus: false`. Do not reuse + `observabilityDashboardTimeRangeAtom`; a second consumer already shares it, so give this page + its own atoms. - Build the header controls: the time-range control and the Filters popover with the Agents - multi-select. Reuse the observability time windowing (the `Sort` control and its - `SortResult`) for the time range, so any window, including a custom start-and-end range, - works; do not build a fixed list of range options. Source the Agents options from the - agents atom confirmed in Phase 1. -- Wire loading and empty states following `AnalyticsDashboard.tsx` (antd `Spin`, a "No data" - empty state per card). - -Done when: changing the time range or the agents filter refetches and the page reflects it, -with correct loading and empty states. - -## Phase 4: summary panel and the four charts - -Goal: the full locked-scope UI. - -- Summary panel: a health donut component (recharts `RadialBarChart` or a small SVG ring) + multi-select, a Harness multi-select, and a configured-Model multi-select. Reuse the + observability time windowing (the `Sort` control and its `SortResult`) for the time range, + so any window, including a custom start-and-end range, works; do not build a fixed list of + range options. Offer day, week, and "whole window" as periods; do not offer a month period, + because calendar months are not expressible and a fixed 30-day stride is not a month. Align + `oldest` to the viewer's local midnight and read `buckets[].interval` back to label the axis + with the period that actually ran. Source the Agents options from the agents atom confirmed + in Phase 1. +- Implement four explicit page states per card, following `AnalyticsDashboard.tsx` (antd + `Spin` for loading): data, no data in this window, metric unavailable (coverage below the + threshold), and request failed. The last two exist because a wrong filter or a killed query + renders as an empty success today; do not collapse them into a single "No data" empty state. + +Done when: changing the time range, the agents filter, the harness filter, or the model filter +refetches and the page reflects it, with all four states wired. + +## Phase 4: summary panel and the charts + +Goal: the full locked-scope UI. Every chart on this page means what its title says and needs no +backend capability that does not exist. + +- **Summary panel:** a health donut component (recharts `RadialBarChart` or a small SVG ring) showing `round(100 x successRate)` with the band label and prose. Below the run-count floor - it renders the neutral "Not enough runs yet" state instead of a band. Plus four stat tiles; - each shows the value, a change badge versus the previous window (green when the change is - good for that metric, red otherwise; note that lower latency and lower cost are good), and a - sparkline of the current window. -- Chart components, each a card with a title, a one-line description, a recharts chart, and a + it renders the neutral "Not enough runs yet" state instead of a band. Plus stat tiles for + total runs, average latency, p95 latency, and total tokens, each with a change badge versus + the previous window (green when the change is good for that metric, red otherwise; lower + latency is good) and a sparkline of the current window. Add a coverage-gated total-cost tile + that renders only when cost coverage clears the threshold and otherwise shows an explicit + "cost data not available for this window", never a zero. +- **Charts**, each a card with a title, a one-line description, a recharts chart, and a toggleable legend: - Runs: stacked bars, successful and failed. - Latency: bars of average with a p95 reference line; tooltip shows average, p95, min, max. - - Costs: stacked bars, prompt and completion. - - Tokens: stacked bars, prompt and completion. -- Lay the four charts out in a two-column grid. All colors come from theme tokens and the - theme scale; verify both light and dark themes and hover, empty, and loading states. - -Done when: the four in-scope charts and the summary panel render correctly in both themes, -and `pnpm lint-fix` passes. - -## Phase 5 (deferred, after a backend change): Tools and Models + - Tokens: total tokens per period, with a coverage label. Show the prompt/completion split + as stacked bars only when the split coverage gate passes; below it, show total only. + - Runs per harness: one `categorical/single` spec, per period. + - Runs per configured model: one `categorical/single` spec, per period, labelled "configured + model" (this is the author's alias, not the model that answered). + - Runs per agent: one `categorical/single` spec over the unioned `workflow_variant` and + `application_variant` reference families, per period. +- There is no Costs prompt/completion chart. The cost split does not exist in the data, and the + one working cost path (`gen_ai.usage.cost`) is a coverage-gated total, shown as the tile + above rather than a per-part chart. +- Lay the charts out in a responsive grid. All colors come from theme tokens and the theme + scale; verify both light and dark themes and hover, empty, unavailable, and loading states. + +Done when: the in-scope charts and the summary panel render correctly in both themes across all +four states, and `pnpm lint-fix` passes. + +## Phase 5 (deferred, after a backend change): Tools and resolved Models Not part of this plan's scope. Recorded so Phase 1 to 4 leave the right boundary. The backend work splits into two prerequisites that are **not co-equal**; they unlock different views, so -Phase 5 itself splits into 5a and 5b. See context.md for the verified engine details. +Phase 5 itself splits into 5a and 5b. See context.md and capability-review.md for the verified +engine details. ### Prerequisite 1: span-focus wiring (unlocks 5a) -Thread `focus` through to the base query so `focus = "span"` reads child spans. -`query.focus` reaches `dao.analytics()` but is never passed to `build_base_cte`, which -hardcodes `WHERE parent_id IS NULL`; make that predicate conditional on `focus`. +Thread `focus` through to the base query so `focus = "span"` reads child spans. `query.focus` +reaches `dao.analytics()` but is never passed to `build_base_cte`, which hardcodes +`WHERE parent_id IS NULL`; make that predicate conditional on `focus`. Today the endpoint +accepts, echoes, and ignores `focus`, so nothing on a root-span page can rely on it. -- **Ship a guard with it**: under `focus = "span"`, cumulative metrics double-count (the - rollup lives on the root span). Reject or auto-map a cumulative spec under `focus = "span"`, - and have per-model cost/tokens use the `incremental` paths, not the cumulative ones. +- **Ship three guards with it**, or it produces wrong numbers: under `focus = "span"`, + cumulative metrics double-count because the rollup lives on every ancestor, so per-model + cost/tokens must use the `incremental` paths; a run count must dedupe because `ag.type.trace` + is stamped on every span in a trace; and non-root rows need an index (the current index is + partial on `parent_id IS NULL`, and the wider scan is a ~5x row fan-out). Do not ship + span-focus before Phase 0's failure-visibility fix, or a span-focus query on a busy project + hits the statement timeout and returns an empty 200. ### Prerequisite 2: group-by dimension (unlocks only per-model cost, in 5b) Add a grouping dimension so a numeric metric (cost, tokens) splits by a categorical path (model name) in one call. Prefer a **per-query** dimension over a per-spec `group_by`: it -matches "one view, one breakdown," leaves `MetricSpec` untouched, and confines the nested -`{group_value: stats}` output shape to one code path. This is the harder change; defer it -until 5a has shipped. +matches "one view, one breakdown," leaves `MetricSpec` untouched, confines the nested +`{group_value: stats}` output shape to one code path, and makes a cardinality cap natural to +enforce. This is the harder change; defer it until 5a has shipped. -### 5a: Tools and Models-share (needs prerequisite 1 only) +### 5a: Tools and resolved-Models share (needs prerequisite 1 only) -Once span-focus lands, these need **no** group-by, because a `categorical/single` spec -already returns per-value frequencies: +Once span-focus lands, these need **no** group-by, because a `categorical/single` spec already +returns per-value frequencies: -- Tools horizontal-bar card: `categorical/single` on `span_name` (+ `status_code` for error - rate), `focus = "span"`. -- Models horizontal-bar card with a click-to-filter legend: `categorical/single` on - `attributes.ag.meta.request.model`, `focus = "span"`. -- Models multi-select in the Filters popover. +- Tools horizontal-bar card: `categorical/single` on `attributes.ag.meta.tool.name`, + `focus = "span"`. Note that specs read the `attributes` JSON only, so the tool name must come + from the attribute path, not from the `span_name` column, which no spec can read. A per-tool + error rate needs the group-by dimension or a per-tool filtered call, because `status_code` is + a column and cannot be a spec either. +- Resolved-Models horizontal-bar card with a click-to-filter legend: `categorical/single` on + `attributes.ag.meta.request.model`, `focus = "span"`. This is the model that actually + answered, distinct from the configured-model chart in Phase 4. +- Resolved-model multi-select in the Filters popover. ### 5b: per-model cost/tokens (needs prerequisites 1 and 2) -Adds the numeric-by-model breakdown once the group-by dimension exists, using the -`incremental` cost/token paths under `focus = "span"`. +Adds the numeric-by-model breakdown once the group-by dimension exists, using the `incremental` +cost/token paths under `focus = "span"`. ### Frontend seam to leave now The request builder in Phase 2 takes `focus` and an arbitrary `specs` list without reshaping, -and the page grid can accept two more chart cards. 5a and 5b add queries and cards without -touching the four existing charts. +and the page grid can accept more chart cards. 5a and 5b add queries and cards without touching +the Phase 4 charts. -Recommendation for the boundary: ship Phase 1 to 4 as a four-chart grid. Do not mount empty -Tools and Models cards in this release; a visible "coming soon" card ages badly. Land the -chart-card component as a reusable shell so Phase 5 only supplies data and series config. +Recommendation for the boundary: ship Phase 1 to 4. Do not mount empty Tools and Models cards +in this release; a visible "coming soon" card ages badly. Land the chart-card component as a +reusable shell so Phase 5 only supplies data and series config. ## Testing and verification -- Follow `docs/designs/testing/README.md`. Add unit tests for the new mapper (pure - function: buckets in, dashboard shape out) and for the health-score computation. These need - no live database. -- Verify the four charts against one live project by running the local stack per the root - `AGENTS.md` local dev loop. +- Follow `docs/designs/testing/README.md`. Add unit tests for the new mapper (pure function: + buckets in, dashboard shape out), for the success/failed split, for the coverage gate (a + metric below threshold is suppressed, not shown as zero), and for the health-score + computation. These need no live database. +- Verify the charts against one live project by running the local stack per the root + `AGENTS.md` local dev loop. Confirm the `trace_type` and `status_code` filters narrow rather + than widen, and confirm the p95 nested read. - Run `pnpm lint-fix` in `web` before committing. Do not commit during the planning phase. ## Risks and unknowns to resolve during the build -- Live-response validation of the metric shape: the nested `pcts.p95` reads correctly and the - prompt/completion split is non-zero on real traffic (resolve in Phase 2). The `type` strings - and the `pcts.p95` nesting are already verified in the backend code. +- **Cost and token-split coverage** (Phase 0). Whether either has usable coverage on the target + data, and the cause of the mid-July collapse. The cost tile and the token split chart depend + on it. Both measured stacks were local dev; production is unverified. +- **Silent-failure validation** (Phase 2). The exact `trace_type` and `status_code` enum + literals that narrow rather than returning an empty 200, and the nested `pcts.p95` read. - The correct agents-list atom for the filter options (resolve in Phase 1). - Whether the multi-agent reference filter uses `or` grouping or a single `in` with multiple values in this dialect (resolve in Phase 2 against the existing filter builder). -- Whether the runner marks a failed run's root span `status_code = ERROR`. The failed-run - count and the health score depend on it; validate on live traffic in Phase 2. +- Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The + failed-run count and the health score depend on it; validate on live traffic in Phase 2. The + root-status definition also misses in-run failures that recovered a clean root (~1.2% of runs + on the measured data); state the definition in a tooltip. - The run-count floor for the neutral health state (tune in Phase 4; start around 20). +- Performance at scale: the 30-day window is the shape most at risk of crossing the 15-second + statement timeout as a project grows. Keep 7 days the default and treat longer windows as an + explicit choice with a loading state. diff --git a/docs/design/agent-analytics/research.md b/docs/design/agent-analytics/research.md index 17efe6a2c5..1ab6e41d1d 100644 --- a/docs/design/agent-analytics/research.md +++ b/docs/design/agent-analytics/research.md @@ -40,13 +40,20 @@ spine and reads a few response fields the current mapper drops. Add a small nested accessor (or extend `metricField` with a two-key form) for the p95 read; do not assume p95 is a flat sibling of `sum`. - What it does not read, and this page needs: - - `costs.cumulative.prompt` and `costs.cumulative.completion` for the Costs split. - - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split. + - `gen_ai.usage.cost` (field `sum`, plus `count` for the coverage gate) for a coverage-gated + total-cost tile. The canonical `costs.cumulative.*` paths hold no data on agent root spans, + so there is no prompt/completion cost split to read; see data-contract.md. + - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split, which + is coverage-gated because it shares the cost field's mid-July coverage collapse. - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for the Latency tooltip and marker. + - Category `freq` arrays on `ag.data.parameters.agent.harness.kind`, + `ag.data.parameters.agent.llm.model`, and the agent `references` paths for the breakdown + charts. - This page does not reuse the existing mapper's `errors.cumulative`-based failure count. A - failed run is a run whose root span status is `ERROR`, which a metric spec cannot - read, so the page gets the failed-run count from a separate status-filtered query. See + failed run is a run whose root span status is `STATUS_CODE_ERROR` (there is no + `STATUS_CODE_OK` on root spans; success is the complement), which a metric spec cannot read, + so the page gets the failed-run count from a separate status-filtered query. See data-contract.md. - `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar count reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for @@ -58,8 +65,10 @@ spine and reads a few response fields the current mapper drops. `references in [{id: appId}]` condition when an app id is present), computes the interval, and calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new page's fetch function. For project scope, omit the single-app reference condition and add - reference conditions only for the agents the user selects in the filter. The new page also - issues a second query per window with a `status_code = ERROR` filter to count failed runs. + reference conditions only for the agents the user selects in the filter, plus the base + `trace_type is invocation` condition on every query. The new page also issues a second query + per window with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` filter + to count failed runs. ## The dashboard state atoms (reuse pattern, new atoms for this page) diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index 6c41a4071a..e2170946f2 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -9,8 +9,12 @@ The workspace holds the plan; implementation has not started and no branch exist ## Locked decisions -1. Frontend-first scope: four charts (Runs, Latency, Costs, Tokens), four stat tiles, health - donut, Agents filter, time-range control. No `api/` change in this plan. +1. Frontend-first scope with two backend prerequisites (Phase 0 in plan.md). Charts: Runs, + Latency, Tokens (coverage-gated split), and runs-per-harness / -configured-model / -agent + breakdowns. No Costs prompt/completion chart; cost is a coverage-gated total tile from + `gen_ai.usage.cost`. Stat tiles, health donut, Agents / Harness / configured-Model filters, + time-range control. The two Phase 0 backend items: make a killed or rejected query + distinguishable from an empty one, and investigate the cost / token-split coverage collapse. 2. New page at project scope named Analytics; the default query aggregates all project agents, and the Agents filter narrows the set. 3. An **agent** is an application/workflow artifact. The Agents multi-select is the only @@ -18,9 +22,9 @@ The workspace holds the plan; implementation has not started and no branch exist 4. The count of agent invocations is called a **run**. The Observability dashboard calls the same metric a "request"; the two are allowed to diverge until Observability is aligned in a later, separate change. -5. A **failed run** is a run whose root span `status_code` is `ERROR` (a run-level outcome, - not a count of errored steps). It comes from a second, status-filtered analytics query, not - a metric spec. +5. A **failed run** is a run whose root span `status_code` is `STATUS_CODE_ERROR` (a run-level + outcome, not a count of errored steps; there is no `STATUS_CODE_OK`, so success is the + complement). It comes from a second, status-filtered analytics query, not a metric spec. 6. The **health score** is the success rate: `round(100 x successRate)`, banded Healthy 85+, Watch 65 to 84, At risk below 65. Latency was dropped from it. Below a run-count floor the donut shows a neutral "Not enough runs yet" state instead of a band. @@ -34,9 +38,11 @@ The workspace holds the plan; implementation has not started and no branch exist The data path from the analytics endpoint to a mapped dashboard shape already exists for the Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability dashboard atoms). This feature reuses that spine. The data-layer work is: pass explicit metric specs for -the prompt/completion split, read the duration min/max/p95 the current mapper drops, and add -the status-filtered failed-run query. The endpoint returns all of these, so the frontend-first -scope needs no backend change. +the total cost (`gen_ai.usage.cost`) and the token split, read the duration min/max/p95 the +current mapper drops, read the category `freq` breakdowns, and add the status-filtered +failed-run query. The endpoint returns the latency, run-count, and breakdown fields directly; +cost and the token split are coverage-gated because their populated paths collapsed in mid-July +(Phase 0). This is why the scope carries two backend prerequisites rather than none. ## Resolved by code verification @@ -48,19 +54,23 @@ scope needs no backend change. `specs?: string` and forwards it, so passing specs is a small entities-layer change. - **Failed-run mechanism**: `status_code` is a table column and metric specs read only the `attributes` JSON (`build_extract_cte`), so a spec cannot target it. Failed runs come from a - second query with a `status_code = ERROR` filter (`status_code` is a first-class filter - field). Four analytics calls total: unfiltered and status-filtered, for the current and the - previous window. + second query with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` + filter (`status_code` is a first-class filter field; the operator must be `is` and the value + the full enum literal, or the backend returns an empty 200). Per window the page issues a + bucketed unfiltered query, a bucketed status-filtered query, and a no-interval unfiltered + query for exact tile numbers including window-level p95; the previous window needs only the + no-interval unfiltered and status-filtered calls. See data-contract.md. ## Open questions to resolve during the build - Agents-list atom for the filter options (Phase 1). - Multi-agent reference filter encoding: a single `in` with all ids, or one condition per agent combined with `or` (Phase 2, against the existing filter builder). -- Whether the runner marks a failed run's root span `status_code = ERROR`. The failed-run - count and health score depend on it; validate on live traffic (Phase 2). -- Live-response validation: the nested `pcts.p95` reads correctly and the prompt/completion - split is non-zero on real traffic (Phase 2). +- Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The + failed-run count and health score depend on it; validate on live traffic (Phase 2). +- Live-response validation: the `trace_type` and `status_code` filters narrow rather than + returning an empty 200, the nested `pcts.p95` reads correctly, and the cost / token-split + coverage on real traffic (Phase 2; may be near zero — the Phase 0 investigation). - The run-count floor for the neutral health state (Phase 4; start around 20). ## Deferred to a later backend change From 244b8ca1b76c02023cddcd54969b299a3b64832a Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Mon, 3 Aug 2026 16:29:26 +0600 Subject: [PATCH 6/9] Refine agent analytics documentation and scope for v1 implementation - Updated the plan.md to clarify the cost chart and token split coverage collapse investigation. - Enhanced research.md to detail the analytics fetch layer and response-to-dashboard mapper. - Introduced scope.md to delineate features for v1 and v2, including backend prerequisites. - Revised status.md to reflect current planning status and locked decisions, emphasizing frontend-first scope and backend dependencies. - Improved clarity and consistency across documentation, ensuring alignment with implementation goals. --- docs/design/agent-analytics/README.md | 97 ++++++------- docs/design/agent-analytics/context.md | 145 +++++++++---------- docs/design/agent-analytics/data-contract.md | 111 +++++--------- docs/design/agent-analytics/plan.md | 112 +++++++------- docs/design/agent-analytics/research.md | 135 +++++++++-------- docs/design/agent-analytics/scope.md | 93 ++++++++++++ docs/design/agent-analytics/status.md | 111 +++++++------- 7 files changed, 414 insertions(+), 390 deletions(-) create mode 100644 docs/design/agent-analytics/scope.md diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md index aa2ede2966..694b921ee9 100644 --- a/docs/design/agent-analytics/README.md +++ b/docs/design/agent-analytics/README.md @@ -1,64 +1,59 @@ # Agent Analytics page -A new project-scoped **Analytics** page for the Agenta web app. It charts how the -project's agents perform over a time window: run volume, success and failure, latency, -cost, and token usage. The page reads from the existing spec-driven analytics endpoint -(`POST /spans/analytics/query`) and reuses the frontend data-layer atoms that already exist -for it, so this work adds a page, not a new data layer. +A new project-scoped **Analytics** page for the Agenta web app. It charts how a project's +agents perform over a chosen window: run volume, success and failure, latency, cost, and +tokens. The page reads the existing spec-driven analytics endpoint +(`POST /spans/analytics/query`) and reuses the frontend data-layer atoms already built for it, +so this work adds a page, not a data layer. ## Reading order -1. **context.md** : why this page exists, what a user sees today, goals and non-goals, and - the scope decisions that are locked. -2. **research.md** : the parts of the codebase this feature reuses, with exact file paths: - the analytics fetch layer, the response-to-dashboard mapper, the sidebar and routing, - and the charting library. Read this before proposing any new file. -3. **data-contract.md** : the request the page sends (time window, filter, metric specs) - and the response fields it reads, including the fields the current mapper drops that - this page needs. -4. **plan.md** : the build broken into phases, with the file list per phase and the clean - boundary where the deferred model and tool views drop in once the backend supports them. -5. **status.md** : current state, open questions, and decisions. This is the source of - truth for progress; update it as work lands. -6. **capability-review.md** : an independent, evidence-backed review of what the analytics - backend can actually answer today, written after the four documents above. It tests each - wanted capability against the running code with live queries, measures how long those - queries take, and proposes a v1 and a v2. It contradicts the four documents above in - several places, so read it before building anything. Where the two disagree, the review - carries the evidence. +1. **context.md** — why the page exists, what a user sees today, goals and non-goals, and the + locked scope decisions. +2. **scope.md** — the v1 / v2 split: what ships now versus what waits for backend work, each as + a ranked table. Read it to know what is in scope before the how. +3. **research.md** — the code this feature reuses, path by path: the analytics fetch layer, the + response-to-dashboard mapper, the sidebar and routing, and the charting library. Read it + before you propose any new file. +4. **data-contract.md** — the request the page sends (window, filter, metric specs) and the + response fields the new mapper reads, including the fields the current mapper drops. +5. **plan.md** — the build in phases, with the file list per phase and the seam where the + deferred model and tool views drop in once the backend supports them. +6. **status.md** — current state, open questions, and decisions. This is the source of truth + for progress; update it as work lands. +7. **capability-review.md** — an independent, evidence-backed review of what the analytics + backend can answer today, written after the documents above. It tests each wanted capability + against the running code with live queries, times those queries, and proposes a v1 and a v2. + It contradicts the earlier documents in several places, so read it before you build. Where + the two disagree, the review carries the evidence. ## Glossary -Terms used across these documents, defined once here. This section is the workspace glossary; -a separate `CONTEXT.md` cannot live here because the filesystem is case-insensitive and would -collide with `context.md`. +Terms used across these documents, defined once. This section is the workspace glossary; a +separate `CONTEXT.md` cannot sit beside `context.md`, because the filesystem is case-insensitive +and the two names collide. -- **Agent**: a configured AI agent in the project, the top-level thing a user builds, runs, - and analyzes. It is the unit this page aggregates over and the unit the Agents filter - narrows to. Not called application, app, workflow, or variant in this page's copy. -- **Run**: one agent invocation. On the backend it is one root span (a span with no - parent). Run count per time bucket equals the count of the `ag.type.trace` metric. This - page says "run"; the Observability dashboard says "request" for the same metric, and the two - are allowed to diverge until Observability is aligned later. Not called request here. -- **Span**: one unit of work inside a run (a model call, a tool call, or the agent step - itself). Model name and tool name live on child spans, not on the root span. -- **Root span**: the top span of a run. Today's analytics endpoint only reads root spans. +- **Agent**: a configured AI agent in the project — the top-level thing a user builds, runs, and + analyzes. It is the unit the page aggregates over and the unit the Agents filter narrows to. + This page's copy never calls it application, app, workflow, or variant. +- **Run**: one agent invocation. On the backend it is one root span (a span with no parent). Run + count per bucket equals the count of the `ag.type.trace` metric. This page says "run"; the + Observability dashboard says "request" for the same metric, and the two may diverge until + Observability is aligned later. +- **Span**: one unit of work inside a run — a model call, a tool call, or the agent step itself. + Model name and tool name live on child spans, not on the root span. +- **Root span**: the top span of a run. Today's analytics endpoint reads root spans only. - **Failed run**: a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome - (there is no `STATUS_CODE_OK` on root spans; success is the complement). Counted from the - `status_code` column, not from the errors metric. + (there is no `STATUS_CODE_OK` on root spans, so success is the complement). It is counted from + the `status_code` column, not from the errors metric. - **Error**: any errored step inside a run. A run can contain errors and still succeed, so an error count is not a failed-run count. -- **Success rate**: successful runs over total runs, where a failed run is the one above. -- **Bucket**: one time slice of the chart x-axis (for example one day, or one hour). The - endpoint returns one metrics object per bucket. +- **Bucket**: one time slice of the chart x-axis — one day, or one hour. The endpoint returns + one metrics object per bucket. - **Metric spec**: a request instruction of the form `{type, path}` that tells the endpoint - which JSON path on the span to summarize and how. The endpoint has no fixed metric list; - it summarizes whatever path a spec names. -- **Focus**: a request field selecting whether the query aggregates over root spans - (`trace`) or all spans (`span`). Today only `trace` works; see context.md. -- **Project scope**: the web app always has exactly one project in context. This page lives - at that level and, by default, aggregates every agent in the project. -- **Health score**: a single 0 to 100 number this page computes in the browser from the - success rate alone (latency was dropped because a fixed latency band mislabels slow-but- - healthy agents). The bands map directly to success: Healthy at 85 and above, Watch from 65 - to 84, At risk below 65. It is a display aid, not a backend metric. + which JSON path on the span to summarize, and how. The endpoint has no fixed metric list; it + summarizes whatever path a spec names. +- **Focus**: a request field that selects whether the query aggregates over root spans (`trace`) + or all spans (`span`). Only `trace` works today; see context.md. +- **Project scope**: the web app always has exactly one project in context. This page lives at + that level and, by default, aggregates every agent in the project. diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md index fa79e0d1cf..54fb8cb46e 100644 --- a/docs/design/agent-analytics/context.md +++ b/docs/design/agent-analytics/context.md @@ -2,116 +2,105 @@ ## What a user sees today -Today the web app has no page that shows agent performance across a project. A team running -several agents cannot open one screen and read how many runs happened, how many failed, how -fast they ran, and what they cost, narrowed to the agents they care about, over a window they -choose. +Today the web app shows no agent performance across a project. A team running several agents +cannot open one screen and read how many runs happened, how many failed, how fast they ran, and +what they cost — narrowed to the agents they care about, over a window they choose. -The frontend already has the data plumbing to answer these questions. The atoms, the fetch -function, and the response mapper that talk to the analytics endpoint exist and are in use. +The frontend already carries the plumbing to answer those questions. The atoms, the fetch +function, and the response mapper that talk to the analytics endpoint all exist and are in use. This feature reuses that plumbing; it does not rebuild the data layer. ## What this feature adds -A new page titled Analytics, reachable from the project sidebar, scoped to the whole project. +A new page named Analytics, reachable from the project sidebar and scoped to the whole project. It shows: -- A header with the page title, a one-line description, a time-range control that accepts any - window, and a Filters popover with an Agents multi-select that narrows the query to chosen - agents. The time-range control reuses the existing observability time windowing (the `Sort` - control and its `SortResult`), so it supports the standard presets and a custom - start-and-end range. It is not a fixed list of options. It opens on the last 7 days. -- A summary panel: a health donut (0 to 100, with a Healthy / Watch / At risk band and a - one-line read-out) and stat tiles (Total runs, Avg latency, p95 latency, Total tokens, and a - coverage-gated Total cost that shows an explicit "not available for this window" instead of a - zero when cost coverage is low). Each tile shows a change badge against the previous window of - equal length and a small trend line of the current window. +- A header: the page title, a one-line description, a time-range control that accepts any + window, and a Filters popover whose Agents multi-select narrows the query to chosen agents. + The time-range control reuses the observability windowing (the `Sort` control and its + `SortResult`), so it takes the standard presets and a custom start-and-end range, not a fixed + list of options. It opens on the last 7 days. - Charts in a grid, each with hover tooltips and a legend whose entries toggle series on and off: - Runs: stacked bars of successful and failed runs per bucket. - - Latency: bars of average latency per bucket, with a p95 marker line; the tooltip shows + - Latency: bars of average latency per bucket, with a per-bucket p95 line; the tooltip shows average, p95, min, and max. - - Tokens: total tokens per bucket, with a coverage label; the prompt/completion split renders + - Cost: total cost per bucket, coverage-gated. When cost coverage is low it reads "not + available for this window" instead of a zero. + - Tokens: total tokens per bucket, with a coverage label. The prompt/completion split renders as stacked bars only when its coverage gate passes. - Runs per harness, runs per configured model, and runs per agent: category breakdowns, one `categorical/single` spec each. - There is no Costs prompt/completion chart: those cost paths hold no data on agent runs, and - the one working cost path (`gen_ai.usage.cost`) is a coverage-gated total shown as the tile - above. See data-contract.md and capability-review.md. + There is no Costs prompt/completion split chart. The split paths hold no data on agent runs, + so the Cost chart shows the total from `gen_ai.usage.cost` only. See data-contract.md and + capability-review.md. ## Locked scope decisions These decisions are settled and drive the plan. Do not reopen them without the requester. -1. Frontend-first, with two backend prerequisites. Build the charts above, the stat tiles, the - health donut, the Agents / Harness / configured-Model filters, and the time-range control - against today's endpoint. The endpoint returns most fields these need directly, but two - backend items gate a trustworthy release and are tracked as Phase 0 in plan.md: making a - killed or rejected query distinguishable from a genuinely empty one, and investigating the - mid-July collapse in cost and token-split coverage. The cost tile and the token split stay - coverage-gated until the second is understood. - -2. Health donut computed in the browser. The health score is the success rate: - `round(100 x successRate)`, banded Healthy at 85 and above, Watch from 65 to 84, At risk - below 65. Latency does not factor in, because a fixed latency band mislabels agents that - are legitimately slow. Below a minimum run count the donut shows a neutral "Not enough runs - yet" state instead of a band. This is a display aid; it is not sent to or stored on the - backend. - -3. New page at project scope. This is a net-new page named Analytics. It aggregates every - agent in the project by default. The Agents multi-select narrows the set, so the default - query carries no single-app reference filter. - -4. A failed run means a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome - (there is no `STATUS_CODE_OK` on root spans; success is the complement). Success rate and the - health score build on that, not on a count of errored steps. data-contract.md has the - definition and the query it needs. - -## Showing model usage and tool usage needs backend work - -Two views are out of scope for this plan because the backend cannot serve them yet. This is an +1. Frontend-first, with two backend prerequisites. Build the charts above, the Agents / Harness / + configured-Model filters, and the time-range control against today's endpoint. The endpoint + returns most of the fields these need directly, but two backend items gate a trustworthy + release, tracked as Phase 0 in plan.md: make a killed or rejected query distinguishable from a + genuinely empty one, and investigate the mid-July collapse in cost and token-split coverage. + The Cost chart and the token split stay coverage-gated until someone explains that collapse. + +2. A net-new page at project scope, named Analytics. It aggregates every agent in the project by + default. The Agents multi-select narrows the set, so the default query carries no single-app + reference filter. + +3. A failed run is a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome + (there is no `STATUS_CODE_OK` on root spans, so success is the complement). The Runs chart's + successful-and-failed split builds on that, not on a count of errored steps. data-contract.md + holds the definition and the query it needs. + +## Model usage and tool usage need backend work + +Two views fall out of scope for this plan, because the backend cannot serve them yet. This is an engineering constraint, not an open design question. Tool usage (calls per tool, tool error rate) and model usage (runs per model, cost per model) read fields that live on child spans: the tool name, the model name, the span type, and the -per-span status. Today's analytics endpoint reads only root spans. It accepts a `focus` field -that would widen the scan to all spans, but the field is inert. So these fields are simply not -reachable through the endpoint today. +per-span status. Today's analytics endpoint reads root spans only. It accepts a `focus` field +that would widen the scan to all spans, but the field is inert, so those fields stay out of +reach. -The two backend prerequisites are not co-equal gates; they unlock different things. +The two backend prerequisites are not co-equal gates; each unlocks a different thing. -- Span-focus wiring unlocks both the tool view and the model-share view in their basic form. - `query.focus` is in scope in `dao.analytics()` but never reaches `build_base_cte`, which +- **Span-focus wiring** unlocks both the tool view and the model-share view in their basic form. + `query.focus` reaches `dao.analytics()` but never reaches `build_base_cte`, which unconditionally applies `WHERE parent_id IS NULL` - (`api/oss/src/dbs/postgres/tracing/utils.py`). Threading `focus` through so `focus = span` - drops that predicate lets the query read child spans. A `categorical/single` spec already - returns per-value frequencies, so counting calls per tool or runs per model needs no - group-by once span focus works. - - Correctness caveat, must ship with the fix: under `focus = span` the cumulative metric - paths double-count. `ag.metrics.*.cumulative` are rollups stored on the root span, so - scanning all spans sums the root total plus every child's. Cost and tokens must switch to - the `incremental` paths under span focus, or the figures come out inflated with no error. - The fix needs a guard that rejects or auto-maps a cumulative spec under `focus = span`. -- Group-by dimension unlocks only per-model cost and tokens. Crossing a numeric metric (cost, - tokens) with a categorical dimension (model name) is the one thing the frequency reducers - cannot do; the extract stage groups by `(timestamp, spec)` only, with no grouping key. This - is the harder, later change. Preferred shape: a per-query `group_by` dimension that splits - every spec by one path, which leaves `MetricSpec` untouched and confines the nested + (`api/oss/src/dbs/postgres/tracing/utils.py`). Thread `focus` through so `focus = span` drops + that predicate, and the query reads child spans. A `categorical/single` spec already returns + per-value frequencies, so counting calls per tool or runs per model needs no group-by once + span focus works. + - One caveat must ship with the fix: under `focus = span`, the cumulative metric paths + double-count. `ag.metrics.*.cumulative` are rollups stored on the root span, so scanning all + spans sums the root total plus every child's. Cost and tokens must switch to the + `incremental` paths under span focus, or the figures inflate with no error. The fix needs a + guard that rejects or auto-maps a cumulative spec under `focus = span`. +- **A group-by dimension** unlocks only per-model cost and tokens. Crossing a numeric metric + (cost, tokens) with a categorical dimension (model name) is the one thing the frequency + reducers cannot do; the extract stage groups by `(timestamp, spec)` only, with no grouping + key. This is the harder, later change. Preferred shape: a per-query `group_by` dimension that + splits every spec by one path, leaves `MetricSpec` untouched, and confines the nested `{group_value: stats}` output to one code path. ## Goals -- Reuse the existing analytics fetch layer, response mapper, and time-windowing control rather - than adding a second path to the same endpoint. -- Deliver the layout and interactions using the repo's charting library and theme tokens, in - both light and dark themes. -- Leave a clean boundary so the deferred model and tool views become an additive change once - the backend prerequisites land, not a rewrite. +- Reuse the analytics fetch layer, the response mapper, and the time-windowing control rather + than add a second path to the same endpoint. +- Deliver the layout and interactions with the repo's charting library and theme tokens, in both + light and dark themes. +- Leave a clean seam, so the deferred model and tool views land as an additive change once the + backend prerequisites ship — not a rewrite. ## Non-goals -- No change to `api/` in this plan. +- No new analytics capability under `api/` for the page itself. The two Phase 0 items harden + existing behavior; they do not add a chart's data path. - No change to the existing Observability page. -- No real-time streaming; the page fetches per time-range selection and caches like the - existing dashboard atoms. +- No real-time streaming. The page fetches per time-range selection and caches like the existing + dashboard atoms. diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md index 5876cfe4bb..f57aa7b49d 100644 --- a/docs/design/agent-analytics/data-contract.md +++ b/docs/design/agent-analytics/data-contract.md @@ -5,14 +5,14 @@ This page talks to one endpoint, `POST /spans/analytics/query`, through the exis page builds and the response fields the new mapper reads, and it flags the one frontend extension: passing explicit metric specs. -Every request and response shape below is taken from the capability review -(`capability-review.md`), which verified each one against live calls on two stacks. Where a -value still has to be confirmed against a live response during the build, the text says so. +Every request and response shape below comes from the capability review +(`capability-review.md`), which verified each against live calls on two stacks. Where a value +still needs confirming against a live response during the build, the text says so. ## Request -The request carries four kinds of fields. Classified by what each field is, not by which -chart it feeds: +The request carries four kinds of field, grouped by what each field is, not by which chart it +feeds: - **Routing** (which tenant and window): `projectId`, `oldest`, `newest`. `projectId` comes from `projectIdAtom`. `oldest` and `newest` are ISO bounds taken from the selected @@ -24,8 +24,7 @@ chart it feeds: - **Policy** (how to slice and aggregate): `focus = "trace"` and `interval` (bucket size in minutes) from `calculateIntervalFromDuration`. The endpoint ignores `focus` entirely and always reads root spans, so `focus = "trace"` is the honest label rather than a switch; - never rely on the echoed `focus` to confirm behaviour. Omit `interval` to collapse the - window into one exact bucket (used for the window-level summary numbers below). + never rely on the echoed `focus` to confirm behaviour. - **Data selection** (what to measure): `specs`, a list of metric specs naming the JSON paths to summarize. See below. - **Data filter** (which spans qualify): `filter`, a `{conditions: [...]}` object. Every @@ -33,12 +32,14 @@ chart it feeds: so annotation traces (evaluator and human-annotation runs) do not inflate the run count and the latency, cost, and token numbers. At project scope with no agent selected, add nothing else so the query spans the whole project. For selected agents, add `references` conditions - on the chosen agent ids. Two things to validate against the existing filter builder in - Phase 2: the exact enum literal for `trace_type` (mirror the `status_code` lesson below — - the accepted value may be prefixed), and whether multiple agent ids encode as a single - `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per agent combined with - `or`. A fallback for the run count that needs no `trace_type` literal: read the `freq` array - of the `attributes.ag.type.trace` categorical spec, which already carries the + on the chosen agent ids. The `trace_type` value is the unprefixed literal `invocation` + (verified against the `TraceType` enum, `api/oss/src/core/otel/dtos.py`); unlike + `status_code`, it is not prefixed, and the backend validates the value against the enum, so a + wrong literal returns an empty 200. The one encoding still to confirm against the filter + builder in Phase 2 is whether multiple agent ids go as a single + `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per agent combined with `or`. + A fallback for the run count that avoids the `trace_type` filter entirely: read the `freq` + array of the `attributes.ag.type.trace` categorical spec, which already carries the invocation-versus-annotation split. A filter typo does not error. An unknown field is logged and dropped, which **widens** the @@ -52,8 +53,10 @@ chart it feeds: defaults give totals for cost, tokens, duration, errors, and the trace and span type counts, but they read the canonical `ag.metrics.costs.cumulative.*` cost paths, which hold no data on agent runs (see the cost note below). The page must pass an explicit `specs` list. Add an -optional `specs` field to `SpansAnalyticsParams` and serialize it to a JSON-string query param -exactly as `filter` is serialized. +optional `specs` field to `SpansAnalyticsParams` and serialize it exactly as `filter` is +serialized. The Fern request type (`QuerySpansAnalyticsRequest`) already declares `specs?: string`, +and `fetchSpansAnalytics` already does `request.filter = JSON.stringify(filter)`, so the change is +`request.specs = JSON.stringify(specs)` on one line beside it — no client regeneration needed. The specs the page sends, by path and type: @@ -84,16 +87,15 @@ Notes on the specs: - **p95 is nested, not a flat field.** The numeric/continuous reducer emits percentiles under a `pcts` object, so the value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and `max` are flat siblings and read directly; only the percentiles sit one level down. All 27 - percentile levels ship on every numeric/continuous spec, so min/max/p95 latency need no + percentile levels ship on every numeric/continuous spec, so the per-bucket p95 line needs no backend work. Confirm the nested shape against one live response in Phase 2. - **Cost has no prompt/completion split, and its one working path is coverage-gated.** The canonical `ag.metrics.costs.cumulative.total`, `.prompt`, and `.completion` paths hold no data on agent root spans (the cost roll-up never crosses the run's OTLP batch boundary to reach the root span). The only populated cost path is `attributes.gen_ai.usage.cost`, the harness's own reported run total, and its coverage collapsed to near zero in mid-July on - both measured stacks for a cause nobody has established yet. The cost tile therefore renders - only when coverage clears a threshold (see "Coverage gating" below), and there is no cost - split to chart. Do not add a Costs prompt/completion chart. + both measured stacks for a cause nobody has established yet. The Cost chart therefore renders + the total only, and only when coverage clears a threshold (see "Coverage gating" below). - **Total tokens works with a coverage label; the split moves with cost.** The prompt/completion token split shares the same mid-July collapse as cost: it is real where the pipeline works and a flat zero band where it does not, which reads as data and is worse @@ -126,33 +128,17 @@ a failed run is "the root span errored"; say so in a tooltip. Catching in-run fa ### The queries per window -Two shapes are needed because window-level percentiles cannot be composed from per-bucket -percentiles (averaging or maxing per-bucket p95s is wrong for any non-uniform distribution). -Counts and sums do compose, so they can be summed from the bucketed calls; only p95 forces the -no-interval call. +The page issues two bucketed queries for the selected window, both with `interval` set. Every +chart reads its data per bucket, so these two calls cover the whole page. -For the **current** window: - -1. **Bucketed, unfiltered** (`interval` set): drives the Runs, Latency, Cost, and Tokens - charts. Also carries the per-bucket failed count once combined with query 2. +1. **Bucketed, unfiltered**: drives the Runs, Latency, Cost, Tokens, and category-breakdown + charts. 2. **Bucketed, status-filtered**: the per-bucket failed run count for the stacked Runs chart. -3. **No-interval, unfiltered** (`interval` omitted, one exact bucket): the exact summary-tile - numbers, including the window-level p95 latency that the bucketed call cannot produce. - -For the **previous** comparison window (change badges only): - -4. **No-interval, unfiltered**: exact previous-window totals and p95. -5. **No-interval, status-filtered**: the previous-window failed count for the previous success - rate. - -Derive the previous window as the equal-length window immediately before the selected one: for -`[oldest, newest]`, the previous window is `[oldest - (newest - oldest), oldest]`. This works -for both preset and custom ranges. -Keep any single request at or under about eight specs, and fan the calls out in parallel with a -per-call error state. On the measured data a 7-day window costs about 0.26 s and a 30-day -window about 1.7 s for the shipped six-spec shape, so 7 days is the default and 30 or 90 days -is an explicit user choice with a loading state. +Keep any single request at or under about eight specs, and fan the two calls out in parallel with +a per-call error state. On the measured data a 7-day window costs about 0.26 s and a 30-day +window about 1.7 s for the six-spec shape, so 7 days is the default and 30 or 90 days is an +explicit user choice with a loading state. ## Response fields the mapper reads @@ -165,32 +151,25 @@ new mapper produces, per bucket: flooring is needed. This departs from the existing observability mapper, which subtracts the `errors.cumulative` sum. That sum counts errored steps, not failed runs, and can exceed the run count, so this page uses the run-level status instead. -- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. For a window average, sum - the duration `sum` fields and divide by the summed `count` fields, or read `mean` from the - no-interval call. Never average per-bucket means. +- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. - `latencyMin`, `latencyMax` = the flat `min` / `max` duration fields. -- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95`. `metricField`'s flat - one-level read does not reach it, so the mapper needs a small `pcts` accessor. The - window-level p95 tile reads this from the no-interval call, not from the buckets. +- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95`, per bucket. `metricField`'s + flat one-level read does not reach it, so the mapper needs a small `pcts` accessor. This drives + the per-bucket p95 line and the tooltip. - `costTotal` = the `gen_ai.usage.cost` `sum`. Rendered only when its coverage gate passes. - `tokensTotal` = the `tokens.cumulative.total` `sum`. - `tokensPrompt`, `tokensCompletion` = the two token split sums. Rendered only when their coverage gate passes; below it they are suppressed, not shown as zero. -And window totals for the summary tiles: total runs, success rate, average latency, p95 -latency, total tokens, and coverage-gated total cost, plus the same figures for the previous -window so the change badges have a baseline. Lower latency and lower cost are the "good" -direction for the badge colour. - ### Coverage gating Cost and the token split can be perfectly expressible and still hold no data, because coverage is zero on the window. For each gated metric, compare its spec `count` against the run count -for the same window. Render the tile or chart only when coverage clears a threshold; below it, -show "cost data is not available for this window" (or the token equivalent) rather than a zero. -A zero reads as a real measurement and is worse than an explicit unavailable state. Total -tokens is shippable with a coverage label rather than a hard gate, because its coverage is -partial rather than collapsed. +for the same window. Render the chart only when coverage clears a threshold; below it, show +"cost data is not available for this window" (or the token equivalent) rather than a zero. A +zero reads as a real measurement and is worse than an explicit unavailable state. Total tokens +is shippable with a coverage label rather than a hard gate, because its coverage is partial +rather than collapsed. ### Response quirks the mapper must handle @@ -207,20 +186,6 @@ partial rather than collapsed. Calendar months are not expressible at all. Label the axis as 24-hour periods; do not offer a month period. -## Health score (browser-side, not in the contract) - -Computed from the window totals, never sent to the backend: - -- `successRate` = successful runs / total runs. -- `health` = round(100 x successRate). The score is the success rate; latency does not factor - in, because a fixed latency band mislabels agents that are legitimately slow. -- Band: Healthy at 85 and above, Watch from 65 to 84, At risk below 65. The bands read - directly as success percentages. -- Low-traffic guard: below a minimum run count in the window, do not band the score. Show a - neutral "Not enough runs yet" donut instead, so a single failure in a quiet window does not - read as At risk. The threshold is a build-time tuning value (start around 20 runs). The stat - tiles and the charts still render whatever data exists. - ## Boundary for the deferred charts The deferred Tools and resolved-Models charts need `focus = "span"`, which is accepted, echoed, diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md index 56ebe55ade..888e7abca9 100644 --- a/docs/design/agent-analytics/plan.md +++ b/docs/design/agent-analytics/plan.md @@ -23,7 +23,7 @@ the page ships behind its flag. - **Investigate the cost and token-split coverage collapse.** `attributes.gen_ai.usage.cost` and the `tokens.cumulative.prompt`/`.completion` split were populated on 70–95% of runs in early July on both measured stacks and fell to roughly zero within a week, cause unknown. - Until this is understood, the cost tile has nothing dependable to show. File it; the first + Until this is understood, the Cost chart has nothing dependable to show. File it; the first four diagnostic checks are in capability-review.md section 4.4 item 4. Both measured stacks were local dev; whether production shows the same collapse is unverified. @@ -50,40 +50,39 @@ correct. ## Phase 2: data layer -Goal: the page can fetch mapped analytics for the current and previous windows. +Goal: the page can fetch mapped analytics for the selected window. - Extend `SpansAnalyticsParams` in `web/packages/agenta-entities/src/trace/api/api.ts` with an - optional `specs` field, serialized to a JSON-string query param like `filter`. Verify the - entities package still builds (`pnpm turbo run build --filter=@agenta/entities`). + optional `specs` field, and in `fetchSpansAnalytics` add `request.specs = JSON.stringify(specs)` + on one line beside the existing `filter` serialization. The Fern `QuerySpansAnalyticsRequest` + already declares `specs?: string`, so no client regeneration is needed. Verify the entities + package still builds (`pnpm turbo run build --filter=@agenta/entities`). - Add a new mapper next to the existing one in `web/oss/src/services/tracing/lib/` (do not - change `analyticsToGeneration`; the observability page depends on it). The new mapper reads - the duration min, max, sum, count, and nested `pcts.p95` fields, the total cost and total - token sums with their `count` for the coverage gate, the coverage-gated prompt/completion - token split, and the category `freq` arrays for the harness, configured-model, and agent - breakdowns. It combines the unfiltered and status-filtered run counts into success and - failed, and returns the per-bucket and window-total shape in data-contract.md. Reuse - `metricField` and `calculateIntervalFromDuration`, and add a small `pcts` accessor for p95 - and a `freq` reader for the breakdowns. + change `analyticsToGeneration`; the observability page depends on it). Per bucket, the new + mapper reads the duration min, max, sum, count, and nested `pcts.p95`; the total cost and + total token sums with their `count` for the coverage gate; the coverage-gated + prompt/completion token split; and the category `freq` arrays for the harness, + configured-model, and agent breakdowns. It combines the unfiltered and status-filtered run + counts into per-bucket success and failed, and returns the per-bucket shape in + data-contract.md. Reuse `metricField` and `calculateIntervalFromDuration`, and add a small + `pcts` accessor for p95 and a `freq` reader for the breakdowns. - Add a fetch function in `web/oss/src/services/tracing/api/` that builds the base conditions (the `trace_type is invocation` condition, plus the selected agents' `references` - conditions), passes the explicit `specs`, and calls `fetchSpansAnalytics`. Per window it - issues the queries in data-contract.md: a bucketed unfiltered query for the charts, a - bucketed status-filtered query for failed runs, and a no-interval unfiltered query for the - exact summary-tile numbers including window-level p95. The status-filtered query adds + conditions), passes the explicit `specs`, and calls `fetchSpansAnalytics`. Per the selected + window it issues the two bucketed queries in data-contract.md: an unfiltered query for the + charts, and a status-filtered query for failed runs. The status-filtered query adds `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` and reads the run count - as failed runs; it needs only the run-count spec. The previous window needs only the - no-interval unfiltered and status-filtered calls for the change badges. + as failed runs; it needs only the run-count spec. - Validate three things against one live response while building this phase, because the - backend fails silently on all three: the nested `pcts.p95` shape reads correctly; the exact - enum literal for the `trace_type` filter and the `STATUS_CODE_ERROR` filter both narrow - rather than returning an empty 200; and the coverage of `gen_ai.usage.cost` and the token - split on real traffic (they may be near zero — this is the Phase 0 investigation surfacing). -- Derive the previous comparison window as the equal-length window immediately before the - selected one: for `[oldest, newest]`, the previous window is - `[oldest - (newest - oldest), oldest]`. This works for both preset and custom ranges. + backend fails silently on all three: the nested `pcts.p95` shape reads correctly; the + `trace_type is invocation` and `status_code is STATUS_CODE_ERROR` filters both narrow rather + than returning an empty 200 (the literals are confirmed against the `TraceType` and + `OTelStatusCode` enums; the live check is that the runner actually stamps those statuses); and + the coverage of `gen_ai.usage.cost` and the token split on real traffic (they may be near zero + — this is the Phase 0 investigation surfacing). -Done when: a temporary log or test shows correct totals for a known window, the failed-run -filter is confirmed to narrow the result, and the previous window is fetched for comparison. +Done when: a temporary log or test shows correct per-bucket totals for a known window, and the +failed-run filter is confirmed to narrow the result. ## Phase 3: state and page assembly @@ -91,12 +90,11 @@ Goal: the page is interactive; controls drive the data. - Add atoms under `web/oss/src/state/analytics/` following `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default: the - last 7 days), an agents-filter atom, a current-window query atom, and a previous-window - query atom (or one query returning both). Each query atom drives the calls per window - described in data-contract.md. Key every atom on project id, time range, and the agents - filter. Set `staleTime` to one minute and `refetchOnWindowFocus: false`. Do not reuse - `observabilityDashboardTimeRangeAtom`; a second consumer already shares it, so give this page - its own atoms. + last 7 days), an agents-filter atom, and a query atom for the selected window. The query atom + drives the two bucketed calls in data-contract.md (unfiltered and status-filtered). Key every + atom on project id, time range, and the agents filter. Set `staleTime` to one minute and + `refetchOnWindowFocus: false`. Do not reuse `observabilityDashboardTimeRangeAtom`; a second + consumer already shares it, so give this page its own atoms. - Build the header controls: the time-range control and the Filters popover with the Agents multi-select, a Harness multi-select, and a configured-Model multi-select. Reuse the observability time windowing (the `Sort` control and its `SortResult`) for the time range, @@ -114,23 +112,18 @@ Goal: the page is interactive; controls drive the data. Done when: changing the time range, the agents filter, the harness filter, or the model filter refetches and the page reflects it, with all four states wired. -## Phase 4: summary panel and the charts +## Phase 4: the charts Goal: the full locked-scope UI. Every chart on this page means what its title says and needs no backend capability that does not exist. -- **Summary panel:** a health donut component (recharts `RadialBarChart` or a small SVG ring) - showing `round(100 x successRate)` with the band label and prose. Below the run-count floor - it renders the neutral "Not enough runs yet" state instead of a band. Plus stat tiles for - total runs, average latency, p95 latency, and total tokens, each with a change badge versus - the previous window (green when the change is good for that metric, red otherwise; lower - latency is good) and a sparkline of the current window. Add a coverage-gated total-cost tile - that renders only when cost coverage clears the threshold and otherwise shows an explicit - "cost data not available for this window", never a zero. -- **Charts**, each a card with a title, a one-line description, a recharts chart, and a - toggleable legend: +- Each chart is a card with a title, a one-line description, a recharts chart, and a toggleable + legend: - Runs: stacked bars, successful and failed. - - Latency: bars of average with a p95 reference line; tooltip shows average, p95, min, max. + - Latency: bars of average with a per-bucket p95 line; tooltip shows average, p95, min, max. + - Cost: total cost per period, coverage-gated. It renders only when cost coverage clears the + threshold and otherwise shows an explicit "cost data not available for this window", never + a zero. There is no prompt/completion split; `gen_ai.usage.cost` is a total only. - Tokens: total tokens per period, with a coverage label. Show the prompt/completion split as stacked bars only when the split coverage gate passes; below it, show total only. - Runs per harness: one `categorical/single` spec, per period. @@ -138,14 +131,11 @@ backend capability that does not exist. model" (this is the author's alias, not the model that answered). - Runs per agent: one `categorical/single` spec over the unioned `workflow_variant` and `application_variant` reference families, per period. -- There is no Costs prompt/completion chart. The cost split does not exist in the data, and the - one working cost path (`gen_ai.usage.cost`) is a coverage-gated total, shown as the tile - above rather than a per-part chart. - Lay the charts out in a responsive grid. All colors come from theme tokens and the theme scale; verify both light and dark themes and hover, empty, unavailable, and loading states. -Done when: the in-scope charts and the summary panel render correctly in both themes across all -four states, and `pnpm lint-fix` passes. +Done when: the in-scope charts render correctly in both themes across all four states, and +`pnpm lint-fix` passes. ## Phase 5 (deferred, after a backend change): Tools and resolved Models @@ -210,9 +200,8 @@ reusable shell so Phase 5 only supplies data and series config. ## Testing and verification - Follow `docs/designs/testing/README.md`. Add unit tests for the new mapper (pure function: - buckets in, dashboard shape out), for the success/failed split, for the coverage gate (a - metric below threshold is suppressed, not shown as zero), and for the health-score - computation. These need no live database. + buckets in, chart shape out), for the success/failed split, and for the coverage gate (a + metric below threshold is suppressed, not shown as zero). These need no live database. - Verify the charts against one live project by running the local stack per the root `AGENTS.md` local dev loop. Confirm the `trace_type` and `status_code` filters narrow rather than widen, and confirm the p95 nested read. @@ -221,18 +210,19 @@ reusable shell so Phase 5 only supplies data and series config. ## Risks and unknowns to resolve during the build - **Cost and token-split coverage** (Phase 0). Whether either has usable coverage on the target - data, and the cause of the mid-July collapse. The cost tile and the token split chart depend - on it. Both measured stacks were local dev; production is unverified. -- **Silent-failure validation** (Phase 2). The exact `trace_type` and `status_code` enum - literals that narrow rather than returning an empty 200, and the nested `pcts.p95` read. + data, and the cause of the mid-July collapse. The Cost chart and the token split depend on it. + Both measured stacks were local dev; production is unverified. +- **Silent-failure validation** (Phase 2). Confirm that the `trace_type is invocation` and + `status_code is STATUS_CODE_ERROR` filters narrow rather than return an empty 200 — the literals + are confirmed against the enums, so the open part is whether the runner stamps those statuses — + and confirm the nested `pcts.p95` read. - The correct agents-list atom for the filter options (resolve in Phase 1). - Whether the multi-agent reference filter uses `or` grouping or a single `in` with multiple values in this dialect (resolve in Phase 2 against the existing filter builder). - Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The - failed-run count and the health score depend on it; validate on live traffic in Phase 2. The - root-status definition also misses in-run failures that recovered a clean root (~1.2% of runs - on the measured data); state the definition in a tooltip. -- The run-count floor for the neutral health state (tune in Phase 4; start around 20). + failed-run count depends on it; validate on live traffic in Phase 2. The root-status definition + also misses in-run failures that recovered a clean root (~1.2% of runs on the measured data); + state the definition in a tooltip. - Performance at scale: the 30-day window is the shape most at risk of crossing the 15-second statement timeout as a project grows. Keep 7 days the default and treat longer windows as an explicit choice with a loading state. diff --git a/docs/design/agent-analytics/research.md b/docs/design/agent-analytics/research.md index 1ab6e41d1d..ac5982d1f1 100644 --- a/docs/design/agent-analytics/research.md +++ b/docs/design/agent-analytics/research.md @@ -1,8 +1,8 @@ # Research: what this feature reuses Every path below was read directly. The takeaway: the data path from endpoint to dashboard -already exists for the Observability page. This feature adds a new page on top of that -spine and reads a few response fields the current mapper drops. +already exists for the Observability page. This feature adds a new page on top of that spine and +reads a few response fields the current mapper drops. ## The analytics fetch layer (reuse, extend by one field) @@ -10,14 +10,14 @@ spine and reads a few response fields the current mapper drops. - `fetchSpansAnalytics(params)` calls `POST /spans/analytics/query` through the Fern client (`getTracesClient().querySpansAnalytics`) and validates the response with - `analyticsResponseSchema`. Returns `null` on a non-2xx response or a shape mismatch. + `analyticsResponseSchema`. It returns `null` on a non-2xx response or a shape mismatch. - `SpansAnalyticsParams` today: `projectId`, `appId`, `focus` (default `trace`), `interval` - (bucket minutes), `oldest` / `newest` (ISO bounds), `filter` (a `{conditions: [...]}` - object, serialized to a JSON-string query param), `abortSignal`. -- It intentionally omits `specs`, so the backend applies its default set. To get the prompt - and completion split this page needs, add an optional `specs` field to - `SpansAnalyticsParams` and pass it through as a JSON-string query param, the same way - `filter` is passed. The endpoint already accepts `specs`; this is a frontend-only change. + (bucket minutes), `oldest` / `newest` (ISO bounds), `filter` (a `{conditions: [...]}` object, + serialized to a JSON-string query param), `abortSignal`. +- It omits `specs` on purpose, so the backend applies its default set. To get the prompt and + completion split this page needs, add an optional `specs` field to `SpansAnalyticsParams` and + serialize it exactly as `filter` is serialized. The Fern `QuerySpansAnalyticsRequest` already + declares `specs?: string`, so the change is one field and one line — a frontend-only change. ## The response-to-dashboard mapper (reuse pattern, new mapper for richer fields) @@ -31,83 +31,80 @@ spine and reads a few response fields the current mapper drops. - `attributes.ag.metrics.duration.cumulative` (fields `sum`, `count`) - `attributes.ag.metrics.errors.cumulative` (field `sum`) - `attributes.ag.type.trace` (field `count`) -- It derives run count from the `ag.type.trace` count, failures from the errors sum, success - as `total - failures`, and average latency as `durationSum / durationCount` in - milliseconds. -- `metricField(metrics, path, field)` is the safe reader for one **flat** numeric field - (`count`, `sum`, `min`, `max`). Reuse it for those. It reads one level deep, so it does - **not** reach percentiles; p95 lives at `metrics[path].pcts.p95`, one level further down. - Add a small nested accessor (or extend `metricField` with a two-key form) for the p95 read; - do not assume p95 is a flat sibling of `sum`. +- It derives run count from the `ag.type.trace` count, failures from the errors sum, success as + `total - failures`, and average latency as `durationSum / durationCount` in milliseconds. +- `metricField(metrics, path, field)` safely reads one **flat** numeric field (`count`, `sum`, + `min`, `max`). Reuse it for those. It reads one level deep, so it does **not** reach + percentiles; p95 lives at `metrics[path].pcts.p95`, one level further down. Add a small nested + accessor (or extend `metricField` with a two-key form) for the p95 read; do not assume p95 is a + flat sibling of `sum`. - What it does not read, and this page needs: - `gen_ai.usage.cost` (field `sum`, plus `count` for the coverage gate) for a coverage-gated total-cost tile. The canonical `costs.cumulative.*` paths hold no data on agent root spans, so there is no prompt/completion cost split to read; see data-contract.md. - - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split, which - is coverage-gated because it shares the cost field's mid-July coverage collapse. - - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for - the Latency tooltip and marker. + - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split, which is + coverage-gated because it shares the cost field's mid-July coverage collapse. + - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for the + Latency tooltip and marker. - Category `freq` arrays on `ag.data.parameters.agent.harness.kind`, `ag.data.parameters.agent.llm.model`, and the agent `references` paths for the breakdown charts. - This page does not reuse the existing mapper's `errors.cumulative`-based failure count. A - failed run is a run whose root span status is `STATUS_CODE_ERROR` (there is no - `STATUS_CODE_OK` on root spans; success is the complement), which a metric spec cannot read, - so the page gets the failed-run count from a separate status-filtered query. See - data-contract.md. -- `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar - count reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for - the time-range-to-interval mapping. + failed run is a run whose root span status is `STATUS_CODE_ERROR` (there is no `STATUS_CODE_OK` + on root spans; success is the complement), which a metric spec cannot read, so the page gets + the failed-run count from a separate status-filtered query. See data-contract.md. +- `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar count + reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for the + time-range-to-interval mapping. `web/oss/src/services/tracing/api/index.ts` - `fetchGenerationsDashboardData(appId, options)` builds the `conditions` array (it pushes a - `references in [{id: appId}]` condition when an app id is present), computes the interval, - and calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new - page's fetch function. For project scope, omit the single-app reference condition and add - reference conditions only for the agents the user selects in the filter, plus the base - `trace_type is invocation` condition on every query. The new page also issues a second query - per window with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` filter - to count failed runs. + `references in [{id: appId}]` condition when an app id is present), computes the interval, and + calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new page's + fetch function. For project scope, drop the single-app reference condition and add reference + conditions only for the agents the user selects, plus the base `trace_type is invocation` + condition on every query. The new page also issues a second query per window, filtered by + `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}`, to count failed runs. ## The dashboard state atoms (reuse pattern, new atoms for this page) `web/oss/src/state/observability/dashboard.ts` -- `observabilityDashboardQueryAtom` is an `atomWithQuery` keyed on app id, project id, and - the time-range atom; it calls `fetchGenerationsDashboardData` with a `staleTime` of one - minute and `refetchOnWindowFocus: false`. +- `observabilityDashboardQueryAtom` is an `atomWithQuery` keyed on app id, project id, and the + time-range atom; it calls `fetchGenerationsDashboardData` with a `staleTime` of one minute and + `refetchOnWindowFocus: false`. - `observabilityDashboardTimeRangeAtom` holds the selected range as a `SortResult`. -- `useObservabilityDashboard()` unwraps loading and fetching flags. -- The new page follows this exact shape with its own atoms: a time-range atom, an - agents-filter atom, a main query atom for the current window, and a second query atom (or - a widened single query) for the previous window that the change badges compare against. +- `useObservabilityDashboard()` unwraps the loading and fetching flags. +- The new page follows this exact shape with its own atoms: a time-range atom, an agents-filter + atom, and a query atom for the selected window. That query drives the two bucketed calls, + unfiltered for the charts and status-filtered for failed runs. ## Existing dashboard UI (reference, not reused directly) `web/oss/src/components/pages/observability/dashboard/` -- `AnalyticsDashboard.tsx` renders `WidgetCard`s and `CustomAreaChart`s from the mapped - data, with a `Sort` time-range selector and antd `Spin` for loading. +- `AnalyticsDashboard.tsx` renders `WidgetCard`s and `CustomAreaChart`s from the mapped data, + with a `Sort` time-range selector and antd `Spin` for loading. - `CustomAreaChart.tsx` wraps recharts area charts. -- These render **area** charts with total figures. This page needs **stacked bar** charts, a - **horizontal bar** chart (for the deferred Tools view), sparklines, and a donut, so it - brings its own chart components rather than bending the area chart. It still follows the - same card-plus-chart composition and the same loading and empty-state conventions. +- These render **area** charts of total figures. This page needs **stacked and grouped bar** + charts, a per-bucket **p95 line** on the latency chart, and a **horizontal bar** chart (for the + deferred Tools view), so it brings its own chart components rather than bend the area chart. It + still follows the same card-plus-chart composition and the same loading and empty-state + conventions. ## Charting library -`web/oss/package.json` depends on **recharts `^3.1.0`**. Existing recharts usage to copy -style and theming from: +`web/oss/package.json` depends on **recharts `^3.1.0`**. Existing recharts usage to copy style +and theming from: - `web/oss/src/components/pages/observability/dashboard/CustomAreaChart.tsx` - `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/BarChart.tsx` - `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/HistogramChart.tsx` -Build the page's charts with recharts (`BarChart` stacked and horizontal, `LineChart` or -`AreaChart` for sparklines, `RadialBarChart` or a small SVG ring for the donut). Do not port -the reference implementation's hand-rolled SVG chart code; it hardcodes hex colors and -duplicates what recharts gives. +Build the page's charts with recharts (`BarChart` stacked and horizontal, `LineChart` for the +per-bucket p95 overlay on the latency chart). Do not port the reference implementation's +hand-rolled SVG chart code; it hardcodes hex colors and duplicates what recharts already gives. ## Routing and sidebar @@ -124,9 +121,9 @@ duplicates what recharts gives. Add `analytics/index.tsx` as the same kind of thin wrapper around a new `components/pages/analytics` module. -- The sidebar project items are defined in - `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`. The Observability entry - is the shape to copy: +- The sidebar project items live in + `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`. The Observability entry is + the shape to copy: ```tsx { @@ -138,16 +135,16 @@ duplicates what recharts gives. } ``` - Add an `Analytics` entry with its own key, a `${projectURL}/analytics` link, and an icon - from `@phosphor-icons/react`. + Add an `Analytics` entry with its own key, a `${projectURL}/analytics` link, and an icon from + `@phosphor-icons/react`. ## The agents list for the filter -The Agents multi-select needs the project's agents as options. The filter narrows the query -by pushing `references in [{id: }]` conditions (the same field the observability -fetch uses for a single app). Source the option list from the existing apps or workflows -state rather than a new endpoint. Confirm the exact atom during Phase 1; candidates are the -app-management or workflow molecule selectors already used by the sidebar's agent switcher. +The Agents multi-select needs the project's agents as options. The filter narrows the query by +pushing `references in [{id: }]` conditions — the same field the observability fetch +uses for a single app. Source the option list from the existing apps or workflows state rather +than a new endpoint. Confirm the exact atom during Phase 1; the candidates are the app-management +or workflow molecule selectors the sidebar's agent switcher already uses. ## Conventions that constrain the build @@ -155,12 +152,12 @@ From `web/AGENTS.md`: - All new API calls go through the Fern client and the per-resource accessors in `@agenta/sdk/resources`. Keep zod validation at the boundary with `safeParseWithLogging`. -- Data fetching uses Jotai `atomWithQuery`; never `useEffect` with manual state. Put every +- Data fetching uses Jotai `atomWithQuery`, never `useEffect` with manual state. Put every reactive dependency in the `queryKey`; set a sensible `staleTime`. - Exactly one project is ever in scope. Do not write multi-project-defensive code. - Styling is Tailwind utility classes plus antd semantic tokens (`bg-colorBgContainer`, - `text-colorText`, and the `--ag-color*` variables). No raw hex, no inline `style`, no - CSS-in-JS except for antd overrides Tailwind cannot express. Implement and verify both - light and dark themes. The reference palette maps onto these tokens; series colors come - from the theme scale, not from literals. + `text-colorText`, and the `--ag-color*` variables). No raw hex, no inline `style`, no CSS-in-JS + except for antd overrides Tailwind cannot express. Implement and verify both light and dark + themes. The reference palette maps onto these tokens; series colors come from the theme scale, + not from literals. - Keep in-code comments to one line. diff --git a/docs/design/agent-analytics/scope.md b/docs/design/agent-analytics/scope.md new file mode 100644 index 0000000000..ba443c3923 --- /dev/null +++ b/docs/design/agent-analytics/scope.md @@ -0,0 +1,93 @@ +# Scope: v1 now, v2 later + +This page draws the line between what ships now (**v1**) and what waits (**v2**). It is the +roadmap view; plan.md holds the build steps, and capability-review.md holds the evidence behind +every verdict. + +## The rule + +- **v1** is the current implementation: the charts and filters we build against today's endpoint + — runs, latency, cost, tokens, and the harness / configured-model / agent breakdowns. +- **v2** is anything that needs backend integration or a major fix — a new query capability, a + storage change, or new instrumentation. +- **Exception, by decision of the requester:** the two backend items in Phase 0 (make a killed or + rejected query surface an error, and investigate the cost / token coverage collapse) stay in + **v1 as blockers**, not v2. v1 does not ship until they land, because without them the page + cannot tell "no data" from "the query died", and cost may read empty with no signal. + +## How to read the tables + +- **Priority** ranks work within each scope by value against effort: `P0` first, then `P1`, `P2`, + `P3`. +- **Value** — how much the feature matters to a user: High / Med / Low. +- **Effort** — how much work it is: `S` (a day or so), `M` (a few days), `L` (a week or more, or + open-ended). +- **Backend blocker** rows are flagged; everything else in v1 is frontend-only. + +--- + +## Scope v1 — build now + +Delivered against today's endpoint, plus the two backend blockers. Every capability here is +verified "ready" in capability-review.md §4.2 unless a note says otherwise. + +| Priority | Feature | What it delivers | Value | Effort | Depends on | +|---|---|---|---|---|---| +| P0 | Page shell, route, sidebar | The Analytics page exists, reachable, project-scoped | Enabler | S | — | +| P0 | Data layer (`specs` field + fetch + mapper) | The page requests explicit specs and maps buckets to chart shape | Enabler | M | — | +| P0 | **B1 — Make killed / rejected queries surface an error** *(backend blocker)* | 504 / 4xx + per-metric `sample_count` instead of a silent empty 200 | High | M | — | +| P0 | Runs per period (success vs failed) | Stacked Runs chart; corrected `status_code is STATUS_CODE_ERROR` filter + second query | High | M | B1 | +| P0 | Latency per period (avg + p95 + min/max) | Latency chart with per-bucket p95 line; ships free on the numeric spec | High | S | — | +| P0 | Time-range control (any window, 7-day default) | Reuses the observability `Sort` / `SortResult` | High | S | — | +| P0 | Four page states (data / no-data / unavailable / failed) | Honest empty vs failure vs coverage-gap, per card | High | M | B1 | +| P1 | **B2 — Investigate cost / token coverage collapse** *(backend blocker)* | Restores dependable coverage for cost and the token split | High | M–? | — | +| P1 | Cost per period (coverage-gated total) | Cost chart from `gen_ai.usage.cost`; renders only above the coverage threshold | High | M | B2 | +| P1 | Tokens per period (total + coverage-gated split) | Tokens chart; the prompt/completion split shows only above threshold | Med | M | B2 (split only) | +| P1 | Filter by agent + runs-per-agent breakdown | Agents multi-select; breakdown unions `workflow_variant` + `application_variant` | High | M | — | +| P2 | Runs per harness + harness filter | Harness breakdown chart and filter | Med | S | — | +| P2 | Runs per configured model + model filter | Configured-model breakdown and filter (author's alias, labeled "configured model") | Med | S | — | + +Notes: + +- **Cost** is "blocked on coverage" in the review (§4.2 item 4). v1 ships the chart + coverage-gated; B2 is what makes it dependable. +- **Configured model** is a proxy for the model that answered — the author's alias. It is + labeled honestly; the real answered model is v2 (needs `focus=span`). +- **Not on the v1 page at all:** tool usage, resolved-model usage, per-model cost, cache tokens, + per-user numbers, skills. All are v2 (below), each for a backend reason. + +--- + +## Scope v2 — future, needs backend work + +Ordered by dependency and combined value-vs-effort. The backend column names the enabling change +and its capability-review.md §8.4 item. + +| Priority | Feature | What it delivers | Value | Effort | Backend work it needs | +|---|---|---|---|---|---| +| P0 | Typed analytics contract | Named metrics / dimensions / aggregations, validated and capped; the foundation the rest build on | High | M | Replace the arbitrary-path protocol (§8.4 item 2) | +| P1 | Stop reading JSONB on the chart path | Hot columns or a per-run facts table; a permanent latency win and a home for invoked facts | High | M–L | Ingest + storage change (§8.4 item 4) | +| P1 | Tool usage per period (which tools ran) | Real tool-call counts, and a path to per-tool error rate | High | M | `focus=span` wiring + 3 guards + non-root index (§8.4 item 3) | +| P1 | Resolved model usage per period | The model that actually answered, distinct from the configured alias | High | M | `focus=span` (shares the wiring) | +| P2 | Per-model cost / tokens | A numeric metric split by model in one call | High | L | `focus=span` + a group-by dimension (§8.4 item 5) | +| P2 | Cost mapped to the canonical path | `gen_ai.usage.cost` → `ag.metrics.costs.cumulative.total`; also fixes evaluation cost | Med | S | Semconv adapter map (§8.4) | +| P2 | Cache tokens per period | Cache read and write per model call | Low–Med | M | `focus=span`, or roll up to the root | +| P2 | Per-user numbers | Runs / latency / cost per user | Med* | S | A nameable `created_by_id` dimension (needs the contract or facts table) (§8.4 item 6) | +| P2 | Calendar-aware periods | True calendar days and months, timezone-correct | Low | M | Timezone-aware bucketing in the backend (§4.4 item 1) | +| P3 | Skills used (invoked) | Which skills the agent actually invoked | Med | L | Runner instrumentation, then promotion to the root (§4.4 item 14) | +| P3 | Pre-aggregation rollups | Fast wide-window queries at high volume | Med | L | A rollup table; sequence last, after facts land (§8.4 item 7) | + +\* Per-user value is conditional: it means nothing until a project has more than one writing +credential. On both measured stacks `created_by_id` had exactly one value per project (§4.4 +item 13). + +--- + +## Where each source draws the line + +- **capability-review.md §4.2** — the per-capability verdict table (ready / proxy / not + available) that seeds v1 versus v2. +- **capability-review.md §8.3** — the v1 beta scope. +- **capability-review.md §8.4** — the v2 backend work, in dependency order. +- **plan.md** — the v1 build, phase by phase (Phase 0 = the two blockers; Phases 1–4 = the page; + Phase 5 = the first slice of v2). diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index e2170946f2..a7dd1378ab 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -4,89 +4,84 @@ Source of truth for progress. Update as work lands. ## Current state -Planning complete, and a grilling session on 2026-08-02 sharpened the design. No code written. -The workspace holds the plan; implementation has not started and no branch exists yet. +Planning is complete, and a grilling session on 2026-08-02 sharpened the design. No code is +written. The workspace holds the plan; implementation has not started, and no branch exists yet. ## Locked decisions -1. Frontend-first scope with two backend prerequisites (Phase 0 in plan.md). Charts: Runs, - Latency, Tokens (coverage-gated split), and runs-per-harness / -configured-model / -agent - breakdowns. No Costs prompt/completion chart; cost is a coverage-gated total tile from - `gen_ai.usage.cost`. Stat tiles, health donut, Agents / Harness / configured-Model filters, - time-range control. The two Phase 0 backend items: make a killed or rejected query - distinguishable from an empty one, and investigate the cost / token-split coverage collapse. -2. New page at project scope named Analytics; the default query aggregates all project agents, - and the Agents filter narrows the set. -3. An **agent** is an application/workflow artifact. The Agents multi-select is the only - filter; it lists the project's agents and narrows by `references`. -4. The count of agent invocations is called a **run**. The Observability dashboard calls the - same metric a "request"; the two are allowed to diverge until Observability is aligned in a - later, separate change. -5. A **failed run** is a run whose root span `status_code` is `STATUS_CODE_ERROR` (a run-level - outcome, not a count of errored steps; there is no `STATUS_CODE_OK`, so success is the - complement). It comes from a second, status-filtered analytics query, not a metric spec. -6. The **health score** is the success rate: `round(100 x successRate)`, banded Healthy 85+, - Watch 65 to 84, At risk below 65. Latency was dropped from it. Below a run-count floor the - donut shows a neutral "Not enough runs yet" state instead of a band. -7. The time-range control opens on the **last 7 days** and accepts any window via the +1. Frontend-first scope, with two backend prerequisites (Phase 0 in plan.md). Charts: Runs, + Latency, Cost (coverage-gated total), Tokens (coverage-gated split), and runs per harness / + configured model / agent. No Costs prompt/completion split chart; `gen_ai.usage.cost` is a + total only. Plus the Agents / Harness / configured-Model filters and the time-range control. + The two Phase 0 items: make a killed or rejected query distinguishable from an empty one, and + investigate the cost / token-split coverage collapse. +2. A net-new page at project scope, named Analytics. The default query aggregates every project + agent; the Agents filter narrows the set. +3. An **agent** is an application/workflow artifact. The Agents multi-select is the only filter; + it lists the project's agents and narrows by `references`. +4. One agent invocation is a **run**. The Observability dashboard calls the same metric a + "request"; the two may diverge until Observability is aligned in a later, separate change. +5. A **failed run** is a run whose root span `status_code` is `STATUS_CODE_ERROR` — a run-level + outcome, not a count of errored steps. There is no `STATUS_CODE_OK`, so success is the + complement. It comes from a second, status-filtered query, not a metric spec. The Runs chart's + successful-and-failed split builds on it. +6. The time-range control opens on the **last 7 days** and accepts any window through the observability `Sort` control and `SortResult`. -8. The deferred **Tools** and **Models** views are omitted entirely in this release, not shown - as placeholders. The chart-card shell is built reusably so they drop in later. +7. The deferred **Tools** and **Models** views ship in no form this release, not even as + placeholders. The chart-card shell is built reusably, so they drop in later. ## Key finding from research The data path from the analytics endpoint to a mapped dashboard shape already exists for the Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability dashboard -atoms). This feature reuses that spine. The data-layer work is: pass explicit metric specs for -the total cost (`gen_ai.usage.cost`) and the token split, read the duration min/max/p95 the -current mapper drops, read the category `freq` breakdowns, and add the status-filtered -failed-run query. The endpoint returns the latency, run-count, and breakdown fields directly; -cost and the token split are coverage-gated because their populated paths collapsed in mid-July -(Phase 0). This is why the scope carries two backend prerequisites rather than none. +atoms). This feature reuses that spine. The data-layer work: pass explicit metric specs for the +total cost (`gen_ai.usage.cost`) and the token split, read the duration min/max/p95 the current +mapper drops, read the category `freq` breakdowns, and add the status-filtered failed-run query. +The endpoint returns the latency, run-count, and breakdown fields directly. Cost and the token +split stay coverage-gated, because their populated paths collapsed in mid-July — which is why +the scope carries two backend prerequisites (Phase 0) rather than none. ## Resolved by code verification -- **Spec `type` strings**: every number metric is `numeric/continuous` (there is no bare - `numeric` in `MetricType`; `DEFAULT_ANALYTICS_SPECS` confirms it). -- **p95 field**: nested at `metrics[path].pcts.p95`, not a flat field; `metricField` does not +- **Spec `type` strings**: every number metric is `numeric/continuous`. `MetricType` has no bare + `numeric`, and `DEFAULT_ANALYTICS_SPECS` confirms it. +- **p95 field**: nested at `metrics[path].pcts.p95`, not a flat field. `metricField` does not reach it, so the mapper needs a small `pcts` accessor. -- **`specs` plumbing**: the Fern `QuerySpansAnalyticsRequest` type already carries - `specs?: string` and forwards it, so passing specs is a small entities-layer change. -- **Failed-run mechanism**: `status_code` is a table column and metric specs read only the - `attributes` JSON (`build_extract_cte`), so a spec cannot target it. Failed runs come from a - second query with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` - filter (`status_code` is a first-class filter field; the operator must be `is` and the value - the full enum literal, or the backend returns an empty 200). Per window the page issues a - bucketed unfiltered query, a bucketed status-filtered query, and a no-interval unfiltered - query for exact tile numbers including window-level p95; the previous window needs only the - no-interval unfiltered and status-filtered calls. See data-contract.md. +- **`specs` plumbing**: the Fern `QuerySpansAnalyticsRequest` type already carries `specs?: string` + and forwards it, so passing specs is a small entities-layer change. +- **Failed-run mechanism**: `status_code` is a table column, and metric specs read the + `attributes` JSON only (`build_extract_cte`), so no spec can target it. Failed runs come from a + second query with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` filter + (`status_code` is a first-class filter field; the operator must be `is` and the value the full + enum literal, or the backend returns an empty 200). Per the selected window the page issues two + bucketed queries — one unfiltered for the charts, one status-filtered for failed runs — and + every chart reads them per bucket. See data-contract.md. ## Open questions to resolve during the build -- Agents-list atom for the filter options (Phase 1). -- Multi-agent reference filter encoding: a single `in` with all ids, or one condition per +- The agents-list atom for the filter options (Phase 1). +- The multi-agent reference filter encoding: a single `in` with all ids, or one condition per agent combined with `or` (Phase 2, against the existing filter builder). - Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The - failed-run count and health score depend on it; validate on live traffic (Phase 2). + failed-run count depends on it; validate on live traffic (Phase 2). - Live-response validation: the `trace_type` and `status_code` filters narrow rather than returning an empty 200, the nested `pcts.p95` reads correctly, and the cost / token-split - coverage on real traffic (Phase 2; may be near zero — the Phase 0 investigation). -- The run-count floor for the neutral health state (Phase 4; start around 20). + coverage on real traffic (Phase 2; it may be near zero — the Phase 0 investigation). ## Deferred to a later backend change -Tools chart, Models chart, Models filter, per-model cost. Split into two sub-phases because -the two backend prerequisites are not co-equal gates: +The Tools chart, the Models chart, the Models filter, and per-model cost. They split into two +sub-phases, because the two backend prerequisites are not co-equal gates: -- **5a, Tools and Models-share**: needs only span-focus wiring (thread `focus` into - `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by needed, because a +- **5a, Tools and Models-share**: needs span-focus wiring only (thread `focus` into + `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by, because a `categorical/single` spec already returns per-value frequencies. Ship a guard: under - `focus = "span"`, cumulative metrics double-count, so cost/tokens must use `incremental` + `focus = "span"`, cumulative metrics double-count, so cost/tokens must use the `incremental` paths. -- **5b, per-model cost/tokens**: additionally needs a group-by dimension. Harder; defer - until 5a ships. +- **5b, per-model cost/tokens**: additionally needs a group-by dimension. Harder; defer until 5a + ships. -Open design decision for 5b: group-by as a per-query dimension (preferred) vs a per-spec +Open design decision for 5b: group-by as a per-query dimension (preferred) versus a per-spec `group_by` field. Phase 5 in plan.md; not scheduled here. ## Source materials @@ -97,5 +92,5 @@ Open design decision for 5b: group-by as a per-query dimension (preferred) vs a `https://claude.ai/code/artifact/75b4f14e-9c9b-407b-9d35-317927fb6772`. - Backend capability notes: `docs/design/agent-analytics/Note.md`. - Endpoint architecture review (the source of the root-span-only and dead-`focus` findings): a - read-only review generated 2026-08-01; its conclusions are captured in context.md and - data-contract.md, so the workspace does not depend on the review file. + read-only review generated 2026-08-01. Its conclusions live in context.md and data-contract.md, + so the workspace does not depend on the review file. From 032358ef2e7180d2d20a85056322fd2ff16c5095 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Mon, 3 Aug 2026 17:54:08 +0600 Subject: [PATCH 7/9] docs(analytics): enhance agent analytics documentation with detailed filter and query specifications --- docs/design/agent-analytics/context.md | 20 ++++-- docs/design/agent-analytics/data-contract.md | 65 ++++++++++++++------ docs/design/agent-analytics/plan.md | 27 +++++--- docs/design/agent-analytics/scope.md | 16 ++--- docs/design/agent-analytics/status.md | 15 +++-- 5 files changed, 97 insertions(+), 46 deletions(-) diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md index 54fb8cb46e..0eb4238ef8 100644 --- a/docs/design/agent-analytics/context.md +++ b/docs/design/agent-analytics/context.md @@ -16,8 +16,10 @@ A new page named Analytics, reachable from the project sidebar and scoped to the It shows: - A header: the page title, a one-line description, a time-range control that accepts any - window, and a Filters popover whose Agents multi-select narrows the query to chosen agents. - The time-range control reuses the observability windowing (the `Sort` control and its + window, and a Filters popover with three multi-selects that narrow every query: Agents (by + `references`), Harness, and configured Model. All three are server-side filter conditions on + root-span fields, so changing any of them refetches (see data-contract.md for the condition + each one adds). The time-range control reuses the observability windowing (the `Sort` control and its `SortResult`), so it takes the standard presets and a custom start-and-end range, not a fixed list of options. It opens on the last 7 days. - Charts in a grid, each with hover tooltips and a legend whose entries toggle series on and @@ -30,7 +32,10 @@ It shows: - Tokens: total tokens per bucket, with a coverage label. The prompt/completion split renders as stacked bars only when its coverage gate passes. - Runs per harness, runs per configured model, and runs per agent: category breakdowns, one - `categorical/single` spec each. + `categorical/single` spec each. "Configured model" is the model alias the agent's author + set on the root span; it is **not** the model that actually answered the run. The answered + model lives on child spans and is the deferred model-usage view below. This chart must be + labelled "configured model" and never implemented as the deferred model-usage view. There is no Costs prompt/completion split chart. The split paths hold no data on agent runs, so the Cost chart shows the total from `gen_ai.usage.cost` only. See data-contract.md and @@ -41,7 +46,9 @@ It shows: These decisions are settled and drive the plan. Do not reopen them without the requester. 1. Frontend-first, with two backend prerequisites. Build the charts above, the Agents / Harness / - configured-Model filters, and the time-range control against today's endpoint. The endpoint + configured-Model filters, and the time-range control against today's endpoint. Harness and + configured model appear both as breakdown charts and as filters; all three filters read + root-span attributes, so none of them needs a backend change. The endpoint returns most of the fields these need directly, but two backend items gate a trustworthy release, tracked as Phase 0 in plan.md: make a killed or rejected query distinguishable from a genuinely empty one, and investigate the mid-July collapse in cost and token-split coverage. @@ -67,7 +74,10 @@ per-span status. Today's analytics endpoint reads root spans only. It accepts a that would widen the scan to all spans, but the field is inert, so those fields stay out of reach. -The two backend prerequisites are not co-equal gates; each unlocks a different thing. +The two deferred-view enablers below — span-focus wiring and a group-by dimension — are not +co-equal gates; each unlocks a different thing. These are distinct from the two Phase 0 blockers +in the locked scope above (query-failure visibility and the coverage investigation); do not +conflate the two pairs. - **Span-focus wiring** unlocks both the tool view and the model-share view in their basic form. `query.focus` reaches `dao.analytics()` but never reaches `build_base_cte`, which diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md index f57aa7b49d..45bda915c2 100644 --- a/docs/design/agent-analytics/data-contract.md +++ b/docs/design/agent-analytics/data-contract.md @@ -32,7 +32,14 @@ feeds: so annotation traces (evaluator and human-annotation runs) do not inflate the run count and the latency, cost, and token numbers. At project scope with no agent selected, add nothing else so the query spans the whole project. For selected agents, add `references` conditions - on the chosen agent ids. The `trace_type` value is the unprefixed literal `invocation` + on the chosen agent ids. For a selected **harness**, add a condition on + `attributes.ag.data.parameters.agent.harness.kind`; for a selected **configured model**, add a + condition on `attributes.ag.data.parameters.agent.llm.model`. Both are root-span attributes and + filter today with no backend change (capability-review.md §4.2 items 11–12); the model filter + narrows by the configured alias, not the model that answered. Combine multiple values within one + filter the same way the agent filter does (the encoding to confirm in Phase 2). All three + filters — Agents, Harness, configured Model — are server-side, so each belongs in the query key + and changing any of them refetches. The `trace_type` value is the unprefixed literal `invocation` (verified against the `TraceType` enum, `api/oss/src/core/otel/dtos.py`); unlike `status_code`, it is not prefixed, and the backend validates the value against the enum, so a wrong literal returns an empty 200. The one encoding still to confirm against the filter @@ -128,27 +135,47 @@ a failed run is "the root span errored"; say so in a tooltip. Catching in-run fa ### The queries per window -The page issues two bucketed queries for the selected window, both with `interval` set. Every -chart reads its data per bucket, so these two calls cover the whole page. - -1. **Bucketed, unfiltered**: drives the Runs, Latency, Cost, Tokens, and category-breakdown - charts. -2. **Bucketed, status-filtered**: the per-bucket failed run count for the stacked Runs chart. - -Keep any single request at or under about eight specs, and fan the two calls out in parallel with -a per-call error state. On the measured data a 7-day window costs about 0.26 s and a 30-day -window about 1.7 s for the six-spec shape, so 7 days is the default and 30 or 90 days is an -explicit user choice with a loading state. +The six core metric specs (run count, latency, cost, total tokens, and the two token-split +specs) plus the four category specs (harness, configured model, and the two agent-reference +families) come to **ten specs** — more than one request should carry. So the page does not put +them in a single call. It fans out into small bucketed calls, each at or under about eight +specs, in parallel with a per-call error state: + +1. **Bucketed core metrics** (six specs): run count, latency, cost, total tokens, and the + prompt/completion split. Drives the Runs, Latency, Cost, and Tokens charts. This is the + "six-spec shape" the timings below were measured on. +2. **Bucketed category breakdowns** (four specs): the harness, configured-model, and two + agent-reference categorical specs. Drives the three breakdown charts. +3. **Bucketed status-filtered** (one spec): the per-bucket failed run count for the stacked Runs + chart. + +Every chart reads its data per bucket. On the measured data a 7-day window costs about 0.26 s +and a 30-day window about 1.7 s for the six-spec core call, so 7 days is the default and 30 or +90 days is an explicit user choice with a loading state; the smaller category and status calls +run alongside it. ## Response fields the mapper reads -The response is `{buckets: [{timestamp, interval, metrics: {: {: value}}}]}`. The -new mapper produces, per bucket: - -- `failed` = the `type.trace` count from the status-filtered query (runs whose root span is - `STATUS_CODE_ERROR`). This is a true failed-run count, never larger than the total. -- `success` = the unfiltered `type.trace` count minus `failed`. It cannot go negative, so no - flooring is needed. This departs from the existing observability mapper, which subtracts the +The response is `{buckets: [{timestamp, interval, metrics: {: {: value}}}]}`. + +**Join the parallel calls by `timestamp`, not by array position.** Each call omits its empty +buckets independently (see "Response quirks"), so the unfiltered, category, and status-filtered +responses can have different bucket counts and different offsets — a bucket with successes but no +failures is present in the unfiltered array and absent from the status-filtered one. The mapper +keys each response's buckets by `timestamp`, builds the union x-axis, and fills a missing bucket +as zero for that call. It must also read `buckets[].interval` back and treat the calls as +comparable only when their effective `interval` agrees; if the backend coarsened one call's +interval differently, surface a mismatch state rather than aligning mismatched buckets. Only +under this timestamp join do the per-bucket claims below hold. The new mapper then produces, per +bucket: + +- `failed` = the `type.trace` count from the status-filtered query for that `timestamp` (runs + whose root span is `STATUS_CODE_ERROR`), or zero if the status-filtered call omitted the + bucket. This is a true failed-run count, never larger than the total for the same bucket. +- `success` = the unfiltered `type.trace` count for that `timestamp` minus `failed`. Because + both counts come from the same bucket after the timestamp join, `success` cannot go negative + and no flooring is needed; without the join (positional pairing across omitted buckets) it + could. This departs from the existing observability mapper, which subtracts the `errors.cumulative` sum. That sum counts errored steps, not failed runs, and can exceed the run count, so this page uses the run-level status instead. - `latencyAvg` = duration `sum` / duration `count`, in milliseconds. diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md index 888e7abca9..9fe85a8766 100644 --- a/docs/design/agent-analytics/plan.md +++ b/docs/design/agent-analytics/plan.md @@ -69,8 +69,9 @@ Goal: the page can fetch mapped analytics for the selected window. - Add a fetch function in `web/oss/src/services/tracing/api/` that builds the base conditions (the `trace_type is invocation` condition, plus the selected agents' `references` conditions), passes the explicit `specs`, and calls `fetchSpansAnalytics`. Per the selected - window it issues the two bucketed queries in data-contract.md: an unfiltered query for the - charts, and a status-filtered query for failed runs. The status-filtered query adds + window it fans out the bucketed calls in data-contract.md, each at or under ~8 specs, in + parallel with a per-call error state: a core-metrics call and a category-breakdown call for + the charts, and a status-filtered call for failed runs. The status-filtered call adds `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` and reads the run count as failed runs; it needs only the run-count spec. - Validate three things against one live response while building this phase, because the @@ -90,27 +91,33 @@ Goal: the page is interactive; controls drive the data. - Add atoms under `web/oss/src/state/analytics/` following `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default: the - last 7 days), an agents-filter atom, and a query atom for the selected window. The query atom - drives the two bucketed calls in data-contract.md (unfiltered and status-filtered). Key every - atom on project id, time range, and the agents filter. Set `staleTime` to one minute and + last 7 days), an agents-filter atom, a harness-filter atom, a configured-model-filter atom, + and a query atom for the selected window. The query atom drives the two bucketed calls in + data-contract.md (unfiltered and status-filtered). Key every atom on project id, time range, + and all three filters (agents, harness, configured model) — all three are server-side, so each + must be in the key to refetch. Set `staleTime` to one minute and `refetchOnWindowFocus: false`. Do not reuse `observabilityDashboardTimeRangeAtom`; a second consumer already shares it, so give this page its own atoms. -- Build the header controls: the time-range control and the Filters popover with the Agents - multi-select, a Harness multi-select, and a configured-Model multi-select. Reuse the +- Build the header controls: the time-range control and the Filters popover with three + multi-selects — Agents, Harness, and configured Model. Each adds the filter condition in + data-contract.md (agents by `references`, harness and configured model by their root-span + attribute paths); the configured-model filter narrows by the author's alias, not the model + that answered. Reuse the observability time windowing (the `Sort` control and its `SortResult`) for the time range, so any window, including a custom start-and-end range, works; do not build a fixed list of range options. Offer day, week, and "whole window" as periods; do not offer a month period, because calendar months are not expressible and a fixed 30-day stride is not a month. Align `oldest` to the viewer's local midnight and read `buckets[].interval` back to label the axis with the period that actually ran. Source the Agents options from the agents atom confirmed - in Phase 1. + in Phase 1; source the Harness and configured-Model options from the distinct values already + in the breakdown charts' `freq` arrays, so the filters need no extra call. - Implement four explicit page states per card, following `AnalyticsDashboard.tsx` (antd `Spin` for loading): data, no data in this window, metric unavailable (coverage below the threshold), and request failed. The last two exist because a wrong filter or a killed query renders as an empty success today; do not collapse them into a single "No data" empty state. -Done when: changing the time range, the agents filter, the harness filter, or the model filter -refetches and the page reflects it, with all four states wired. +Done when: changing the time range or any of the three filters (agents, harness, configured +model) refetches and the page reflects it, with all four states wired. ## Phase 4: the charts diff --git a/docs/design/agent-analytics/scope.md b/docs/design/agent-analytics/scope.md index ba443c3923..0612bca116 100644 --- a/docs/design/agent-analytics/scope.md +++ b/docs/design/agent-analytics/scope.md @@ -6,8 +6,9 @@ every verdict. ## The rule -- **v1** is the current implementation: the charts and filters we build against today's endpoint - — runs, latency, cost, tokens, and the harness / configured-model / agent breakdowns. +- **v1** is the planned release scope: the charts and Agents filter we will build against + today's endpoint — runs, latency, cost, tokens, and the harness / configured-model / agent + breakdowns. No code is written yet (see status.md); this is the plan, not a shipped state. - **v2** is anything that needs backend integration or a major fix — a new query capability, a storage change, or new instrumentation. - **Exception, by decision of the requester:** the two backend items in Phase 0 (make a killed or @@ -28,8 +29,10 @@ every verdict. ## Scope v1 — build now -Delivered against today's endpoint, plus the two backend blockers. Every capability here is -verified "ready" in capability-review.md §4.2 unless a note says otherwise. +Planned against today's endpoint, plus the two backend blockers. No code is written yet (see +status.md). Every capability here is verified backend-*available* in capability-review.md §4.2 +unless a note says otherwise — "ready" describes what the endpoint can answer today, not shipped +UI, and B1 and B2 remain release blockers. | Priority | Feature | What it delivers | Value | Effort | Depends on | |---|---|---|---|---|---| @@ -43,9 +46,8 @@ verified "ready" in capability-review.md §4.2 unless a note says otherwise. | P1 | **B2 — Investigate cost / token coverage collapse** *(backend blocker)* | Restores dependable coverage for cost and the token split | High | M–? | — | | P1 | Cost per period (coverage-gated total) | Cost chart from `gen_ai.usage.cost`; renders only above the coverage threshold | High | M | B2 | | P1 | Tokens per period (total + coverage-gated split) | Tokens chart; the prompt/completion split shows only above threshold | Med | M | B2 (split only) | -| P1 | Filter by agent + runs-per-agent breakdown | Agents multi-select; breakdown unions `workflow_variant` + `application_variant` | High | M | — | -| P2 | Runs per harness + harness filter | Harness breakdown chart and filter | Med | S | — | -| P2 | Runs per configured model + model filter | Configured-model breakdown and filter (author's alias, labeled "configured model") | Med | S | — | +| P1 | Breakdown charts: runs per harness / configured model / agent | Three `categorical/single` breakdowns; agent unions `workflow_variant` + `application_variant` | High | M | — | +| P1 | Filters: agents, harness, configured model | Three server-side filters on root-span fields; no backend change (§4.2 items 10–12). Agents by `references`; the model filter narrows the configured alias | High | M | — | Notes: diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index a7dd1378ab..ba52d4b9b2 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -12,13 +12,17 @@ written. The workspace holds the plan; implementation has not started, and no br 1. Frontend-first scope, with two backend prerequisites (Phase 0 in plan.md). Charts: Runs, Latency, Cost (coverage-gated total), Tokens (coverage-gated split), and runs per harness / configured model / agent. No Costs prompt/completion split chart; `gen_ai.usage.cost` is a - total only. Plus the Agents / Harness / configured-Model filters and the time-range control. - The two Phase 0 items: make a killed or rejected query distinguishable from an empty one, and + total only. Plus the Agents / Harness / configured-Model filters and the time-range control; + harness and configured model are both breakdown charts and filters (decision 3). The two + Phase 0 items: make a killed or rejected query distinguishable from an empty one, and investigate the cost / token-split coverage collapse. 2. A net-new page at project scope, named Analytics. The default query aggregates every project agent; the Agents filter narrows the set. -3. An **agent** is an application/workflow artifact. The Agents multi-select is the only filter; - it lists the project's agents and narrows by `references`. +3. An **agent** is an application/workflow artifact. The Agents multi-select lists the project's + agents and narrows by `references`. v1 has three filters — Agents, Harness, and configured + Model — all server-side conditions on root-span fields, so all three filter today with no + backend change. The configured-Model filter narrows by the author's alias, not the model that + answered (that is the deferred resolved-Model view). 4. One agent invocation is a **run**. The Observability dashboard calls the same metric a "request"; the two may diverge until Observability is aligned in a later, separate change. 5. A **failed run** is a run whose root span `status_code` is `STATUS_CODE_ERROR` — a run-level @@ -71,7 +75,8 @@ the scope carries two backend prerequisites (Phase 0) rather than none. ## Deferred to a later backend change The Tools chart, the Models chart, the Models filter, and per-model cost. They split into two -sub-phases, because the two backend prerequisites are not co-equal gates: +sub-phases, because the two deferred-view enablers (span-focus wiring and a group-by dimension — +distinct from the two Phase 0 blockers) are not co-equal gates: - **5a, Tools and Models-share**: needs span-focus wiring only (thread `focus` into `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by, because a From ebf33fe85d4b4a8278604a7ffff84eb9514f1f00 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 4 Aug 2026 12:08:30 +0600 Subject: [PATCH 8/9] docs(analytics): update backend prerequisites in Phase 0 to clarify v1 and v2 scope --- docs/design/agent-analytics/plan.md | 8 +++-- docs/design/agent-analytics/scope.md | 42 +++++++++++++++------------ docs/design/agent-analytics/status.md | 21 +++++++------- 3 files changed, 40 insertions(+), 31 deletions(-) diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md index 9fe85a8766..6d1adc7fd8 100644 --- a/docs/design/agent-analytics/plan.md +++ b/docs/design/agent-analytics/plan.md @@ -8,10 +8,12 @@ File paths follow the existing observability feature so the new page reads as a it, not a new pattern. Every request and response shape referenced here is specified in data-contract.md and verified in capability-review.md. -## Phase 0: backend prerequisites +## Phase 0: backend fixes (v2 — does not gate v1) -Two backend items gate a trustworthy v1. Neither is frontend work; list and track them before -the page ships behind its flag. +Two backend items would make the v1 page more trustworthy, but by decision of the requester they +are **v2**, not v1 blockers (see scope.md). v1 ships against today's endpoint without them, +accepting the limits each describes below. Neither is frontend work; list and track them for the +v2 backend track. - **Make a killed or rejected query say so.** Today a statement-timeout, a rejected filter, a malformed window, and a genuinely empty window all return `{"count": 0, "buckets": []}` with diff --git a/docs/design/agent-analytics/scope.md b/docs/design/agent-analytics/scope.md index 0612bca116..a75320ef5b 100644 --- a/docs/design/agent-analytics/scope.md +++ b/docs/design/agent-analytics/scope.md @@ -11,10 +11,12 @@ every verdict. breakdowns. No code is written yet (see status.md); this is the plan, not a shipped state. - **v2** is anything that needs backend integration or a major fix — a new query capability, a storage change, or new instrumentation. -- **Exception, by decision of the requester:** the two backend items in Phase 0 (make a killed or - rejected query surface an error, and investigate the cost / token coverage collapse) stay in - **v1 as blockers**, not v2. v1 does not ship until they land, because without them the page - cannot tell "no data" from "the query died", and cost may read empty with no signal. +- **The two backend items (B1, B2) are v2, not v1 blockers.** By decision of the requester, making + a killed or rejected query surface an error, and investigating the cost / token coverage + collapse, are backend work and follow the base rule above — they move to v2. v1 ships against + today's endpoint without them, accepting the two known limits they would have fixed: the page + cannot tell "no data" from "the query died", and cost may read empty with no signal. Both are + tracked in the v2 table below. ## How to read the tables @@ -23,36 +25,38 @@ every verdict. - **Value** — how much the feature matters to a user: High / Med / Low. - **Effort** — how much work it is: `S` (a day or so), `M` (a few days), `L` (a week or more, or open-ended). -- **Backend blocker** rows are flagged; everything else in v1 is frontend-only. +- Every v1 row is frontend-only; the two backend items (B1, B2) are v2, in the v2 table. --- ## Scope v1 — build now -Planned against today's endpoint, plus the two backend blockers. No code is written yet (see -status.md). Every capability here is verified backend-*available* in capability-review.md §4.2 -unless a note says otherwise — "ready" describes what the endpoint can answer today, not shipped -UI, and B1 and B2 remain release blockers. +Planned against today's endpoint. No code is written yet (see status.md). Every capability here is +verified backend-*available* in capability-review.md §4.2 unless a note says otherwise — "ready" +describes what the endpoint can answer today, not shipped UI. B1 and B2 (the two backend fixes) +are now v2, so nothing in v1 blocks on them; v1 lives with the limits noted below. | Priority | Feature | What it delivers | Value | Effort | Depends on | |---|---|---|---|---|---| | P0 | Page shell, route, sidebar | The Analytics page exists, reachable, project-scoped | Enabler | S | — | | P0 | Data layer (`specs` field + fetch + mapper) | The page requests explicit specs and maps buckets to chart shape | Enabler | M | — | -| P0 | **B1 — Make killed / rejected queries surface an error** *(backend blocker)* | 504 / 4xx + per-metric `sample_count` instead of a silent empty 200 | High | M | — | -| P0 | Runs per period (success vs failed) | Stacked Runs chart; corrected `status_code is STATUS_CODE_ERROR` filter + second query | High | M | B1 | +| P0 | Runs per period (success vs failed) | Stacked Runs chart; corrected `status_code is STATUS_CODE_ERROR` filter + second query | High | M | — | | P0 | Latency per period (avg + p95 + min/max) | Latency chart with per-bucket p95 line; ships free on the numeric spec | High | S | — | | P0 | Time-range control (any window, 7-day default) | Reuses the observability `Sort` / `SortResult` | High | S | — | -| P0 | Four page states (data / no-data / unavailable / failed) | Honest empty vs failure vs coverage-gap, per card | High | M | B1 | -| P1 | **B2 — Investigate cost / token coverage collapse** *(backend blocker)* | Restores dependable coverage for cost and the token split | High | M–? | — | -| P1 | Cost per period (coverage-gated total) | Cost chart from `gen_ai.usage.cost`; renders only above the coverage threshold | High | M | B2 | -| P1 | Tokens per period (total + coverage-gated split) | Tokens chart; the prompt/completion split shows only above threshold | Med | M | B2 (split only) | +| P0 | Four page states (data / no-data / unavailable / failed) | Honest empty vs failure vs coverage-gap, per card | High | M | — | +| P1 | Cost per period (coverage-gated total) | Cost chart from `gen_ai.usage.cost`; renders only above the coverage threshold | High | M | — | +| P1 | Tokens per period (total + coverage-gated split) | Tokens chart; the prompt/completion split shows only above threshold | Med | M | — | | P1 | Breakdown charts: runs per harness / configured model / agent | Three `categorical/single` breakdowns; agent unions `workflow_variant` + `application_variant` | High | M | — | | P1 | Filters: agents, harness, configured model | Three server-side filters on root-span fields; no backend change (§4.2 items 10–12). Agents by `references`; the model filter narrows the configured alias | High | M | — | Notes: +- **The two backend fixes (B1, B2) are v2**, and v1 lives with their absence. The four page states + are all built, but until B1 the "failed / query-died" state cannot be told apart from "no data", + so a killed query reads as an honest empty. The Cost chart stays coverage-gated and may read + unavailable until B2. Both fixes and their limits are in the v2 table. - **Cost** is "blocked on coverage" in the review (§4.2 item 4). v1 ships the chart - coverage-gated; B2 is what makes it dependable. + coverage-gated; B2 (now v2) is what makes it dependable. - **Configured model** is a proxy for the model that answered — the author's alias. It is labeled honestly; the real answered model is v2 (needs `focus=span`). - **Not on the v1 page at all:** tool usage, resolved-model usage, per-model cost, cache tokens, @@ -68,6 +72,8 @@ and its capability-review.md §8.4 item. | Priority | Feature | What it delivers | Value | Effort | Backend work it needs | |---|---|---|---|---|---| | P0 | Typed analytics contract | Named metrics / dimensions / aggregations, validated and capped; the foundation the rest build on | High | M | Replace the arbitrary-path protocol (§8.4 item 2) | +| P0 | B1 — Make killed / rejected queries surface an error | 504 / 4xx + per-metric `sample_count` instead of a silent empty 200; lets the page tell "no data" from "the query died", and unblocks span-focus | High | M | Typed timeout / filter error in core + router `HTTPException` + per-metric `sample_count` (§8.4 item 1) | +| P1 | B2 — Investigate cost / token coverage collapse | Restores dependable coverage for cost and the token split | High | M–? | Diagnose the mid-July collapse of `gen_ai.usage.cost` and the token split (§4.4 item 4) | | P1 | Stop reading JSONB on the chart path | Hot columns or a per-run facts table; a permanent latency win and a home for invoked facts | High | M–L | Ingest + storage change (§8.4 item 4) | | P1 | Tool usage per period (which tools ran) | Real tool-call counts, and a path to per-tool error rate | High | M | `focus=span` wiring + 3 guards + non-root index (§8.4 item 3) | | P1 | Resolved model usage per period | The model that actually answered, distinct from the configured alias | High | M | `focus=span` (shares the wiring) | @@ -91,5 +97,5 @@ item 13). available) that seeds v1 versus v2. - **capability-review.md §8.3** — the v1 beta scope. - **capability-review.md §8.4** — the v2 backend work, in dependency order. -- **plan.md** — the v1 build, phase by phase (Phase 0 = the two blockers; Phases 1–4 = the page; - Phase 5 = the first slice of v2). +- **plan.md** — the v1 build, phase by phase (Phases 1–4 = the page). Phase 0 (the two backend + fixes, B1 and B2) and Phase 5 are v2 backend work and do not gate the v1 page. diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md index ba52d4b9b2..f556b052dd 100644 --- a/docs/design/agent-analytics/status.md +++ b/docs/design/agent-analytics/status.md @@ -9,13 +9,14 @@ written. The workspace holds the plan; implementation has not started, and no br ## Locked decisions -1. Frontend-first scope, with two backend prerequisites (Phase 0 in plan.md). Charts: Runs, - Latency, Cost (coverage-gated total), Tokens (coverage-gated split), and runs per harness / - configured model / agent. No Costs prompt/completion split chart; `gen_ai.usage.cost` is a - total only. Plus the Agents / Harness / configured-Model filters and the time-range control; - harness and configured model are both breakdown charts and filters (decision 3). The two - Phase 0 items: make a killed or rejected query distinguishable from an empty one, and - investigate the cost / token-split coverage collapse. +1. Frontend-first scope. The two backend items (Phase 0 in plan.md) are **v2, not v1 blockers** + (requester decision, see scope.md); v1 ships against today's endpoint without them. Charts: + Runs, Latency, Cost (coverage-gated total), Tokens (coverage-gated split), and runs per + harness / configured model / agent. No Costs prompt/completion split chart; `gen_ai.usage.cost` + is a total only. Plus the Agents / Harness / configured-Model filters and the time-range + control; harness and configured model are both breakdown charts and filters (decision 3). The + two deferred Phase 0 items: make a killed or rejected query distinguishable from an empty one, + and investigate the cost / token-split coverage collapse. 2. A net-new page at project scope, named Analytics. The default query aggregates every project agent; the Agents filter narrows the set. 3. An **agent** is an application/workflow artifact. The Agents multi-select lists the project's @@ -42,8 +43,8 @@ atoms). This feature reuses that spine. The data-layer work: pass explicit metri total cost (`gen_ai.usage.cost`) and the token split, read the duration min/max/p95 the current mapper drops, read the category `freq` breakdowns, and add the status-filtered failed-run query. The endpoint returns the latency, run-count, and breakdown fields directly. Cost and the token -split stay coverage-gated, because their populated paths collapsed in mid-July — which is why -the scope carries two backend prerequisites (Phase 0) rather than none. +split stay coverage-gated, because their populated paths collapsed in mid-July — which is why the +scope carries two backend items (Phase 0), now deferred to v2 rather than gating v1. ## Resolved by code verification @@ -76,7 +77,7 @@ the scope carries two backend prerequisites (Phase 0) rather than none. The Tools chart, the Models chart, the Models filter, and per-model cost. They split into two sub-phases, because the two deferred-view enablers (span-focus wiring and a group-by dimension — -distinct from the two Phase 0 blockers) are not co-equal gates: +distinct from the two Phase 0 backend items) are not co-equal gates: - **5a, Tools and Models-share**: needs span-focus wiring only (thread `focus` into `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by, because a From 5006c0cf7fbe112409ea87ce588c1f7a3fc38dc7 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 4 Aug 2026 17:12:53 +0600 Subject: [PATCH 9/9] Refactor agent analytics documentation: update research, scope, and status sections - Revised the research section to clarify the analytics engine's functionality and the data path from endpoint to dashboard. - Updated the scope document to delineate features for v1 and v2, emphasizing backend dependencies and capabilities. - Removed the status document as its content is now integrated into the scope and research sections, reflecting the current state of development and decisions made. --- docs/design/agent-analytics/README.md | 107 +++---- docs/design/agent-analytics/context.md | 116 ------- docs/design/agent-analytics/data-contract.md | 223 ------------- docs/design/agent-analytics/plan.md | 237 -------------- docs/design/agent-analytics/research.md | 311 +++++++++---------- docs/design/agent-analytics/scope.md | 163 +++++----- docs/design/agent-analytics/status.md | 102 ------ 7 files changed, 272 insertions(+), 987 deletions(-) delete mode 100644 docs/design/agent-analytics/context.md delete mode 100644 docs/design/agent-analytics/data-contract.md delete mode 100644 docs/design/agent-analytics/plan.md delete mode 100644 docs/design/agent-analytics/status.md diff --git a/docs/design/agent-analytics/README.md b/docs/design/agent-analytics/README.md index 694b921ee9..8a69c5af52 100644 --- a/docs/design/agent-analytics/README.md +++ b/docs/design/agent-analytics/README.md @@ -1,59 +1,60 @@ # Agent Analytics page -A new project-scoped **Analytics** page for the Agenta web app. It charts how a project's -agents perform over a chosen window: run volume, success and failure, latency, cost, and -tokens. The page reads the existing spec-driven analytics endpoint -(`POST /spans/analytics/query`) and reuses the frontend data-layer atoms already built for it, -so this work adds a page, not a data layer. - -## Reading order - -1. **context.md** — why the page exists, what a user sees today, goals and non-goals, and the - locked scope decisions. -2. **scope.md** — the v1 / v2 split: what ships now versus what waits for backend work, each as - a ranked table. Read it to know what is in scope before the how. -3. **research.md** — the code this feature reuses, path by path: the analytics fetch layer, the - response-to-dashboard mapper, the sidebar and routing, and the charting library. Read it - before you propose any new file. -4. **data-contract.md** — the request the page sends (window, filter, metric specs) and the - response fields the new mapper reads, including the fields the current mapper drops. -5. **plan.md** — the build in phases, with the file list per phase and the seam where the - deferred model and tool views drop in once the backend supports them. -6. **status.md** — current state, open questions, and decisions. This is the source of truth - for progress; update it as work lands. -7. **capability-review.md** — an independent, evidence-backed review of what the analytics - backend can answer today, written after the documents above. It tests each wanted capability - against the running code with live queries, times those queries, and proposes a v1 and a v2. - It contradicts the earlier documents in several places, so read it before you build. Where - the two disagree, the review carries the evidence. +A project-scoped **Analytics** page for the Agenta web app. It charts how a project's agents +perform over a chosen window: run volume, success and failure, latency, cost, and tokens. The +page reads the existing analytics endpoint (`POST /spans/analytics/query`) and reuses the +frontend data layer already built for it, so this work adds a page, not a data layer. -## Glossary +## State + +**Planned, not built.** No code is written and no branch exists. The scope is designed against +today's endpoint; most of the open work — and every wanted-but-missing view — is v2 backend work. + +## The docs + +- **[scope.md](scope.md)** — the scope line: what the backend can answer **today** (with honest + labels) versus what **needs v2 work** (blockers first, then deferred features, each ranked with + the backend change it needs). Start here. Scan its tables in a few seconds. +- **[research.md](research.md)** — ground truth: how the analytics engine actually works + (root-spans-only, the dead `focus` field, specs and percentiles, the silent-failure modes), + what live queries prove, and the code the page reuses. Read it before proposing any change. +- **[capability-review.md](capability-review.md)** — the evidence base: every verdict tested + against the running code with live queries on two stacks. Where a summary and the review + disagree, the review carries the evidence. Its line citations are frozen at commit + `31c0781d42`; the load-bearing ones are re-verified and corrected in research.md. -Terms used across these documents, defined once. This section is the workspace glossary; a -separate `CONTEXT.md` cannot sit beside `context.md`, because the filesystem is case-insensitive -and the two names collide. +## The short version + +**Today** the endpoint answers: runs (success vs failed), latency (avg + p95 + min/max), runs per +agent, and filters on agent / harness / configured model. Cost and the token split are +expressible but coverage-gated. Breakdowns by harness and configured model are run-counts only, +so the useful version — cost and tokens per harness/model — is v2. It reads root spans only, and +it fails silently — a killed query looks like an empty one. + +**v2** is where the open problems live: two defects (silent query failures, and a mid-July +collapse in cost/token coverage), one contract decision, and the wall behind every missing view. +Some need only a group-by dimension (cost/tokens per harness and configured model, which are +root-span attributes); the rest wait on a `focus=span` fix because the endpoint cannot read child +spans — tool usage, the model that actually answered, per-resolved-model cost, and cache tokens. + +## Glossary -- **Agent**: a configured AI agent in the project — the top-level thing a user builds, runs, and - analyzes. It is the unit the page aggregates over and the unit the Agents filter narrows to. - This page's copy never calls it application, app, workflow, or variant. +- **Agent**: a configured AI agent in the project — the unit the page aggregates over. Never + called application, app, workflow, or variant in the page copy. - **Run**: one agent invocation. On the backend it is one root span (a span with no parent). Run - count per bucket equals the count of the `ag.type.trace` metric. This page says "run"; the - Observability dashboard says "request" for the same metric, and the two may diverge until - Observability is aligned later. -- **Span**: one unit of work inside a run — a model call, a tool call, or the agent step itself. - Model name and tool name live on child spans, not on the root span. -- **Root span**: the top span of a run. Today's analytics endpoint reads root spans only. -- **Failed run**: a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome - (there is no `STATUS_CODE_OK` on root spans, so success is the complement). It is counted from - the `status_code` column, not from the errors metric. -- **Error**: any errored step inside a run. A run can contain errors and still succeed, so an - error count is not a failed-run count. -- **Bucket**: one time slice of the chart x-axis — one day, or one hour. The endpoint returns - one metrics object per bucket. -- **Metric spec**: a request instruction of the form `{type, path}` that tells the endpoint - which JSON path on the span to summarize, and how. The endpoint has no fixed metric list; it - summarizes whatever path a spec names. -- **Focus**: a request field that selects whether the query aggregates over root spans (`trace`) - or all spans (`span`). Only `trace` works today; see context.md. -- **Project scope**: the web app always has exactly one project in context. This page lives at - that level and, by default, aggregates every agent in the project. + count is the count of the `ag.type.trace` metric. The Observability page calls the same metric + a "request"; the two may diverge until Observability is aligned later. +- **Root span** / **child span**: the top span of a run versus every other span in it. Today's + endpoint reads root spans only. Model name and tool name live on child spans. +- **Configured / resolved / invoked**: what the author wrote down (the model alias, the tool + list) / what the system turned it into at run time / what the run actually did. Today's page can + chart only *configured* values; *resolved* and *invoked* are v2. +- **Failed run**: a run whose root span `status_code` is `STATUS_CODE_ERROR`. There is no + `STATUS_CODE_OK` on root spans, so success is the complement. It is a run-level outcome, not a + count of errored steps. +- **Bucket**: one time slice of the x-axis — a fixed-width period offset from the window start, + not a calendar day. Calendar months are not expressible. +- **Metric spec**: a `{type, path}` instruction naming a JSON path to summarize. The endpoint + has no fixed metric list; it summarizes whatever path a spec names. +- **Coverage-gated**: a metric that is expressible but often holds no data, so the chart renders + only when enough runs carry the value and otherwise says so, rather than showing a zero. diff --git a/docs/design/agent-analytics/context.md b/docs/design/agent-analytics/context.md deleted file mode 100644 index 0eb4238ef8..0000000000 --- a/docs/design/agent-analytics/context.md +++ /dev/null @@ -1,116 +0,0 @@ -# Context - -## What a user sees today - -Today the web app shows no agent performance across a project. A team running several agents -cannot open one screen and read how many runs happened, how many failed, how fast they ran, and -what they cost — narrowed to the agents they care about, over a window they choose. - -The frontend already carries the plumbing to answer those questions. The atoms, the fetch -function, and the response mapper that talk to the analytics endpoint all exist and are in use. -This feature reuses that plumbing; it does not rebuild the data layer. - -## What this feature adds - -A new page named Analytics, reachable from the project sidebar and scoped to the whole project. -It shows: - -- A header: the page title, a one-line description, a time-range control that accepts any - window, and a Filters popover with three multi-selects that narrow every query: Agents (by - `references`), Harness, and configured Model. All three are server-side filter conditions on - root-span fields, so changing any of them refetches (see data-contract.md for the condition - each one adds). The time-range control reuses the observability windowing (the `Sort` control and its - `SortResult`), so it takes the standard presets and a custom start-and-end range, not a fixed - list of options. It opens on the last 7 days. -- Charts in a grid, each with hover tooltips and a legend whose entries toggle series on and - off: - - Runs: stacked bars of successful and failed runs per bucket. - - Latency: bars of average latency per bucket, with a per-bucket p95 line; the tooltip shows - average, p95, min, and max. - - Cost: total cost per bucket, coverage-gated. When cost coverage is low it reads "not - available for this window" instead of a zero. - - Tokens: total tokens per bucket, with a coverage label. The prompt/completion split renders - as stacked bars only when its coverage gate passes. - - Runs per harness, runs per configured model, and runs per agent: category breakdowns, one - `categorical/single` spec each. "Configured model" is the model alias the agent's author - set on the root span; it is **not** the model that actually answered the run. The answered - model lives on child spans and is the deferred model-usage view below. This chart must be - labelled "configured model" and never implemented as the deferred model-usage view. - - There is no Costs prompt/completion split chart. The split paths hold no data on agent runs, - so the Cost chart shows the total from `gen_ai.usage.cost` only. See data-contract.md and - capability-review.md. - -## Locked scope decisions - -These decisions are settled and drive the plan. Do not reopen them without the requester. - -1. Frontend-first, with two backend prerequisites. Build the charts above, the Agents / Harness / - configured-Model filters, and the time-range control against today's endpoint. Harness and - configured model appear both as breakdown charts and as filters; all three filters read - root-span attributes, so none of them needs a backend change. The endpoint - returns most of the fields these need directly, but two backend items gate a trustworthy - release, tracked as Phase 0 in plan.md: make a killed or rejected query distinguishable from a - genuinely empty one, and investigate the mid-July collapse in cost and token-split coverage. - The Cost chart and the token split stay coverage-gated until someone explains that collapse. - -2. A net-new page at project scope, named Analytics. It aggregates every agent in the project by - default. The Agents multi-select narrows the set, so the default query carries no single-app - reference filter. - -3. A failed run is a run whose root span status is `STATUS_CODE_ERROR`, a run-level outcome - (there is no `STATUS_CODE_OK` on root spans, so success is the complement). The Runs chart's - successful-and-failed split builds on that, not on a count of errored steps. data-contract.md - holds the definition and the query it needs. - -## Model usage and tool usage need backend work - -Two views fall out of scope for this plan, because the backend cannot serve them yet. This is an -engineering constraint, not an open design question. - -Tool usage (calls per tool, tool error rate) and model usage (runs per model, cost per model) -read fields that live on child spans: the tool name, the model name, the span type, and the -per-span status. Today's analytics endpoint reads root spans only. It accepts a `focus` field -that would widen the scan to all spans, but the field is inert, so those fields stay out of -reach. - -The two deferred-view enablers below — span-focus wiring and a group-by dimension — are not -co-equal gates; each unlocks a different thing. These are distinct from the two Phase 0 blockers -in the locked scope above (query-failure visibility and the coverage investigation); do not -conflate the two pairs. - -- **Span-focus wiring** unlocks both the tool view and the model-share view in their basic form. - `query.focus` reaches `dao.analytics()` but never reaches `build_base_cte`, which - unconditionally applies `WHERE parent_id IS NULL` - (`api/oss/src/dbs/postgres/tracing/utils.py`). Thread `focus` through so `focus = span` drops - that predicate, and the query reads child spans. A `categorical/single` spec already returns - per-value frequencies, so counting calls per tool or runs per model needs no group-by once - span focus works. - - One caveat must ship with the fix: under `focus = span`, the cumulative metric paths - double-count. `ag.metrics.*.cumulative` are rollups stored on the root span, so scanning all - spans sums the root total plus every child's. Cost and tokens must switch to the - `incremental` paths under span focus, or the figures inflate with no error. The fix needs a - guard that rejects or auto-maps a cumulative spec under `focus = span`. -- **A group-by dimension** unlocks only per-model cost and tokens. Crossing a numeric metric - (cost, tokens) with a categorical dimension (model name) is the one thing the frequency - reducers cannot do; the extract stage groups by `(timestamp, spec)` only, with no grouping - key. This is the harder, later change. Preferred shape: a per-query `group_by` dimension that - splits every spec by one path, leaves `MetricSpec` untouched, and confines the nested - `{group_value: stats}` output to one code path. - -## Goals - -- Reuse the analytics fetch layer, the response mapper, and the time-windowing control rather - than add a second path to the same endpoint. -- Deliver the layout and interactions with the repo's charting library and theme tokens, in both - light and dark themes. -- Leave a clean seam, so the deferred model and tool views land as an additive change once the - backend prerequisites ship — not a rewrite. - -## Non-goals - -- No new analytics capability under `api/` for the page itself. The two Phase 0 items harden - existing behavior; they do not add a chart's data path. -- No change to the existing Observability page. -- No real-time streaming. The page fetches per time-range selection and caches like the existing - dashboard atoms. diff --git a/docs/design/agent-analytics/data-contract.md b/docs/design/agent-analytics/data-contract.md deleted file mode 100644 index 45bda915c2..0000000000 --- a/docs/design/agent-analytics/data-contract.md +++ /dev/null @@ -1,223 +0,0 @@ -# Data contract: request the page sends, response fields it reads - -This page talks to one endpoint, `POST /spans/analytics/query`, through the existing -`fetchSpansAnalytics`. Nothing here changes the endpoint. It documents the exact request the -page builds and the response fields the new mapper reads, and it flags the one frontend -extension: passing explicit metric specs. - -Every request and response shape below comes from the capability review -(`capability-review.md`), which verified each against live calls on two stacks. Where a value -still needs confirming against a live response during the build, the text says so. - -## Request - -The request carries four kinds of field, grouped by what each field is, not by which chart it -feeds: - -- **Routing** (which tenant and window): `projectId`, `oldest`, `newest`. `projectId` comes - from `projectIdAtom`. `oldest` and `newest` are ISO bounds taken from the selected - `SortResult`, the same range object the observability windowing uses: a standard preset - resolves to a start with `newest` omitted (meaning "now"), and a custom range supplies both - `oldest` and `newest`. Any window is valid; there is no fixed set of options. Align `oldest` - to the viewer's local midnight, because buckets are fixed-width offsets from `oldest`, not - calendar days (see "Response quirks" below). -- **Policy** (how to slice and aggregate): `focus = "trace"` and `interval` (bucket size in - minutes) from `calculateIntervalFromDuration`. The endpoint ignores `focus` entirely and - always reads root spans, so `focus = "trace"` is the honest label rather than a switch; - never rely on the echoed `focus` to confirm behaviour. -- **Data selection** (what to measure): `specs`, a list of metric specs naming the JSON - paths to summarize. See below. -- **Data filter** (which spans qualify): `filter`, a `{conditions: [...]}` object. Every - query carries a base `{field: "trace_type", operator: "is", value: "invocation"}` condition - so annotation traces (evaluator and human-annotation runs) do not inflate the run count and - the latency, cost, and token numbers. At project scope with no agent selected, add nothing - else so the query spans the whole project. For selected agents, add `references` conditions - on the chosen agent ids. For a selected **harness**, add a condition on - `attributes.ag.data.parameters.agent.harness.kind`; for a selected **configured model**, add a - condition on `attributes.ag.data.parameters.agent.llm.model`. Both are root-span attributes and - filter today with no backend change (capability-review.md §4.2 items 11–12); the model filter - narrows by the configured alias, not the model that answered. Combine multiple values within one - filter the same way the agent filter does (the encoding to confirm in Phase 2). All three - filters — Agents, Harness, configured Model — are server-side, so each belongs in the query key - and changing any of them refetches. The `trace_type` value is the unprefixed literal `invocation` - (verified against the `TraceType` enum, `api/oss/src/core/otel/dtos.py`); unlike - `status_code`, it is not prefixed, and the backend validates the value against the enum, so a - wrong literal returns an empty 200. The one encoding still to confirm against the filter - builder in Phase 2 is whether multiple agent ids go as a single - `{operator: "in", value: [{id: a}, {id: b}]}` or one condition per agent combined with `or`. - A fallback for the run count that avoids the `trace_type` filter entirely: read the `freq` - array of the `attributes.ag.type.trace` categorical spec, which already carries the - invocation-versus-annotation split. - - A filter typo does not error. An unknown field is logged and dropped, which **widens** the - result rather than narrowing it, and an invalid operator or value returns HTTP 200 with an - empty result. Verify every filter the page sends once against a known-good window during - development; there is no runtime signal. - -### The `specs` extension - -`fetchSpansAnalytics` omits `specs` today, so the backend applies its default set. The -defaults give totals for cost, tokens, duration, errors, and the trace and span type counts, -but they read the canonical `ag.metrics.costs.cumulative.*` cost paths, which hold no data on -agent runs (see the cost note below). The page must pass an explicit `specs` list. Add an -optional `specs` field to `SpansAnalyticsParams` and serialize it exactly as `filter` is -serialized. The Fern request type (`QuerySpansAnalyticsRequest`) already declares `specs?: string`, -and `fetchSpansAnalytics` already does `request.filter = JSON.stringify(filter)`, so the change is -`request.specs = JSON.stringify(specs)` on one line beside it — no client regeneration needed. - -The specs the page sends, by path and type: - -| Purpose | path | type | fields read | -| --- | --- | --- | --- | -| Run count | `attributes.ag.type.trace` | categorical/single | `count` (or the `freq` array) | -| Latency | `attributes.ag.metrics.duration.cumulative` | numeric/continuous | `count`, `sum`, `min`, `max`, `pcts.p95` | -| Total cost (coverage-gated) | `attributes.gen_ai.usage.cost` | numeric/continuous | `sum`, `count` | -| Total tokens | `attributes.ag.metrics.tokens.cumulative.total` | numeric/continuous | `sum`, `count` | -| Prompt tokens (coverage-gated split) | `attributes.ag.metrics.tokens.cumulative.prompt` | numeric/continuous | `sum` | -| Completion tokens (coverage-gated split) | `attributes.ag.metrics.tokens.cumulative.completion` | numeric/continuous | `sum` | - -The category breakdown charts (runs per harness, per configured model, per agent) each send -one extra `categorical/single` spec and read its `freq` array: - -| Purpose | path | type | fields read | -| --- | --- | --- | --- | -| Runs per harness | `attributes.ag.data.parameters.agent.harness.kind` | categorical/single | `freq` | -| Runs per configured model | `attributes.ag.data.parameters.agent.llm.model` | categorical/single | `freq` | -| Runs per agent | `attributes.ag.references.workflow_variant.id` and `attributes.ag.references.application_variant.id` | categorical/single | `freq` (union both families) | - -Notes on the specs: - -- **The `type` strings are verified against the backend.** `MetricType` - (`api/oss/src/core/tracing/dtos.py`) has only `numeric/continuous` and `numeric/discrete` - (there is no plain `numeric`). Use `numeric/continuous` for every number metric; a bare - `numeric` is silently dropped. -- **p95 is nested, not a flat field.** The numeric/continuous reducer emits percentiles under - a `pcts` object, so the value is at `metrics[path].pcts.p95`. `count`, `sum`, `min`, and - `max` are flat siblings and read directly; only the percentiles sit one level down. All 27 - percentile levels ship on every numeric/continuous spec, so the per-bucket p95 line needs no - backend work. Confirm the nested shape against one live response in Phase 2. -- **Cost has no prompt/completion split, and its one working path is coverage-gated.** The - canonical `ag.metrics.costs.cumulative.total`, `.prompt`, and `.completion` paths hold no - data on agent root spans (the cost roll-up never crosses the run's OTLP batch boundary to - reach the root span). The only populated cost path is `attributes.gen_ai.usage.cost`, the - harness's own reported run total, and its coverage collapsed to near zero in mid-July on - both measured stacks for a cause nobody has established yet. The Cost chart therefore renders - the total only, and only when coverage clears a threshold (see "Coverage gating" below). -- **Total tokens works with a coverage label; the split moves with cost.** The - prompt/completion token split shares the same mid-July collapse as cost: it is real where - the pipeline works and a flat zero band where it does not, which reads as data and is worse - than an empty chart. Coverage-gate the split the same way as cost. - -### Failed runs come from a second, filtered query - -A failed run is a run whose root span `status_code` is `STATUS_CODE_ERROR`. `status_code` is a -table column, and metric specs read only the `attributes` JSON (`build_extract_cte` extracts -`attributes #> path`), so no spec can target it. Instead, the page runs a second analytics -query for the same window with an added filter condition on `status_code` and reads the run -count: - -- Failed-run query filter: the base `trace_type` condition and the agent conditions (if any), - plus `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}`. `status_code` is - a first-class filter field (`api/oss/src/core/tracing/utils/filtering.py`). The operator - must be `is` and the value must be the full enum literal `STATUS_CODE_ERROR`. An `eq` - operator or a bare `ERROR` value both raise inside the backend and come back as HTTP 200 - with an empty result, which would silently report zero failures forever. -- There is **no** `STATUS_CODE_OK` on root spans; a clean run's status is `STATUS_CODE_UNSET`. - So success is the complement of failure, never a positive `STATUS_CODE_OK` filter. -- It needs only the run-count spec (`attributes.ag.type.trace`), so it is a cheap query. -- Per bucket: `failed` = the filtered run count; `success` = the unfiltered run count minus - it. - -**Blind spot to state in the UI.** Root-span status does not see failures inside a run that -recovered a clean root. On the measured data that is about 1.2% of runs. The v1 definition of -a failed run is "the root span errored"; say so in a tooltip. Catching in-run failures needs -`focus = "span"`, which does not work today (see the deferred boundary). - -### The queries per window - -The six core metric specs (run count, latency, cost, total tokens, and the two token-split -specs) plus the four category specs (harness, configured model, and the two agent-reference -families) come to **ten specs** — more than one request should carry. So the page does not put -them in a single call. It fans out into small bucketed calls, each at or under about eight -specs, in parallel with a per-call error state: - -1. **Bucketed core metrics** (six specs): run count, latency, cost, total tokens, and the - prompt/completion split. Drives the Runs, Latency, Cost, and Tokens charts. This is the - "six-spec shape" the timings below were measured on. -2. **Bucketed category breakdowns** (four specs): the harness, configured-model, and two - agent-reference categorical specs. Drives the three breakdown charts. -3. **Bucketed status-filtered** (one spec): the per-bucket failed run count for the stacked Runs - chart. - -Every chart reads its data per bucket. On the measured data a 7-day window costs about 0.26 s -and a 30-day window about 1.7 s for the six-spec core call, so 7 days is the default and 30 or -90 days is an explicit user choice with a loading state; the smaller category and status calls -run alongside it. - -## Response fields the mapper reads - -The response is `{buckets: [{timestamp, interval, metrics: {: {: value}}}]}`. - -**Join the parallel calls by `timestamp`, not by array position.** Each call omits its empty -buckets independently (see "Response quirks"), so the unfiltered, category, and status-filtered -responses can have different bucket counts and different offsets — a bucket with successes but no -failures is present in the unfiltered array and absent from the status-filtered one. The mapper -keys each response's buckets by `timestamp`, builds the union x-axis, and fills a missing bucket -as zero for that call. It must also read `buckets[].interval` back and treat the calls as -comparable only when their effective `interval` agrees; if the backend coarsened one call's -interval differently, surface a mismatch state rather than aligning mismatched buckets. Only -under this timestamp join do the per-bucket claims below hold. The new mapper then produces, per -bucket: - -- `failed` = the `type.trace` count from the status-filtered query for that `timestamp` (runs - whose root span is `STATUS_CODE_ERROR`), or zero if the status-filtered call omitted the - bucket. This is a true failed-run count, never larger than the total for the same bucket. -- `success` = the unfiltered `type.trace` count for that `timestamp` minus `failed`. Because - both counts come from the same bucket after the timestamp join, `success` cannot go negative - and no flooring is needed; without the join (positional pairing across omitted buckets) it - could. This departs from the existing observability mapper, which subtracts the - `errors.cumulative` sum. That sum counts errored steps, not failed runs, and can exceed the - run count, so this page uses the run-level status instead. -- `latencyAvg` = duration `sum` / duration `count`, in milliseconds. -- `latencyMin`, `latencyMax` = the flat `min` / `max` duration fields. -- `latencyP95` = the **nested** `metrics[durationPath].pcts.p95`, per bucket. `metricField`'s - flat one-level read does not reach it, so the mapper needs a small `pcts` accessor. This drives - the per-bucket p95 line and the tooltip. -- `costTotal` = the `gen_ai.usage.cost` `sum`. Rendered only when its coverage gate passes. -- `tokensTotal` = the `tokens.cumulative.total` `sum`. -- `tokensPrompt`, `tokensCompletion` = the two token split sums. Rendered only when their - coverage gate passes; below it they are suppressed, not shown as zero. - -### Coverage gating - -Cost and the token split can be perfectly expressible and still hold no data, because coverage -is zero on the window. For each gated metric, compare its spec `count` against the run count -for the same window. Render the chart only when coverage clears a threshold; below it, show -"cost data is not available for this window" (or the token equivalent) rather than a zero. A -zero reads as a real measurement and is worse than an explicit unavailable state. Total tokens -is shippable with a coverage label rather than a hard gate, because its coverage is partial -rather than collapsed. - -### Response quirks the mapper must handle - -- **Empty buckets are omitted.** The mapper must build its own x-axis, or gaps read as missing - days rather than as zero days. -- **The top-level `count` is the number of buckets, not the number of runs.** Read run counts - from the per-bucket `type.trace` metric. -- **The requested `interval` may have been coarsened silently** above 1024 buckets. Only - `buckets[].interval` reports what actually ran. If the UI lets the user pick a period, read - the effective interval back and label the chart with it. -- **A wrong filter renders as zero, not as an error**, as noted under the request filter. -- **Buckets are 24-hour periods aligned to `oldest`**, not calendar days, and `date_bin` steps - by a fixed duration, so buckets drift off local midnight across a daylight-saving transition. - Calendar months are not expressible at all. Label the axis as 24-hour periods; do not offer - a month period. - -## Boundary for the deferred charts - -The deferred Tools and resolved-Models charts need `focus = "span"`, which is accepted, echoed, -and ignored today, so it always reads root spans. Actual tool names live at -`attributes.ag.meta.tool.name` and resolved model ids at `attributes.ag.meta.request.model`, -both on child spans only. Per-model cost additionally needs a group-by dimension the endpoint -does not have. Keep the request builder able to take a different `focus` and extra specs -without reshaping, so the follow-up change adds a second query rather than rewriting this one. diff --git a/docs/design/agent-analytics/plan.md b/docs/design/agent-analytics/plan.md deleted file mode 100644 index 6d1adc7fd8..0000000000 --- a/docs/design/agent-analytics/plan.md +++ /dev/null @@ -1,237 +0,0 @@ -# Plan - -Build in four phases. Each phase is independently reviewable and leaves the app working. The -deferred Tools and resolved-Models charts are a fifth phase that lands after a backend change; -the first four phases finish the locked scope. - -File paths follow the existing observability feature so the new page reads as a sibling of -it, not a new pattern. Every request and response shape referenced here is specified in -data-contract.md and verified in capability-review.md. - -## Phase 0: backend fixes (v2 — does not gate v1) - -Two backend items would make the v1 page more trustworthy, but by decision of the requester they -are **v2**, not v1 blockers (see scope.md). v1 ships against today's endpoint without them, -accepting the limits each describes below. Neither is frontend work; list and track them for the -v2 backend track. - -- **Make a killed or rejected query say so.** Today a statement-timeout, a rejected filter, a - malformed window, and a genuinely empty window all return `{"count": 0, "buckets": []}` with - HTTP 200. The page cannot tell "no data" from "the query died", and the cost coverage gate - in Phase 4 depends on that distinction. The fix crosses two layers: raise a typed - timeout/filtering error in core and translate it at the router into an `HTTPException` - (504 for timeout, 4xx for a bad filter or window), and add a per-metric `sample_count` to - the response. See capability-review.md section 8.4 item 1. -- **Investigate the cost and token-split coverage collapse.** `attributes.gen_ai.usage.cost` - and the `tokens.cumulative.prompt`/`.completion` split were populated on 70–95% of runs in - early July on both measured stacks and fell to roughly zero within a week, cause unknown. - Until this is understood, the Cost chart has nothing dependable to show. File it; the first - four diagnostic checks are in capability-review.md section 4.4 item 4. Both measured stacks - were local dev; whether production shows the same collapse is unverified. - -Done when: an empty result is distinguishable from a failure at the API surface, and the cost -coverage question has an owner and a tracking issue. - -## Phase 1: page shell, route, and navigation - -Goal: an empty Analytics page reachable from the sidebar, scoped to the project. - -- Add the route wrapper `web/oss/src/pages/w/[workspace_id]/p/[project_id]/analytics/index.tsx`, - a thin default export around a new page module, mirroring the observability page file. -- Add the page module `web/oss/src/components/pages/analytics/index.tsx` with the header - (title, description) and a placeholder body. -- Add the sidebar entry in - `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`: key `app-analytics-link`, - title `Analytics`, link `${projectURL}/analytics`, an icon from `@phosphor-icons/react`, - disabled when there is no project URL. Place it next to the Observability entry. -- Confirm the agents-list source for the later filter (app-management or workflow molecule - selectors) and note the chosen atom in status.md. - -Done when: the sidebar shows Analytics, the route renders the header, both themes look -correct. - -## Phase 2: data layer - -Goal: the page can fetch mapped analytics for the selected window. - -- Extend `SpansAnalyticsParams` in `web/packages/agenta-entities/src/trace/api/api.ts` with an - optional `specs` field, and in `fetchSpansAnalytics` add `request.specs = JSON.stringify(specs)` - on one line beside the existing `filter` serialization. The Fern `QuerySpansAnalyticsRequest` - already declares `specs?: string`, so no client regeneration is needed. Verify the entities - package still builds (`pnpm turbo run build --filter=@agenta/entities`). -- Add a new mapper next to the existing one in `web/oss/src/services/tracing/lib/` (do not - change `analyticsToGeneration`; the observability page depends on it). Per bucket, the new - mapper reads the duration min, max, sum, count, and nested `pcts.p95`; the total cost and - total token sums with their `count` for the coverage gate; the coverage-gated - prompt/completion token split; and the category `freq` arrays for the harness, - configured-model, and agent breakdowns. It combines the unfiltered and status-filtered run - counts into per-bucket success and failed, and returns the per-bucket shape in - data-contract.md. Reuse `metricField` and `calculateIntervalFromDuration`, and add a small - `pcts` accessor for p95 and a `freq` reader for the breakdowns. -- Add a fetch function in `web/oss/src/services/tracing/api/` that builds the base conditions - (the `trace_type is invocation` condition, plus the selected agents' `references` - conditions), passes the explicit `specs`, and calls `fetchSpansAnalytics`. Per the selected - window it fans out the bucketed calls in data-contract.md, each at or under ~8 specs, in - parallel with a per-call error state: a core-metrics call and a category-breakdown call for - the charts, and a status-filtered call for failed runs. The status-filtered call adds - `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` and reads the run count - as failed runs; it needs only the run-count spec. -- Validate three things against one live response while building this phase, because the - backend fails silently on all three: the nested `pcts.p95` shape reads correctly; the - `trace_type is invocation` and `status_code is STATUS_CODE_ERROR` filters both narrow rather - than returning an empty 200 (the literals are confirmed against the `TraceType` and - `OTelStatusCode` enums; the live check is that the runner actually stamps those statuses); and - the coverage of `gen_ai.usage.cost` and the token split on real traffic (they may be near zero - — this is the Phase 0 investigation surfacing). - -Done when: a temporary log or test shows correct per-bucket totals for a known window, and the -failed-run filter is confirmed to narrow the result. - -## Phase 3: state and page assembly - -Goal: the page is interactive; controls drive the data. - -- Add atoms under `web/oss/src/state/analytics/` following - `state/observability/dashboard.ts`: a time-range atom holding a `SortResult` (default: the - last 7 days), an agents-filter atom, a harness-filter atom, a configured-model-filter atom, - and a query atom for the selected window. The query atom drives the two bucketed calls in - data-contract.md (unfiltered and status-filtered). Key every atom on project id, time range, - and all three filters (agents, harness, configured model) — all three are server-side, so each - must be in the key to refetch. Set `staleTime` to one minute and - `refetchOnWindowFocus: false`. Do not reuse `observabilityDashboardTimeRangeAtom`; a second - consumer already shares it, so give this page its own atoms. -- Build the header controls: the time-range control and the Filters popover with three - multi-selects — Agents, Harness, and configured Model. Each adds the filter condition in - data-contract.md (agents by `references`, harness and configured model by their root-span - attribute paths); the configured-model filter narrows by the author's alias, not the model - that answered. Reuse the - observability time windowing (the `Sort` control and its `SortResult`) for the time range, - so any window, including a custom start-and-end range, works; do not build a fixed list of - range options. Offer day, week, and "whole window" as periods; do not offer a month period, - because calendar months are not expressible and a fixed 30-day stride is not a month. Align - `oldest` to the viewer's local midnight and read `buckets[].interval` back to label the axis - with the period that actually ran. Source the Agents options from the agents atom confirmed - in Phase 1; source the Harness and configured-Model options from the distinct values already - in the breakdown charts' `freq` arrays, so the filters need no extra call. -- Implement four explicit page states per card, following `AnalyticsDashboard.tsx` (antd - `Spin` for loading): data, no data in this window, metric unavailable (coverage below the - threshold), and request failed. The last two exist because a wrong filter or a killed query - renders as an empty success today; do not collapse them into a single "No data" empty state. - -Done when: changing the time range or any of the three filters (agents, harness, configured -model) refetches and the page reflects it, with all four states wired. - -## Phase 4: the charts - -Goal: the full locked-scope UI. Every chart on this page means what its title says and needs no -backend capability that does not exist. - -- Each chart is a card with a title, a one-line description, a recharts chart, and a toggleable - legend: - - Runs: stacked bars, successful and failed. - - Latency: bars of average with a per-bucket p95 line; tooltip shows average, p95, min, max. - - Cost: total cost per period, coverage-gated. It renders only when cost coverage clears the - threshold and otherwise shows an explicit "cost data not available for this window", never - a zero. There is no prompt/completion split; `gen_ai.usage.cost` is a total only. - - Tokens: total tokens per period, with a coverage label. Show the prompt/completion split - as stacked bars only when the split coverage gate passes; below it, show total only. - - Runs per harness: one `categorical/single` spec, per period. - - Runs per configured model: one `categorical/single` spec, per period, labelled "configured - model" (this is the author's alias, not the model that answered). - - Runs per agent: one `categorical/single` spec over the unioned `workflow_variant` and - `application_variant` reference families, per period. -- Lay the charts out in a responsive grid. All colors come from theme tokens and the theme - scale; verify both light and dark themes and hover, empty, unavailable, and loading states. - -Done when: the in-scope charts render correctly in both themes across all four states, and -`pnpm lint-fix` passes. - -## Phase 5 (deferred, after a backend change): Tools and resolved Models - -Not part of this plan's scope. Recorded so Phase 1 to 4 leave the right boundary. The backend -work splits into two prerequisites that are **not co-equal**; they unlock different views, so -Phase 5 itself splits into 5a and 5b. See context.md and capability-review.md for the verified -engine details. - -### Prerequisite 1: span-focus wiring (unlocks 5a) - -Thread `focus` through to the base query so `focus = "span"` reads child spans. `query.focus` -reaches `dao.analytics()` but is never passed to `build_base_cte`, which hardcodes -`WHERE parent_id IS NULL`; make that predicate conditional on `focus`. Today the endpoint -accepts, echoes, and ignores `focus`, so nothing on a root-span page can rely on it. - -- **Ship three guards with it**, or it produces wrong numbers: under `focus = "span"`, - cumulative metrics double-count because the rollup lives on every ancestor, so per-model - cost/tokens must use the `incremental` paths; a run count must dedupe because `ag.type.trace` - is stamped on every span in a trace; and non-root rows need an index (the current index is - partial on `parent_id IS NULL`, and the wider scan is a ~5x row fan-out). Do not ship - span-focus before Phase 0's failure-visibility fix, or a span-focus query on a busy project - hits the statement timeout and returns an empty 200. - -### Prerequisite 2: group-by dimension (unlocks only per-model cost, in 5b) - -Add a grouping dimension so a numeric metric (cost, tokens) splits by a categorical path -(model name) in one call. Prefer a **per-query** dimension over a per-spec `group_by`: it -matches "one view, one breakdown," leaves `MetricSpec` untouched, confines the nested -`{group_value: stats}` output shape to one code path, and makes a cardinality cap natural to -enforce. This is the harder change; defer it until 5a has shipped. - -### 5a: Tools and resolved-Models share (needs prerequisite 1 only) - -Once span-focus lands, these need **no** group-by, because a `categorical/single` spec already -returns per-value frequencies: - -- Tools horizontal-bar card: `categorical/single` on `attributes.ag.meta.tool.name`, - `focus = "span"`. Note that specs read the `attributes` JSON only, so the tool name must come - from the attribute path, not from the `span_name` column, which no spec can read. A per-tool - error rate needs the group-by dimension or a per-tool filtered call, because `status_code` is - a column and cannot be a spec either. -- Resolved-Models horizontal-bar card with a click-to-filter legend: `categorical/single` on - `attributes.ag.meta.request.model`, `focus = "span"`. This is the model that actually - answered, distinct from the configured-model chart in Phase 4. -- Resolved-model multi-select in the Filters popover. - -### 5b: per-model cost/tokens (needs prerequisites 1 and 2) - -Adds the numeric-by-model breakdown once the group-by dimension exists, using the `incremental` -cost/token paths under `focus = "span"`. - -### Frontend seam to leave now - -The request builder in Phase 2 takes `focus` and an arbitrary `specs` list without reshaping, -and the page grid can accept more chart cards. 5a and 5b add queries and cards without touching -the Phase 4 charts. - -Recommendation for the boundary: ship Phase 1 to 4. Do not mount empty Tools and Models cards -in this release; a visible "coming soon" card ages badly. Land the chart-card component as a -reusable shell so Phase 5 only supplies data and series config. - -## Testing and verification - -- Follow `docs/designs/testing/README.md`. Add unit tests for the new mapper (pure function: - buckets in, chart shape out), for the success/failed split, and for the coverage gate (a - metric below threshold is suppressed, not shown as zero). These need no live database. -- Verify the charts against one live project by running the local stack per the root - `AGENTS.md` local dev loop. Confirm the `trace_type` and `status_code` filters narrow rather - than widen, and confirm the p95 nested read. -- Run `pnpm lint-fix` in `web` before committing. Do not commit during the planning phase. - -## Risks and unknowns to resolve during the build - -- **Cost and token-split coverage** (Phase 0). Whether either has usable coverage on the target - data, and the cause of the mid-July collapse. The Cost chart and the token split depend on it. - Both measured stacks were local dev; production is unverified. -- **Silent-failure validation** (Phase 2). Confirm that the `trace_type is invocation` and - `status_code is STATUS_CODE_ERROR` filters narrow rather than return an empty 200 — the literals - are confirmed against the enums, so the open part is whether the runner stamps those statuses — - and confirm the nested `pcts.p95` read. -- The correct agents-list atom for the filter options (resolve in Phase 1). -- Whether the multi-agent reference filter uses `or` grouping or a single `in` with multiple - values in this dialect (resolve in Phase 2 against the existing filter builder). -- Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The - failed-run count depends on it; validate on live traffic in Phase 2. The root-status definition - also misses in-run failures that recovered a clean root (~1.2% of runs on the measured data); - state the definition in a tooltip. -- Performance at scale: the 30-day window is the shape most at risk of crossing the 15-second - statement timeout as a project grows. Keep 7 days the default and treat longer windows as an - explicit choice with a loading state. diff --git a/docs/design/agent-analytics/research.md b/docs/design/agent-analytics/research.md index ac5982d1f1..9ca70f9862 100644 --- a/docs/design/agent-analytics/research.md +++ b/docs/design/agent-analytics/research.md @@ -1,163 +1,148 @@ -# Research: what this feature reuses - -Every path below was read directly. The takeaway: the data path from endpoint to dashboard -already exists for the Observability page. This feature adds a new page on top of that spine and -reads a few response fields the current mapper drops. - -## The analytics fetch layer (reuse, extend by one field) - -`web/packages/agenta-entities/src/trace/api/api.ts` - -- `fetchSpansAnalytics(params)` calls `POST /spans/analytics/query` through the Fern client - (`getTracesClient().querySpansAnalytics`) and validates the response with - `analyticsResponseSchema`. It returns `null` on a non-2xx response or a shape mismatch. -- `SpansAnalyticsParams` today: `projectId`, `appId`, `focus` (default `trace`), `interval` - (bucket minutes), `oldest` / `newest` (ISO bounds), `filter` (a `{conditions: [...]}` object, - serialized to a JSON-string query param), `abortSignal`. -- It omits `specs` on purpose, so the backend applies its default set. To get the prompt and - completion split this page needs, add an optional `specs` field to `SpansAnalyticsParams` and - serialize it exactly as `filter` is serialized. The Fern `QuerySpansAnalyticsRequest` already - declares `specs?: string`, so the change is one field and one line — a frontend-only change. - -## The response-to-dashboard mapper (reuse pattern, new mapper for richer fields) - -`web/oss/src/services/tracing/lib/helpers.ts` - -- `analyticsToGeneration(analytics, range)` reduces the bucket list into - `GenerationDashboardData`. It reads these dotted metric paths, which match the backend's - default specs: - - `attributes.ag.metrics.costs.cumulative.total` (field `sum`) - - `attributes.ag.metrics.tokens.cumulative.total` (field `sum`) - - `attributes.ag.metrics.duration.cumulative` (fields `sum`, `count`) - - `attributes.ag.metrics.errors.cumulative` (field `sum`) - - `attributes.ag.type.trace` (field `count`) -- It derives run count from the `ag.type.trace` count, failures from the errors sum, success as - `total - failures`, and average latency as `durationSum / durationCount` in milliseconds. -- `metricField(metrics, path, field)` safely reads one **flat** numeric field (`count`, `sum`, - `min`, `max`). Reuse it for those. It reads one level deep, so it does **not** reach - percentiles; p95 lives at `metrics[path].pcts.p95`, one level further down. Add a small nested - accessor (or extend `metricField` with a two-key form) for the p95 read; do not assume p95 is a - flat sibling of `sum`. -- What it does not read, and this page needs: - - `gen_ai.usage.cost` (field `sum`, plus `count` for the coverage gate) for a coverage-gated - total-cost tile. The canonical `costs.cumulative.*` paths hold no data on agent root spans, - so there is no prompt/completion cost split to read; see data-contract.md. - - `tokens.cumulative.prompt` and `tokens.cumulative.completion` for the Tokens split, which is - coverage-gated because it shares the cost field's mid-July coverage collapse. - - `duration.cumulative` flat fields `min`, `max`, and the nested `pcts.p95` percentile for the - Latency tooltip and marker. - - Category `freq` arrays on `ag.data.parameters.agent.harness.kind`, - `ag.data.parameters.agent.llm.model`, and the agent `references` paths for the breakdown - charts. -- This page does not reuse the existing mapper's `errors.cumulative`-based failure count. A - failed run is a run whose root span status is `STATUS_CODE_ERROR` (there is no `STATUS_CODE_OK` - on root spans; success is the complement), which a metric spec cannot read, so the page gets - the failed-run count from a separate status-filtered query. See data-contract.md. -- `calculateIntervalFromDuration(durationMinutes)` picks a bucket size that keeps the bar count - reasonable and stays under the backend's ~1024-bucket limit. Reuse it directly for the - time-range-to-interval mapping. - -`web/oss/src/services/tracing/api/index.ts` - -- `fetchGenerationsDashboardData(appId, options)` builds the `conditions` array (it pushes a - `references in [{id: appId}]` condition when an app id is present), computes the interval, and - calls `fetchSpansAnalytics({focus: "trace", ...})`. This is the template for the new page's - fetch function. For project scope, drop the single-app reference condition and add reference - conditions only for the agents the user selects, plus the base `trace_type is invocation` - condition on every query. The new page also issues a second query per window, filtered by - `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}`, to count failed runs. - -## The dashboard state atoms (reuse pattern, new atoms for this page) - -`web/oss/src/state/observability/dashboard.ts` - -- `observabilityDashboardQueryAtom` is an `atomWithQuery` keyed on app id, project id, and the - time-range atom; it calls `fetchGenerationsDashboardData` with a `staleTime` of one minute and - `refetchOnWindowFocus: false`. -- `observabilityDashboardTimeRangeAtom` holds the selected range as a `SortResult`. -- `useObservabilityDashboard()` unwraps the loading and fetching flags. -- The new page follows this exact shape with its own atoms: a time-range atom, an agents-filter - atom, and a query atom for the selected window. That query drives the two bucketed calls, - unfiltered for the charts and status-filtered for failed runs. - -## Existing dashboard UI (reference, not reused directly) - -`web/oss/src/components/pages/observability/dashboard/` - -- `AnalyticsDashboard.tsx` renders `WidgetCard`s and `CustomAreaChart`s from the mapped data, - with a `Sort` time-range selector and antd `Spin` for loading. -- `CustomAreaChart.tsx` wraps recharts area charts. -- These render **area** charts of total figures. This page needs **stacked and grouped bar** - charts, a per-bucket **p95 line** on the latency chart, and a **horizontal bar** chart (for the - deferred Tools view), so it brings its own chart components rather than bend the area chart. It - still follows the same card-plus-chart composition and the same loading and empty-state - conventions. - -## Charting library - -`web/oss/package.json` depends on **recharts `^3.1.0`**. Existing recharts usage to copy style -and theming from: - -- `web/oss/src/components/pages/observability/dashboard/CustomAreaChart.tsx` -- `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/BarChart.tsx` -- `web/oss/src/components/EvalRunDetails/components/EvaluatorMetricsChart/HistogramChart.tsx` - -Build the page's charts with recharts (`BarChart` stacked and horizontal, `LineChart` for the -per-bucket p95 overlay on the latency chart). Do not port the reference implementation's -hand-rolled SVG chart code; it hardcodes hex colors and duplicates what recharts already gives. - -## Routing and sidebar - -- Pages live under `web/oss/src/pages/w/[workspace_id]/p/[project_id]/`. Existing folders: - `observability`, `evaluations`, `annotations`, `apps`, `settings`, and others. The - observability page file is a thin wrapper: - - ```tsx - import ObservabilityTabs from "@/oss/components/pages/observability" - const GlobalObservability = () => - export default () => - ``` - - Add `analytics/index.tsx` as the same kind of thin wrapper around a new - `components/pages/analytics` module. - -- The sidebar project items live in - `web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx`. The Observability entry is - the shape to copy: - - ```tsx - { - key: "app-observability-link", - title: "Observability", - link: `${projectURL}/observability`, - icon: , - disabled: !hasProjectURL, - } - ``` - - Add an `Analytics` entry with its own key, a `${projectURL}/analytics` link, and an icon from - `@phosphor-icons/react`. - -## The agents list for the filter - -The Agents multi-select needs the project's agents as options. The filter narrows the query by -pushing `references in [{id: }]` conditions — the same field the observability fetch -uses for a single app. Source the option list from the existing apps or workflows state rather -than a new endpoint. Confirm the exact atom during Phase 1; the candidates are the app-management -or workflow molecule selectors the sidebar's agent switcher already uses. - -## Conventions that constrain the build - -From `web/AGENTS.md`: - -- All new API calls go through the Fern client and the per-resource accessors in - `@agenta/sdk/resources`. Keep zod validation at the boundary with `safeParseWithLogging`. -- Data fetching uses Jotai `atomWithQuery`, never `useEffect` with manual state. Put every - reactive dependency in the `queryKey`; set a sensible `staleTime`. -- Exactly one project is ever in scope. Do not write multi-project-defensive code. -- Styling is Tailwind utility classes plus antd semantic tokens (`bg-colorBgContainer`, - `text-colorText`, and the `--ag-color*` variables). No raw hex, no inline `style`, no CSS-in-JS - except for antd overrides Tailwind cannot express. Implement and verify both light and dark - themes. The reference palette maps onto these tokens; series colors come from the theme scale, - not from literals. -- Keep in-code comments to one line. +# Research — ground truth + +How the analytics engine actually works, what live queries prove, and the code the page reuses. +Every claim here was read from the current tree or measured against a running stack. Where a +number comes from measurement, [capability-review.md](capability-review.md) (`§`) holds the probe. + +Citations are current as of this rewrite. The deep review's own line numbers are frozen at commit +`31c0781d42` and have drifted; the load-bearing ones were re-verified and are corrected below. + +--- + +## How the engine works today + +### It reads root spans only + +`build_base_cte` (`api/oss/src/dbs/postgres/tracing/utils.py:1042`) unconditionally applies +`WHERE parent_id IS NULL` (`:1073`). One run is one root span, so the endpoint counts and +summarizes runs — never the child spans where the tool name, the model that answered, and the +per-span status live. This single fact decides what the page can and cannot chart. + +### The `focus` field is dead + +`focus` reaches `dao.analytics()` but is never passed to `build_base_cte` +(`api/oss/src/dbs/postgres/tracing/dao.py:378-387`), so the `parent_id IS NULL` predicate is +never conditional on it. The endpoint accepts `focus`, echoes it back, and ignores it. Reading +the echoed query to confirm behaviour will confirm a parameter that did nothing. + +### Metrics are `{type, path}` specs over JSON, with no fixed list + +A spec names a JSON path and a type; the engine summarizes whatever path it names. Two types +matter to this page: + +- **`numeric/continuous`** emits `count`, `sum`, `min`, `max`, and 27 percentiles nested under a + `pcts` object — so p95 is at `metrics[path].pcts.p95`, not a flat sibling. (`PERCENTILE_LEVELS` + at `utils.py:924-955`; `percentile_cont` at `utils.py:1266-1287`; nesting at `utils.py:1994`.) +- **`categorical/single`** emits a per-value `freq` table per bucket — runs per harness, per + model, per agent, in one spec each. + +The exact type strings are validated by the `MetricType` enum +(`api/oss/src/core/tracing/dtos.py:240-249`): `numeric/continuous`, `numeric/discrete`, +`categorical/single`, and others. There is **no** bare `numeric` — a wrong string contributes +zero rows silently (`utils.py:1187` numeric, `utils.py:1661` categorical). The default spec set +(`DEFAULT_ANALYTICS_SPECS`, `api/oss/src/core/tracing/service.py:91-98`) reads the canonical +`ag.metrics.costs.cumulative.total` cost path, which holds no data on agent roots — which is why +the page must pass explicit specs. + +### `status_code` is a column, not a spec target + +Specs read the `attributes` JSON only (`build_extract_cte` extracts `attributes #> path`, +`utils.py:1120`), and the extract stage groups by `(timestamp, spec_index)` with no grouping key +— so no spec can split a numeric metric by a category, and no spec can read `status_code`. A +failed run is a run whose root span `status_code` is `STATUS_CODE_ERROR`, so the failure count +comes from a **separate filtered query**. `status_code` is a first-class filter field +(`api/oss/src/core/tracing/utils/filtering.py:508-509`); the operator must be `is` and the value +the full enum literal, or the query returns an empty 200. There is no `STATUS_CODE_OK` — a clean +run is `STATUS_CODE_UNSET`, so success is the complement. + +### Buckets are fixed-width, not calendar + +`date_bin` steps by a fixed duration offset from the window start, in UTC — so buckets drift off +local midnight across a DST transition, and calendar months are not expressible. Above 1024 +buckets the interval is silently coarsened (`_MAX_ALLOWED_BUCKETS`, `utils.py:808`; `_get_stride` +walks to a coarser stride at `:848-859`) — only `buckets[].interval` reports what actually ran. + +### It fails silently, four ways + +A statement-timeout, a rejected filter, a malformed window, and a genuinely empty window all +return `{"count": 0, "buckets": []}` with HTTP 200. An unknown filter field is logged and dropped +(`filtering.py:542-546`), which **widens** the result. The analytics DAO method is wrapped in +`@suppress_exceptions(default=[])` (`dao.py:305`), the route in +`@suppress_exceptions(default=AnalyticsResponse(), exclude=[HTTPException])` +(`api/oss/src/apis/fastapi/tracing/router.py:431`), and only `HTTPException` passes through both +that and the outer `intercept_exceptions` (`api/oss/src/utils/exceptions.py:98-99, 129-130`). +That is why fixing the silent failure (v2 B1) has to cross two layers. + +### Cost lives on one unmapped path + +The canonical `ag.metrics.costs.cumulative.*` paths are empty on agent roots — the roll-up never +crosses the run's OTLP batch boundary to reach the root. The only populated cost path is +`gen_ai.usage.cost`, which the semconv adapter does **not** map to the canonical path +(`api/oss/src/apis/fastapi/otlp/extractors/adapters/logfire_adapter.py:148-196`). The evaluations +service builds its own spec list including `ag.metrics.costs.cumulative.total` and never reads the +default set (`api/oss/src/core/evaluations/service.py:141-158, 1565-1571`), so mapping the cost at +ingest — not in the endpoint defaults — is what fixes both surfaces. + +--- + +## What live queries prove + +Measured on two local dev stacks over `2026-07-01` → `2026-08-03` (§1.3). No production data was +probed, so **every coverage percentage is unverified on production traffic**; capability claims +hold on any dataset. + +- **Run count equals the root-span count exactly.** Analytics returned 7,529 for the window; + direct SQL over the same window returned 7,529 (§4.4 item 1). +- **Latency, harness, and agent identity have near-total coverage** (~99%, ~99%, 96%). These are + the dependable columns. +- **Cost and the token split collapsed.** Populated on 70–95% of runs in early July, then near + zero within a week, cause unknown — the v2 B2 blocker. +- **Per-user is meaningless today.** `created_by_id` is a typed column + (`api/oss/src/dbs/postgres/tracing/mappings.py:210`) but had exactly one distinct value per + project on both stacks. +- **Performance sets the limits, not capability.** A 7-day core call costs ~0.26s, a 30-day one + ~1.7s — so 7 days is the sane default. A `focus=span` scan is a ~5x row fan-out (2.60s vs + 0.68s), which is why span-focus must ship behind the failure-visibility fix. +- **Almost nothing is tested.** One analytics unit test exists + (`api/oss/tests/pytest/unit/tracing/test_analytics_bucket_order.py`), asserting bucket order. + +--- + +## The code the page reuses + +The endpoint-to-dashboard path already exists for the Observability page. A new page sits on top +of that spine and reads a few fields the current mapper drops — it does not rebuild the data layer. + +- **Fetch layer** — `web/packages/agenta-entities/src/trace/api/api.ts`: `fetchSpansAnalytics` + calls `POST /spans/analytics/query` through the Fern client. `SpansAnalyticsParams` (`:293`) + omits `specs` on purpose. The Fern `QuerySpansAnalyticsRequest` already declares `specs?: string` + beside `filter?: string`, so passing explicit specs is one added field, one serialized line, no + client regeneration. +- **Mapper** — `web/oss/src/services/tracing/lib/helpers.ts`: `analyticsToGeneration` (`:106`) + reduces buckets to the dashboard shape; `metricField` (`:88`) reads one **flat** field and does + not reach `pcts.p95`; `calculateIntervalFromDuration` (`:37`) picks a bucket size under the + 1024 ceiling. The existing mapper derives failures from `errors.cumulative` — a count of + errored steps, not failed runs — so a new page needs its own mapper and the status-filtered + query. Do not change the existing one; Observability depends on it. +- **Fetch template** — `web/oss/src/services/tracing/api/index.ts`: + `fetchGenerationsDashboardData` builds the `conditions` array and calls + `fetchSpansAnalytics({focus: "trace", ...})`. +- **State atoms** — `web/oss/src/state/observability/dashboard.ts`: the `atomWithQuery` pattern + keyed on project, range, and filters, `staleTime` one minute, `refetchOnWindowFocus: false`. + `observabilityDashboardTimeRangeAtom` already has a second consumer, so a new page needs its own + atoms. +- **Charts** — recharts `^3.1.0` (`web/oss/package.json`). Style and theming to copy: + `observability/dashboard/CustomAreaChart.tsx` and + `EvalRunDetails/.../EvaluatorMetricsChart/{BarChart,HistogramChart}.tsx`. The existing charts + are area charts of totals; this page needs stacked/grouped bars and a per-bucket p95 line. +- **Route + sidebar** — pages live under + `web/oss/src/pages/w/[workspace_id]/p/[project_id]/`; the observability page is a thin wrapper. + The sidebar's `app-observability-link` entry + (`web/oss/src/components/Sidebar/hooks/useSidebarConfig/index.tsx:141`) is the shape to copy. +- **Agents list** — source the agent filter options from the existing apps/workflows state the + sidebar's agent switcher already uses, not a new endpoint. + +Conventions that constrain any build (`web/AGENTS.md`): all calls through the Fern client with +zod at the boundary; Jotai `atomWithQuery` with every dependency in the key, never `useEffect`; +exactly one project in scope; Tailwind + antd semantic tokens, no raw hex or inline style, both +themes; one-line comments. diff --git a/docs/design/agent-analytics/scope.md b/docs/design/agent-analytics/scope.md index a75320ef5b..46ad1a02b2 100644 --- a/docs/design/agent-analytics/scope.md +++ b/docs/design/agent-analytics/scope.md @@ -1,101 +1,78 @@ -# Scope: v1 now, v2 later - -This page draws the line between what ships now (**v1**) and what waits (**v2**). It is the -roadmap view; plan.md holds the build steps, and capability-review.md holds the evidence behind -every verdict. - -## The rule - -- **v1** is the planned release scope: the charts and Agents filter we will build against - today's endpoint — runs, latency, cost, tokens, and the harness / configured-model / agent - breakdowns. No code is written yet (see status.md); this is the plan, not a shipped state. -- **v2** is anything that needs backend integration or a major fix — a new query capability, a - storage change, or new instrumentation. -- **The two backend items (B1, B2) are v2, not v1 blockers.** By decision of the requester, making - a killed or rejected query surface an error, and investigating the cost / token coverage - collapse, are backend work and follow the base rule above — they move to v2. v1 ships against - today's endpoint without them, accepting the two known limits they would have fixed: the page - cannot tell "no data" from "the query died", and cost may read empty with no signal. Both are - tracked in the v2 table below. - -## How to read the tables - -- **Priority** ranks work within each scope by value against effort: `P0` first, then `P1`, `P2`, - `P3`. -- **Value** — how much the feature matters to a user: High / Med / Low. -- **Effort** — how much work it is: `S` (a day or so), `M` (a few days), `L` (a week or more, or - open-ended). -- Every v1 row is frontend-only; the two backend items (B1, B2) are v2, in the v2 table. +# Scope — possible today vs needs v2 work ---- +The scope line for the Analytics page: what the backend can answer **today**, and what needs a +backend change in **v2**. This is a scoping doc, not a build guide — it says what is in and what +is out, not how to wire it. The mechanics behind every verdict are in [research.md](research.md); +the live-query evidence is in [capability-review.md](capability-review.md) (`§`). -## Scope v1 — build now - -Planned against today's endpoint. No code is written yet (see status.md). Every capability here is -verified backend-*available* in capability-review.md §4.2 unless a note says otherwise — "ready" -describes what the endpoint can answer today, not shipped UI. B1 and B2 (the two backend fixes) -are now v2, so nothing in v1 blocks on them; v1 lives with the limits noted below. - -| Priority | Feature | What it delivers | Value | Effort | Depends on | -|---|---|---|---|---|---| -| P0 | Page shell, route, sidebar | The Analytics page exists, reachable, project-scoped | Enabler | S | — | -| P0 | Data layer (`specs` field + fetch + mapper) | The page requests explicit specs and maps buckets to chart shape | Enabler | M | — | -| P0 | Runs per period (success vs failed) | Stacked Runs chart; corrected `status_code is STATUS_CODE_ERROR` filter + second query | High | M | — | -| P0 | Latency per period (avg + p95 + min/max) | Latency chart with per-bucket p95 line; ships free on the numeric spec | High | S | — | -| P0 | Time-range control (any window, 7-day default) | Reuses the observability `Sort` / `SortResult` | High | S | — | -| P0 | Four page states (data / no-data / unavailable / failed) | Honest empty vs failure vs coverage-gap, per card | High | M | — | -| P1 | Cost per period (coverage-gated total) | Cost chart from `gen_ai.usage.cost`; renders only above the coverage threshold | High | M | — | -| P1 | Tokens per period (total + coverage-gated split) | Tokens chart; the prompt/completion split shows only above threshold | Med | M | — | -| P1 | Breakdown charts: runs per harness / configured model / agent | Three `categorical/single` breakdowns; agent unions `workflow_variant` + `application_variant` | High | M | — | -| P1 | Filters: agents, harness, configured model | Three server-side filters on root-span fields; no backend change (§4.2 items 10–12). Agents by `references`; the model filter narrows the configured alias | High | M | — | - -Notes: - -- **The two backend fixes (B1, B2) are v2**, and v1 lives with their absence. The four page states - are all built, but until B1 the "failed / query-died" state cannot be told apart from "no data", - so a killed query reads as an honest empty. The Cost chart stays coverage-gated and may read - unavailable until B2. Both fixes and their limits are in the v2 table. -- **Cost** is "blocked on coverage" in the review (§4.2 item 4). v1 ships the chart - coverage-gated; B2 (now v2) is what makes it dependable. -- **Configured model** is a proxy for the model that answered — the author's alias. It is - labeled honestly; the real answered model is v2 (needs `focus=span`). -- **Not on the v1 page at all:** tool usage, resolved-model usage, per-model cost, cache tokens, - per-user numbers, skills. All are v2 (below), each for a backend reason. +Priority ranks value against effort (P0 first). Effort is S (a day), M (a few days), L (a week+). --- -## Scope v2 — future, needs backend work - -Ordered by dependency and combined value-vs-effort. The backend column names the enabling change -and its capability-review.md §8.4 item. - -| Priority | Feature | What it delivers | Value | Effort | Backend work it needs | -|---|---|---|---|---|---| -| P0 | Typed analytics contract | Named metrics / dimensions / aggregations, validated and capped; the foundation the rest build on | High | M | Replace the arbitrary-path protocol (§8.4 item 2) | -| P0 | B1 — Make killed / rejected queries surface an error | 504 / 4xx + per-metric `sample_count` instead of a silent empty 200; lets the page tell "no data" from "the query died", and unblocks span-focus | High | M | Typed timeout / filter error in core + router `HTTPException` + per-metric `sample_count` (§8.4 item 1) | -| P1 | B2 — Investigate cost / token coverage collapse | Restores dependable coverage for cost and the token split | High | M–? | Diagnose the mid-July collapse of `gen_ai.usage.cost` and the token split (§4.4 item 4) | -| P1 | Stop reading JSONB on the chart path | Hot columns or a per-run facts table; a permanent latency win and a home for invoked facts | High | M–L | Ingest + storage change (§8.4 item 4) | -| P1 | Tool usage per period (which tools ran) | Real tool-call counts, and a path to per-tool error rate | High | M | `focus=span` wiring + 3 guards + non-root index (§8.4 item 3) | -| P1 | Resolved model usage per period | The model that actually answered, distinct from the configured alias | High | M | `focus=span` (shares the wiring) | -| P2 | Per-model cost / tokens | A numeric metric split by model in one call | High | L | `focus=span` + a group-by dimension (§8.4 item 5) | -| P2 | Cost mapped to the canonical path | `gen_ai.usage.cost` → `ag.metrics.costs.cumulative.total`; also fixes evaluation cost | Med | S | Semconv adapter map (§8.4) | -| P2 | Cache tokens per period | Cache read and write per model call | Low–Med | M | `focus=span`, or roll up to the root | -| P2 | Per-user numbers | Runs / latency / cost per user | Med* | S | A nameable `created_by_id` dimension (needs the contract or facts table) (§8.4 item 6) | -| P2 | Calendar-aware periods | True calendar days and months, timezone-correct | Low | M | Timezone-aware bucketing in the backend (§4.4 item 1) | -| P3 | Skills used (invoked) | Which skills the agent actually invoked | Med | L | Runner instrumentation, then promotion to the root (§4.4 item 14) | -| P3 | Pre-aggregation rollups | Fast wide-window queries at high volume | Med | L | A rollup table; sequence last, after facts land (§8.4 item 7) | - -\* Per-user value is conditional: it means nothing until a project has more than one writing -credential. On both measured stacks `created_by_id` had exactly one value per project (§4.4 -item 13). +## V1 = Possible today ---- +Works against today's endpoint with no backend change. Each is verified backend-available in +§4.2. The label matters: several of these are proxies that must be named honestly in the UI. + +| Capability | What the number actually means | Ready? | +|---|---|---| +| Runs per period | Root spans in the window, in fixed-width buckets. Calendar months are not expressible | **Yes**, with a `trace_type` filter | +| Success vs failed runs | Root-span status. Blind to failures inside a run that recovered a clean root (~1.2%) | **Yes**, with the corrected `status_code` filter | +| Average latency per period | Mean root-span wall clock | **Yes** | +| Latency min / max / p95 | Exact percentile over root durations; 27 percentiles ship free on every numeric spec | **Yes** | +| Total tokens per period | Harness-reported total tokens | **Yes**, with a coverage label | +| Runs per agent | Agent identity from `references`; two naming families unioned | **Yes**, run counts only | +| Filter by agent / harness / configured model | Server-side conditions on root-span fields | **Yes** | -## Where each source draws the line -- **capability-review.md §4.2** — the per-capability verdict table (ready / proxy / not - available) that seeds v1 versus v2. -- **capability-review.md §8.3** — the v1 beta scope. -- **capability-review.md §8.4** — the v2 backend work, in dependency order. -- **plan.md** — the v1 build, phase by phase (Phases 1–4 = the page). Phase 0 (the two backend - fixes, B1 and B2) and Phase 5 are v2 backend work and do not gate the v1 page. +--- + +## V2 Missing features/issues + +One table, ordered by priority (P0 first) and, within a tier, **fixes before missing +capabilities**. The **Type** column says whether a row repairs something that exists but is broken +(**Fix**) or builds a capability that isn't there yet (**Missing**). The "Blocked by / fix" column +is the gate: for a Fix it is the repair, for a Missing item the capability it waits on — one gate +often unlocks several rows. The wall behind most missing items: the endpoint reads root spans only, +and the `focus` field that would widen the scan to child spans is accepted, echoed, and ignored +(§4.3, [research.md](research.md)). + +| Priority | Type | Item | Blocked by / fix | Effort | +|---|---|---|---|---| +| **P0** | Fix | **B1** — killed / rejected queries return an empty HTTP 200, indistinguishable from no data | Raise a typed timeout/filter error in core; re-raise as `HTTPException` (504 / 4xx) at the router; add a per-metric `sample_count` | M | +| **P0** | Fix | **B2** — cost + token-split coverage collapsed to near zero mid-July, cause unknown | Diagnose the collapse (first checks in §4.4 item 4); both measured stacks were local dev, production unverified | M–? | +| **P0** | Fix | **Contract** — request is an arbitrary JSON path validated by silence, pinning every UI to today's ingest shape | Keep it, or move to a typed contract (named metrics/dimensions, filter grammar, caps); gates F1–F5 | M | +| P1 | Fix | **Total cost per period** — harness-reported run total (`gen_ai.usage.cost`); total only, canonical paths empty on roots. UI built + coverage-gated, so it lights up on the B2 fix | B2 | S (UI done) | +| P1 | Fix | **Prompt / completion token split** — same field family as total tokens. UI built + coverage-gated, lights up on the B2 fix | B2 | S (UI done) | +| P1 | Fix | **Stop reading JSONB on the chart path** (F2) — a permanent latency win + a home for invoked facts | Ingest + storage change (hot columns or a facts table) | M–L | +| P1 | Missing | **Tool usage per period** — which tools actually ran | `focus=span` wiring (F1) | M | +| P1 | Missing | **Resolved model usage** — the model that answered | `focus=span` (shares F1) | M | +| P2 | Fix | **Cost on the canonical path** — map `gen_ai.usage.cost` → `ag.metrics.costs.cumulative.total` | Semconv adapter map | S | +| P2 | Missing | **Cost / tokens per harness & per configured model** — the breakdowns worth showing; today's give run counts only | A group-by dimension (F3); both are root-span attributes, so **no** `focus=span` | M | +| P2 | Missing | **Cost / tokens per resolved model** — split by the model that answered | `focus=span` **and** a group-by dimension (F3) | L | +| P2 | Missing | **Cache tokens per period** | `focus=span`, or roll up to the root | M | +| P2 | Missing | **Calendar-aware periods** — true days/months, timezone-correct | Timezone-aware bucketing | M | +| P3 | Missing | **Skills used (invoked)** | Runner instrumentation, then promotion to the root | L | +| P3 | Missing | **Pre-aggregation rollups** — fast wide-window queries at scale | A rollup table, sequenced last | L | + + +### What the gates require + +- **F1 — span-focus wiring.** Make `WHERE parent_id IS NULL` conditional on `focus`. Ship three + guards or it returns wrong numbers: cumulative metrics double-count (use the `incremental` + paths), the run count needs a dedupe (`ag.type.trace` is on every span), and non-root rows need + an index (~5x row fan-out). Do not ship it before B1. §8.4 item 3. +- **F2 — storage.** Reading one number out of a 10 KB JSONB value costs a full detoast; the fix + is storage, not SQL. Promote hot fields to typed columns, or build a per-run facts table (which + also homes invoked tools and resolved models). Both need a late-arriving-batch plan. §8.4 item 4. +- **F3 — a group-by dimension.** No request can split a numeric metric by a category today. Prefer + a per-series dimension under the typed contract over one `group_by` path per query, and cap + cardinality. Harder; defer until F1 ships. §8.4 item 5. + +### Also unresolved before the page leaves beta + +Not analytics questions, but the page cannot ship past beta without answers (§8.5): whether +Analytics **replaces or sits beside** Observability (and the two deprecated, uncalled analytics +routes still in the clients); **test coverage** (one analytics unit test exists, asserting bucket +order); **tenancy tests** (nothing proves a filter cannot reach another project's rows); a +**performance gate** on production-shaped data; and a **rollout flag** until B2 is answered. diff --git a/docs/design/agent-analytics/status.md b/docs/design/agent-analytics/status.md deleted file mode 100644 index f556b052dd..0000000000 --- a/docs/design/agent-analytics/status.md +++ /dev/null @@ -1,102 +0,0 @@ -# Status - -Source of truth for progress. Update as work lands. - -## Current state - -Planning is complete, and a grilling session on 2026-08-02 sharpened the design. No code is -written. The workspace holds the plan; implementation has not started, and no branch exists yet. - -## Locked decisions - -1. Frontend-first scope. The two backend items (Phase 0 in plan.md) are **v2, not v1 blockers** - (requester decision, see scope.md); v1 ships against today's endpoint without them. Charts: - Runs, Latency, Cost (coverage-gated total), Tokens (coverage-gated split), and runs per - harness / configured model / agent. No Costs prompt/completion split chart; `gen_ai.usage.cost` - is a total only. Plus the Agents / Harness / configured-Model filters and the time-range - control; harness and configured model are both breakdown charts and filters (decision 3). The - two deferred Phase 0 items: make a killed or rejected query distinguishable from an empty one, - and investigate the cost / token-split coverage collapse. -2. A net-new page at project scope, named Analytics. The default query aggregates every project - agent; the Agents filter narrows the set. -3. An **agent** is an application/workflow artifact. The Agents multi-select lists the project's - agents and narrows by `references`. v1 has three filters — Agents, Harness, and configured - Model — all server-side conditions on root-span fields, so all three filter today with no - backend change. The configured-Model filter narrows by the author's alias, not the model that - answered (that is the deferred resolved-Model view). -4. One agent invocation is a **run**. The Observability dashboard calls the same metric a - "request"; the two may diverge until Observability is aligned in a later, separate change. -5. A **failed run** is a run whose root span `status_code` is `STATUS_CODE_ERROR` — a run-level - outcome, not a count of errored steps. There is no `STATUS_CODE_OK`, so success is the - complement. It comes from a second, status-filtered query, not a metric spec. The Runs chart's - successful-and-failed split builds on it. -6. The time-range control opens on the **last 7 days** and accepts any window through the - observability `Sort` control and `SortResult`. -7. The deferred **Tools** and **Models** views ship in no form this release, not even as - placeholders. The chart-card shell is built reusably, so they drop in later. - -## Key finding from research - -The data path from the analytics endpoint to a mapped dashboard shape already exists for the -Observability page (`fetchSpansAnalytics`, `analyticsToGeneration`, the observability dashboard -atoms). This feature reuses that spine. The data-layer work: pass explicit metric specs for the -total cost (`gen_ai.usage.cost`) and the token split, read the duration min/max/p95 the current -mapper drops, read the category `freq` breakdowns, and add the status-filtered failed-run query. -The endpoint returns the latency, run-count, and breakdown fields directly. Cost and the token -split stay coverage-gated, because their populated paths collapsed in mid-July — which is why the -scope carries two backend items (Phase 0), now deferred to v2 rather than gating v1. - -## Resolved by code verification - -- **Spec `type` strings**: every number metric is `numeric/continuous`. `MetricType` has no bare - `numeric`, and `DEFAULT_ANALYTICS_SPECS` confirms it. -- **p95 field**: nested at `metrics[path].pcts.p95`, not a flat field. `metricField` does not - reach it, so the mapper needs a small `pcts` accessor. -- **`specs` plumbing**: the Fern `QuerySpansAnalyticsRequest` type already carries `specs?: string` - and forwards it, so passing specs is a small entities-layer change. -- **Failed-run mechanism**: `status_code` is a table column, and metric specs read the - `attributes` JSON only (`build_extract_cte`), so no spec can target it. Failed runs come from a - second query with a `{field: "status_code", operator: "is", value: "STATUS_CODE_ERROR"}` filter - (`status_code` is a first-class filter field; the operator must be `is` and the value the full - enum literal, or the backend returns an empty 200). Per the selected window the page issues two - bucketed queries — one unfiltered for the charts, one status-filtered for failed runs — and - every chart reads them per bucket. See data-contract.md. - -## Open questions to resolve during the build - -- The agents-list atom for the filter options (Phase 1). -- The multi-agent reference filter encoding: a single `in` with all ids, or one condition per - agent combined with `or` (Phase 2, against the existing filter builder). -- Whether the runner marks a failed run's root span `status_code = STATUS_CODE_ERROR`. The - failed-run count depends on it; validate on live traffic (Phase 2). -- Live-response validation: the `trace_type` and `status_code` filters narrow rather than - returning an empty 200, the nested `pcts.p95` reads correctly, and the cost / token-split - coverage on real traffic (Phase 2; it may be near zero — the Phase 0 investigation). - -## Deferred to a later backend change - -The Tools chart, the Models chart, the Models filter, and per-model cost. They split into two -sub-phases, because the two deferred-view enablers (span-focus wiring and a group-by dimension — -distinct from the two Phase 0 backend items) are not co-equal gates: - -- **5a, Tools and Models-share**: needs span-focus wiring only (thread `focus` into - `build_base_cte`, make `WHERE parent_id IS NULL` conditional). No group-by, because a - `categorical/single` spec already returns per-value frequencies. Ship a guard: under - `focus = "span"`, cumulative metrics double-count, so cost/tokens must use the `incremental` - paths. -- **5b, per-model cost/tokens**: additionally needs a group-by dimension. Harder; defer until 5a - ships. - -Open design decision for 5b: group-by as a per-query dimension (preferred) versus a per-spec -`group_by` field. Phase 5 in plan.md; not scheduled here. - -## Source materials - -- Decoded reference implementation: the artifact was unpacked to plain source. The page logic - (data model, charts, KPIs) is the `Component` class; the layout is the `x-dc` template. - Original artifact: - `https://claude.ai/code/artifact/75b4f14e-9c9b-407b-9d35-317927fb6772`. -- Backend capability notes: `docs/design/agent-analytics/Note.md`. -- Endpoint architecture review (the source of the root-span-only and dead-`focus` findings): a - read-only review generated 2026-08-01. Its conclusions live in context.md and data-contract.md, - so the workspace does not depend on the review file.