[docs]: Add Agent Analytics design dashboard - #5648
Conversation
…mentation - Introduced the Analytics page to visualize agent performance metrics. - Created context, data contract, plan, research, and status documentation for the new feature. - Ensured the new page reuses existing data-fetching mechanisms without altering the backend.
…s, refine glossary terms, and enhance data contract details
dosubot
Bot
added
the
size:XL
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdded seven design documents for a project-scoped Agent Analytics page. The documents define page scope, backend capabilities, analytics contracts, chart behavior, implementation phases, deferred backend work, validation, and unresolved questions. ChangesAgent Analytics
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Let me know your thoguhts @mmabrouk |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c6d17e97-baab-493b-a6b2-554de3f419cc
📒 Files selected for processing (6)
docs/design/agent-analytics/README.mddocs/design/agent-analytics/context.mddocs/design/agent-analytics/data-contract.mddocs/design/agent-analytics/plan.mddocs/design/agent-analytics/research.mddocs/design/agent-analytics/status.md
…ytics backend The design workspace in this PR plans the Analytics page on the assumption that the backend already answers everything the first release needs, and that no backend change is required. This adds an independent review that tests that assumption against the running code. The review answers the two framing questions (how analytics works today, and how the three mounted routes and two implementations differ), then takes each wanted capability one at a time with a verdict, the exact request that produces it, and the live probe that proved or disproved it. It ends with measured latency, a scale projection, the frontend requirements, and a v1 and v2 proposal. Findings that change the plan: - Six of the fourteen wanted capabilities work today and mean what the wish list assumes. Four work only under a narrower label, such as "configured model" rather than the model that answered. Four are not available today at any price. - The query engine is more capable than the plan assumes. It summarizes any JSON path, returns 27 percentiles on any numeric metric, and returns a per-value frequency table per bucket on any categorical metric. Three capabilities the plan defers work today. - The plan's failed-run filter is rejected by the backend and returns an empty success, so the page would report perfect health on a project with a 9.5% failure rate. - The cost chart reads JSON paths that hold no data on either dataset measured. A different path does hold cost, and its coverage fell from about 70% of runs to near zero in mid-July on both datasets. The cause is unknown and is the top open item. - A 30-day query takes 1.7 seconds at today's volume, and a query killed by timeout returns HTTP 200 with an empty result rather than an error. Seven days costs 0.26 seconds and should be the default window. Evidence: every claim carries a file:line citation or a live probe result. Probes ran against two local development stacks; no production data was queried, so coverage percentages are unverified on production traffic. Raw probe payloads and query plans are not committed; Appendix A indexes them. Docs only. No code changes. Claude-Session: https://claude.ai/code/session_01RkWWQUNNzRbaB5jnCAdjYA
|
I pushed It is an independent capability review of the analytics backend, written after your four documents and tested against the running code. Every claim carries either a Where it agrees with your plan: the spine you picked is right. Where it disagrees, and why the plan needs a revision before implementation starts:
Section 8 has a v1 that stays within the existing backend and a v2 with the backend work in order. Section 7 lists the metric-meaning decisions that need an answer before anyone writes code. I did not edit |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/design/agent-analytics/README.md (1)
39-42: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winExclude annotation traces from the documented run metric.
This glossary defines a run as an agent invocation but equates run count with
ag.type.trace. The capability review states that this metric also includes annotation traces unless filtered. Document the requiredtrace_type is invocationfilter, or qualify the metric definition. Otherwise run volume and derived health values can include non-agent traces.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: de39af7a-39f6-4acf-abd7-500c5cf51bab
📒 Files selected for processing (2)
docs/design/agent-analytics/README.mddocs/design/agent-analytics/capability-review.md
- Updated the plan to clarify backend prerequisites for Phase 0, including distinguishing between empty results and failures in API responses. - Enhanced Phase 1 details to specify the structure of analytics queries and the validation process for metrics. - Revised Phase 3 to include specific state management for the analytics page and ensure proper handling of loading and empty states. - Expanded Phase 4 to detail the summary panel and chart requirements, including coverage-gated metrics and responsive design considerations. - Clarified the definitions of failed runs and health scores in the status documentation, ensuring consistency with backend expectations. - Added details on the necessary data paths and metrics for the new analytics page, emphasizing the need for coverage-gated data due to previous collapses. - Documented open questions and risks to address during the build process, focusing on validation of metrics and response handling.
- Updated the plan.md to clarify the cost chart and token split coverage collapse investigation. - Enhanced research.md to detail the analytics fetch layer and response-to-dashboard mapper. - Introduced scope.md to delineate features for v1 and v2, including backend prerequisites. - Revised status.md to reflect current planning status and locked decisions, emphasizing frontend-first scope and backend dependencies. - Improved clarity and consistency across documentation, ensuring alignment with implementation goals.
There was a problem hiding this comment.
Actionable comments posted: 12
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5069ca29-2b3d-4025-b811-b652a21125fd
📒 Files selected for processing (7)
docs/design/agent-analytics/README.mddocs/design/agent-analytics/context.mddocs/design/agent-analytics/data-contract.mddocs/design/agent-analytics/plan.mddocs/design/agent-analytics/research.mddocs/design/agent-analytics/scope.mddocs/design/agent-analytics/status.md
…filter and query specifications
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/design/agent-analytics/data-contract.md (1)
191-199: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winSpecify the coverage threshold and zero-run behavior.
coverage clears a thresholdis not an implementable rule. Define the formula, numeric threshold, scope of the calculation, and the result when the window has zero runs. Otherwise clients can make different availability decisions or calculate an invalid ratio.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 0a5dc396-3925-4181-89a9-420831ff139f
📒 Files selected for processing (5)
docs/design/agent-analytics/context.mddocs/design/agent-analytics/data-contract.mddocs/design/agent-analytics/plan.mddocs/design/agent-analytics/scope.mddocs/design/agent-analytics/status.md
🚧 Files skipped from review as they are similar to previous changes (3)
- docs/design/agent-analytics/status.md
- docs/design/agent-analytics/scope.md
- docs/design/agent-analytics/plan.md
dosubot
Bot
added
size:XXL
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2d3d5d50-9dc5-4a3e-aa1d-e55aaa2574bd
📒 Files selected for processing (3)
docs/design/agent-analytics/plan.mddocs/design/agent-analytics/scope.mddocs/design/agent-analytics/status.md
🚧 Files skipped from review as they are similar to previous changes (2)
- docs/design/agent-analytics/status.md
- docs/design/agent-analytics/plan.md
…tatus sections - Revised the research section to clarify the analytics engine's functionality and the data path from endpoint to dashboard. - Updated the scope document to delineate features for v1 and v2, emphasizing backend dependencies and capabilities. - Removed the status document as its content is now integrated into the scope and research sections, reflecting the current state of development and decisions made.
dosubot
Bot
added
size:L
2 participants
Context
We want a project-scoped Analytics page in the web app that charts how a project's agents perform over a time window: run volume, success and failure, latency, cost, and token usage. Before writing any UI code, this PR lands the design workspace that scopes the page, maps the code it will reuse, and pins the data contract. It adds documentation only. No
web/orapi/code changes.Changes
A new
docs/design/agent-analytics/workspace with a fixed reading order:POST /spans/analytics/queryto a mapped dashboard shape already exists for the Observability page, so this feature reuses that spine rather than adding a new data layer.The scope is deliberately frontend-first: four charts, four stat tiles, a browser-computed health donut, an Agents filter, and a time-range control, with no
api/change. A 2026-08-02 code-verification pass confirmed the endpoint already returns everything the in-scope charts need, so passing explicit metric specs is a two-line entities-layer change, not new plumbing.Notes
Deisgn preview
Summary
Total 3 sections
In short: No backend changes required for the first iteration