docs(agent-workflows): Plan usage and cost telemetry

[mmabrouk]

Context

Pi and Claude expose cache-aware token and cost information, but the agent runner reduces every
run to four ambiguous numbers. That loses cache buckets, currency, provenance, and model scope,
and it conflates ACP context-window occupancy with billed token usage. Tracing then repeats totals
on leaf and parent spans while the API recalculates producer-supplied cost, so the displayed cost
can be inaccurate or counted more than once.

This design starts at the pre-production harness, runner, and runner-to-service boundaries. It
keeps exported span attribution and service-to-API semantic-convention changes behind an explicit
CTO approval gate.

What this proposes

The runner normalizes each harness into inclusive input/output token totals with cache and
reasoning subcategories, monetary cost with currency and provenance, model-scoped observations,
and a separate context-utilization gauge. Pi and Claude adapters reconcile fields by authority and
temporality instead of replacing one whole usage object with another.

The plan also inventories the wider trace path: span hierarchy, model and response identity,
content policy, status/errors, timing, adapters, storage, rollups, query presets, and UI display.
It proposes one-owner incremental attribution and an Agenta cost semantic convention, but defers
all API-visible behavior changes until the CTO approves the convention, rollup, and cross-batch
strategy.

Before:

{"usage":{"input":10,"output":5,"total":15,"cost":0.001}}

Proposed direction:

{
  "usage": {
    "tokens": {"input":12000,"output":800,"total":12800,"details":{"cacheReadInput":10500}},
    "costs": {"total":{"amount":0.021,"currency":"USD"},"provenance":{"kind":"reported","source":"provider"}}
  },
  "context": {"usedTokens":12800,"windowTokens":200000}
}

How to review

1. Read context.md for scope and boundary constraints.
2. Read research.md and trace-inventory.md for the end-to-end findings and deferred gaps.
3. Review interface-design.md, especially inclusive token arithmetic, model-scoped cost
   provenance, final versus provisional events, and the service-to-API proposal.
4. Review plan.md for the runner-first sequence and the CTO gate before exported trace changes.
5. Check qa.md for cache-heavy, multi-model, partial/error, streaming, and no-double-counting
   coverage.

Decisions requested

• Approve or revise the canonical runner-to-service usage shape and normalization rules.
• Choose the cross-batch parent-summary direction to take into CTO review.
• Decide whether runner and service deployment can skew, requiring temporary old-wire parsing.
• Decide whether the first Vercel projection exposes cache details or only inclusive totals.
• Confirm the scope of the CTO approval packet for Agenta cost semantics, rollups, UI labels, and
  public documentation.

Tests / notes

This PR contains design documents only. The research used the live runner, installed Pi and Claude
ACP packages, Python service contracts, API adapters and tree rollups, observability UI paths,
existing tests, and public tracing documentation.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 12, 2026 4:20pm

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 01c56fd7-3e5f-4cf8-bdc3-2461105ac081

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/agent-usage-telemetry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mmabrouk]

🤖 The AI agent says: Please review the canonical runner-to-service usage shape, inclusive token and cache semantics, model-scoped cost provenance, and the phase boundary that requires CTO approval before any API-visible span attribution or semantic-convention change. The five open decisions are listed in status.md and the PR body.

[mmabrouk]

Closing — tracked under #5505.

[mmabrouk]

Tracked in #5505 (related open PRs in the same metering area: #5039, #5040).

[mmabrouk]

This design was closed without review. I had an independent staff-level reviewer (Codex, gpt-5.6, maximum reasoning effort, read-only over the repo) answer the five open decisions in status.md and critique the design. It read the code and the installed dependencies rather than trusting a summary. Posting the result here so the answers live with the design.

Verdict

The semantic foundation is right, but the document is not implementation-ready.

Approved as written: inclusive canonical input and output; explicit cost provenance; a documented Agenta cost extension; one incremental owner per usage observation; context occupancy kept separate from billable usage.

Rejected or needing revision: the unresolved cross-batch strategy; arbitrary numeric-key rollups; the ambiguous temporality of ag.metrics.unit.costs.*; PR #5352's exclusive AgentUsage.input contract.

The five decisions

1. Cross-batch parent summaries. Export exactly one explicitly cumulative run summary on the SDK workflow root, derived from the runner's terminal result. Keep incremental usage only on leaf model-call spans. The root carries ag.metrics.acc.* marked with scope, temporality, finality and a schema version, and parents stop repeating totals as gen_ai.usage.*, which ingest reads as incremental. The data already crosses the boundary, since the SDK receives the runner's final aggregate today. The alternative, a durable ingestion reconciler, would mean building a stateful trace reducer for late arrivals, retries and finalization purely to recover a total the runner already hands over. One hardening is required either way: re-ingestion currently overwrites every mutable span field on conflict, so a summary needs deterministic single emission or a finality rule, or a stale replay can overwrite a newer final value.

2. Version skew. Yes, it can happen, and it is not hypothetical. Railway, Docker Compose and the Kubernetes chart all deploy the services and the runner as independently versioned artifacts. Add usage.schemaVersion, ship a service that reads both shapes, then the new runner, watch the legacy rate drain, then remove the parser. The legacy parser must mark old input as exclusive rather than claiming it is canonical, because it cannot reconstruct cache tokens.

3. First Vercel projection. Inclusive input, output and total only. Cache and reasoning stay internal. Exposing them now creates a second compatibility surface before the canonical schema is stable.

4. Rollup strategy. A generic reducer governed by a versioned bucket schema. Not arbitrary numeric keys. Each registered bucket declares its type, scope, temporality, rollup operation, parent-inclusion relationship, currency rule and missing-versus-zero rule. Cache-read tokens are additive across calls but already inside canonical input. Context occupancy is a gauge and needs latest-observation semantics. Costs can only be summed when the currency matches. Summing every number under metrics would trade one bug for a class of double counting and gauge corruption. Unknown buckets get stored but not rolled until registered.

5. First UI change. Provenance of the total first, and not as a binary. Three honest labels: reported, harness-calculated, estimated by Agenta. Source and currency in the tooltip. If provenance is mixed, say "Mixed" and require a breakdown rather than presenting one homogeneous number. Defer cache and reasoning breakdowns, because they need transport, types, trace queries and completeness state, not just a visual change.

The inclusive-versus-exclusive contradiction, settled

The design says input is inclusive of cache. PR #5352 says the opposite. The ruling: both are right about different things. #5352 correctly describes raw harness semantics. Pi's own OpenAI adapter subtracts cache buckets from the provider's prompt count to produce its raw input, and Claude ACP accumulates the four buckets separately then sums all of them into its total. But a canonical boundary exists precisely to normalize vendor contracts, and the OpenTelemetry GenAI convention defines input as inclusive with cache as a subcategory. So canonical input should be inclusive; raw exclusive input survives as details.uncachedInput.

It confirmed this by running the pinned litellm 1.92.0 with 93 uncached and 13,463 cache-read tokens:

Shape passed to litellm	GPT-5.3 Codex input cost	Claude Sonnet 4.6 input cost
93, no cache arguments	$0.00016275	$0.000279
93, plus cache arguments	$0.002356025	$0.0040389
13,556 inclusive, plus cache arguments	$0.002518775	$0.0043179

The middle row silently loses the 93 ordinary tokens. Litellm derives ordinary input by subtracting the cache details from the prompt count it is given, so it expects the inclusive figure. This holds on both the OpenAI and the Anthropic pricing routes.

What the July design did not answer, and an implementer hits on day one

Which summary is authoritative and when it becomes final. What happens after cancellation or partial completion. Whether a later child batch may update a root. How a replay is prevented from replacing a newer final value. Whether ACP PromptResponse.usage is turn-scoped or session-cumulative per adapter, since the generated ACP type describes totals "across session". Exact span ownership per harness. Missing versus measured zero. Currency mismatch. Pricing catalog revision and alias normalization. Sampling and dropped spans. Backfill of historical traces.

What is now obsolete

Four fixes landed this week, in #5708, #5709 and #5710. Context occupancy is no longer reported as token usage, though it is dropped rather than preserved as a gauge. A runner batch with an external parent no longer yields an empty tree, though rollup is still batch-local. Streaming span activation is repaired. Producer cost is no longer discarded, but that mapping is a compatibility workaround with ambiguous span scope, not a final contract, because the runner emits gen_ai.usage.cost on both leaf model-call spans and the agent summary, and an attribute name cannot tell ingest which it is.

One thing to state plainly in the design

This is observability telemetry, not a billing ledger. If Agenta ever charges customers from these numbers, best-effort OTLP delivery, floating-point dollars, sampling and overwrite-on-retry storage are all unacceptable. Billing would need a durable idempotent usage ledger with integer minor units and invoice reconciliation. Worth writing down before someone assumes otherwise.

Immediate follow-up

Ship the cached-cost fix in #5540 before the larger design, since it is a current correctness bug and can be tested against today's known exclusive runner semantics. That work is in progress and also carries two related defects the review surfaced: calculate_costs never falls back to ag.meta.request.model, so estimation is dead for every non-codex agent span, and the new cost-preservation guard tests for a non-zero value rather than attribute presence, so a genuine reported zero gets overwritten.