Skip to content

docs(agent-workflows): Plan usage and cost telemetry #5253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mmabrouk wants to merge 1 commit into from
Closed

docs(agent-workflows): Plan usage and cost telemetry #5253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/design/agent-workflows/projects/agent-usage-telemetry/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Agent usage telemetry

This project defines how agent harnesses report token usage, context utilization, and monetary
cost from the harness boundary through the runner and Python service into Agenta tracing.

The immediate symptom is inaccurate cost in traces for Pi and Claude. The underlying problem is
broader: the current four-number usage object conflates billable token usage, ACP context-window
occupancy, and cost from sources with different authority. It also loses cache detail and places
the same totals on several span levels without an attribution rule.

The proposal starts with the pre-production runner and runner-to-service contracts. It also
documents a service-to-API semantic-convention proposal for review. That later boundary requires
CTO approval before implementation.

## Files

- [context.md](context.md): problem, scope, goals, and constraints.
- [research.md](research.md): current path, information-loss points, existing API behavior, and
broader tracing gaps.
- [trace-inventory.md](trace-inventory.md): end-to-end inventory of runner trace fields, adapter
handling, known gaps, and this project's scope.
- [interface-design.md](interface-design.md): proposed canonical usage model, source mappings,
aggregation rules, and semantic-convention proposal.
- [plan.md](plan.md): phased implementation and review sequence.
- [qa.md](qa.md): contract, trace, API, and live verification matrix.
- [status.md](status.md): current state, decisions, approvals, and blockers.

## Recommended reading order

Read `context.md`, then `research.md`, `trace-inventory.md`, `interface-design.md`, and `plan.md`. Reviewers deciding
the service-to-API boundary should also read the semantic-convention section in
`interface-design.md` and the approval gate in `status.md`.
71 changes: 71 additions & 0 deletions docs/design/agent-workflows/projects/agent-usage-telemetry/context.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Context

## Problem

Agent traces show token totals and monetary cost for Pi and Claude runs, but those numbers do not
have one stable meaning today.

The runner reduces every harness to:

```json
{"input": 10, "output": 5, "total": 15, "cost": 0.001}
```

That shape cannot say whether input includes cached tokens, whether cost came from the provider or
a local model catalog, which currency applies, or whether an ACP number describes billed usage or
current context occupancy. Pi and Claude both expose more information than this contract retains.

Tracing adds a second problem. The runner places usage on leaf LLM spans and repeats run totals on
agent and workflow spans to bridge separately exported OTLP batches. Agenta ingestion treats
values as incremental and rolls children into parents, so repeated totals can be counted more than
once. The API also recalculates supported-model costs with LiteLLM even when a producer supplied a
cost.

## Boundary constraints

- The harness-to-runner and runner-to-Python-service interfaces are pre-production. We can change
them while defining the correct model.
- The service-to-API tracing boundary is more stable. A semantic-convention or ingestion change
needs CTO approval.
- Existing trace storage is JSON-path based and can retain additional numeric attributes. The
hard part is agreeing on meaning, normalization, rollup, and compatibility.
- A tracing failure must not fail an agent run. Missing telemetry must remain distinguishable from
reported zero.

## Goals

1. Preserve the usage and cost facts Pi, Claude, and future harnesses actually report.
2. Separate billed token usage, monetary cost, and context-window utilization by semantic role.
3. Define inclusive input/output totals and cache/reasoning subcategories consistently with the
supported OpenTelemetry GenAI conventions.
4. Preserve cost currency, provenance, and reported-versus-estimated status.
5. Define field-level reconciliation when final responses, stream updates, and Pi writeback each
provide part of the result.
6. Attribute incremental usage exactly once in a trace and derive parent cumulative totals without
double counting.
7. Propose an Agenta cost semantic convention, cache-aware fallback behavior, UI implications, and
documentation updates for CTO review.
8. Cover partial, cancelled, resumed, multi-turn, and cost-only runs instead of only successful
single-turn requests.

## Non-goals

- Reconstruct provider invoices when the provider or harness reports no usage.
- Pretend ACP exposes per-model-call detail that it does not expose.
- Add a new pricing service in the runner.
- Change the service-to-API contract before approval.
- Redesign the observability UI in the first implementation phase.

## Success criteria

- Cache-heavy Pi and Claude fixtures retain inclusive input, cache-read, cache-creation, output,
total, and cost provenance through the runner-to-service result.
- ACP `used` and `size` appear only as context-utilization fields.
- Streamed provisional context never overrides final billed usage.
- Every monetary value carries a currency and source, including zero.
- A trace attributes billable usage to one level only and produces the expected cumulative total.
- The API proposal states when reported cost wins, when estimation is allowed, and how every token
and cost bucket rolls up.
- Public semantic-convention and cost-tracking documentation is updated in the implementation that
changes the stable tracing boundary.

278 changes: 278 additions & 0 deletions docs/design/agent-workflows/projects/agent-usage-telemetry/interface-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
# Interface design

## Design rules

The contract groups fields by semantic role:

- `usage`: consumed/generated units and monetary charges for this run;
- `context`: a point-in-time runtime gauge, not billing data;
- tracing metadata: how a fact maps onto a span and whether it is incremental or cumulative.

Missing and zero are different. Every number must be finite and non-negative. Producers preserve
reported totals instead of silently replacing them with arithmetic when a provider's semantics do
not match the canonical invariant.

## Proposed runner-to-service result

```jsonc
{
"usage": {
"tokens": {
"input": 12000, // inclusive: uncached + cacheRead + cacheCreation
"output": 800, // inclusive: includes reasoning when reported that way
"total": 12800,
"details": {
"uncachedInput": 1000,
"cacheReadInput": 10500,
"cacheCreationInput": 500,
"reasoningOutput": 300
}
},
"costs": {
"total": {
"amount": 0.021,
"currency": "USD"
},
"details": {
"uncachedInput": 0.003,
"output": 0.012,
"cacheReadInput": 0.001,
"cacheCreationInput": 0.005
},
"provenance": {
"kind": "reported", // reported | calculated | estimated
"source": "provider", // provider | harness | agenta
"sourceName": "anthropic"
}
},
"status": "final"
},
"context": {
"usedTokens": 12800,
"windowTokens": 200000,
"source": "acp",
"observedAt": "2026-07-12T12:00:00Z"
}
}
```

The final naming should follow the existing wire's camelCase convention. `usage.tokens` and
`usage.costs` are run-level cumulative facts. They do not claim per-LLM-call fidelity.

### Token invariants

- `input` follows current OTel GenAI meaning and includes every input token.
- Cache-read and cache-creation counts are subcategories of `input`, not additive siblings after
normalization.
- `output` includes every output token. Reasoning is a subcategory when the provider reports it.
- `total`, when canonical, equals `input + output`.
- `details.uncachedInput + cacheReadInput + cacheCreationInput` should equal `input` when all
three mutually exclusive raw buckets are known. A missing detail stays missing.
- Never calculate a missing cache bucket by subtraction unless the source contract guarantees all
other buckets are complete.

### Cost invariants

- Currency is required whenever any monetary amount exists. Version one supports USD but keeps
currency explicit.
- `reported` means a provider or provider SDK supplied the amount.
- `calculated` means the harness calculated it from a model catalog and observed token buckets.
- `estimated` means Agenta calculated it after ingestion as a fallback.
- Zero is a valid reported amount. Absence means unknown.
- A total may exist without component costs. Do not invent a prompt/output split.
- `uncachedInput`, `cacheReadInput`, `cacheCreationInput`, and `output` are mutually exclusive
cost components when all are known. Their sum equals `total`.
- Compatibility `prompt` cost is inclusive input-side cost. It equals uncached input plus cache
read plus cache creation and must not be added to those details again.
- Cost provenance is metadata, not a numeric metric bucket.

### Context invariants

- `usedTokens` and `windowTokens` are gauges observed at one time.
- They never populate `usage.tokens.total`.
- A newer context snapshot replaces an older snapshot. Usage snapshots instead follow their
declared run/turn scope and aggregation rules.

## Harness normalization

| Source | Canonical mapping | Authority |
|---|---|---|
| Pi message `input`, `cacheRead`, `cacheWrite` | sum to inclusive input; retain exclusive details | final per Pi model call |
| Pi `output`, optional reasoning detail | inclusive output plus detail | final per Pi model call |
| Pi `cost.*` | cost components/total, USD, calculated by harness unless provider reporting is proven | calculated |
| Claude `PromptResponse.usage` | normalize input + cached-read + cached-write, output, and explicit total | final run tokens |
| Claude ACP `usage_update.used/size` | context used/window only | latest gauge |
| Claude ACP `cost.amount/currency` | run cost snapshot | provider-reported via SDK |

Future adapters must document whether their input/output fields are inclusive before mapping them.

## Internal observation type

Normalization operates on observations before producing the aggregate result:

```jsonc
{
"scope": "model_call",
"temporality": "delta",
"status": "provisional",
"resource": {
"provider": "anthropic",
"model": "claude-sonnet-4-6",
"responseModel": "claude-sonnet-4-6",
"operation": "chat"
},
"tokens": { /* normalized token facts */ },
"costs": { /* amount, currency, provenance */ },
"context": { /* only for gauge observations */ },
"observedAt": "2026-07-12T12:00:00Z"
}
```

Resource identity is required before Agenta estimates a component. Multi-model runs aggregate
reported monetary values only when currencies match. Estimation remains model-scoped and sums the
components after pricing each model separately. The final run result may expose one aggregate plus
optional `breakdown` entries keyed by provider, response model, operation, and currency. It must
not collapse heterogeneous observations before estimation.

Cost provenance is attached to each breakdown entry. The aggregate may carry one provenance only
when every component has the same kind and source. Otherwise its provenance is `mixed` and the
breakdown is required.

Token status and validation quality survive on the final result. If a provider total conflicts
with normalized inclusive input plus output, retain the reported total as `reportedTotal`, emit
the normalized arithmetic total as `total`, and mark `validation.status="mismatch"`. Do not send
the inconsistent provider number as the canonical rollup total.

## Reconciliation reducer

Replace whole-object precedence with a field-level reducer:

1. Validate every observation and retain its source, scope, temporality, and final/provisional
status internally.
2. Use final Pi message/writeback or final Claude prompt response for token usage.
3. Use the latest final provider-reported cost. Fall back to harness-calculated cost. Do not
estimate in the runner.
4. Keep the latest ACP context gauge separately.
5. Accumulate incremental Pi model-call observations once. Do not sum repeated cumulative ACP
snapshots.
6. Preserve partial usage on cancellation and error when the harness reported it.
7. Emit exactly one final run-level usage record. Provisional context uses a distinct event type.

Recommended neutral events:

```jsonc
{"type": "context_usage", "usedTokens": 12000, "windowTokens": 200000}
{"type": "usage", "scope": "run", "status": "final", "usage": { /* shape above */ }}
```

The Vercel adapter may initially project inclusive input/output/total to its established usage
metadata. It must not project `context_usage` as billed usage.

## Exact result and event placement

`AgentRunResult.usage` holds the final run aggregate and optional resource breakdown. A sibling
`AgentRunResult.context` holds the last context gauge for diagnostics. The terminal result always
contains the final usage when known, including on partial/error completion.

Streaming uses `context_usage` for provisional gauges and at most one `usage` event with
`status=final` for the settled run aggregate. The terminal result repeats the final aggregate as
the authoritative transport result. Stream adapters de-duplicate the event/result pair rather
than choosing whichever arrived first.

## Trace attribution

### Incremental owner

Billable usage belongs exactly once:

- Pi with per-call instrumentation: each leaf LLM span owns its incremental usage.
- ACP without per-call usage: the synthetic run LLM span owns the run usage and is explicitly a
run-level approximation.
- Agent and workflow parents expose derived cumulative values only. They do not repeat the same
values as incremental metrics.

Separate OTLP batches currently prevent the API from deriving a parent total in one ingest pass.
The implementation must choose one reviewed bridge:

1. export a clearly marked cumulative summary on the remote parent;
2. teach ingestion to reconcile later-arriving children before cumulative query results;
3. stop duplicating and accept that an isolated parent batch has no total until trace-level query
aggregation.

The current approach, repeating `gen_ai.usage.*` on every level, is not acceptable because those
attributes are interpreted as incremental.

## Service-to-API semantic-convention proposal

This section is a proposal pending CTO approval.

### Attribute classification

The implementation must pin one supported OTel GenAI semantic-convention version and maintain one
authoritative mapping table. Every emitted field is classified as `otel_standard`,
`compatibility_alias`, or `agenta_extension`.

Input and output usage are current OTel GenAI fields. Cache and reasoning dotted fields are
already supported by Agenta's Logfire adapter and appear in current OTel GenAI registries, but
their exact status must be verified against the pinned version before the design calls them
standard. Until then, treat them as supported compatibility fields:

```text
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cache_read.input_tokens
gen_ai.usage.cache_creation.input_tokens
gen_ai.usage.reasoning.output_tokens
```

Keep legacy prompt/completion and total attributes only as documented compatibility aliases at
ingestion. Pin the supported OTel GenAI vocabulary/version in one authoritative mapping table
instead of letting `semconv.py` and individual adapters drift.

### Agenta cost extension

OTel currently has no standard GenAI monetary cost attribute. Emit the established Agenta metric
namespace:

```text
ag.metrics.unit.costs.total
ag.metrics.unit.costs.input
ag.metrics.unit.costs.output
ag.metrics.unit.costs.cache_read_input
ag.metrics.unit.costs.cache_creation_input

ag.meta.cost.kind = reported | calculated | estimated
ag.meta.cost.source = provider | harness | agenta
ag.meta.cost.source_name = anthropic | pi | litellm | ...
ag.meta.cost.currency = USD
```

Compatibility can continue exposing prompt/completion cost, with these definitions:

- `prompt` is the total input-side cost, including uncached input, cache reads, and cache creation;
- `completion` is total output-side cost;
- detailed buckets are subcomponents and must not be added again to `prompt` or `total`.

The API must preserve producer-supplied cost. LiteLLM estimation runs only when no reported or
harness-calculated total exists. Estimated values carry `kind=estimated`, `source=agenta`, and the
pricing source/version when available.

### Agenta token extension and rollups

Ingestion should normalize standard OTel fields into canonical Agenta token metrics with inclusive
input/output plus optional cache/reasoning subcategories. Rollups must retain arbitrary approved
numeric buckets or use a schema table, rather than hardcoding three keys.

The semantic-convention documentation must state whether each subcategory is included in its
parent and define total arithmetic. This prevents cache and reasoning double counting.

## Compatibility and versioning

- Change the pre-production `/run` result and event schema directly, with updated goldens.
- Keep a temporary parser for the old flat four-field result only if deployed runner/service skew
can occur during rollout.
- Do not silently reinterpret old `input` as inclusive. Version or normalize based on the wire
shape.
- Keep service-to-API additions backward compatible. Existing prompt/completion/total queries
continue to work while detailed paths become available.
- Update public docs in the same PR that changes API semantics.
Loading
Loading