feat: Token count for agents

[Ayaz-Microsoft]

Purpose

Count total input and output tokens used by each agent at various stages and show in workbook for analysis.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
src/backend
app.py	720	135	81%	47, 64, 71–76, 79, 84, 119–120, 165, 244, 262, 281, 288, 412–413, 519, 522, 525–527, 536–537, 540, 542–544, 553–556, 566–567, 570–576, 579–581, 587–588, 590, 601, 603–608, 611–613, 620, 630–631, 633, 732–736, 744, 747, 755, 758–761, 770–771, 774, 783, 785, 813, 822–823, 837–838, 857–858, 860–861, 868–869, 872–873, 1017–1019, 1023–1025, 1064–1065, 1103, 1106, 1108, 1134–1135, 1137–1139, 1141, 1161–1162, 1164–1165, 1236–1237, 1268–1269, 1346–1347, 1607–1609, 1611–1612, 1619–1621, 1623, 1758, 1776, 1838–1839, 1843–1844
llm_token_telemetry.py	416	113	72%	121, 168–177, 190, 205–206, 209–211, 280, 288–290, 348, 361–377, 459–460, 467, 471–472, 478, 500, 506, 509–511, 517–519, 530, 548, 550–552, 568, 572, 583–584, 593, 614–616, 625–627, 639, 657, 673–675, 689–691, 712, 739–741, 777, 799, 806, 896–898, 906, 909–910, 982, 984–985, 987–994, 996–1002, 1004–1005, 1010, 1020, 1024
orchestrator.py	768	190	75%	38–40, 549, 552–558, 560–564, 568, 574–575, 587, 591, 594–595, 603, 607–610, 619, 624–625, 631–632, 637–638, 645, 753, 941, 984–985, 989–990, 999, 1001–1002, 1004, 1010–1012, 1014, 1049–1050, 1053–1059, 1088, 1113–1116, 1118–1120, 1127–1128, 1130–1131, 1134–1136, 1138, 1148–1151, 1155–1156, 1158–1159, 1170–1171, 1173–1180, 1189–1190, 1193–1194, 1219, 1258–1259, 1278–1279, 1335–1336, 1339–1340, 1350–1352, 1438, 1466, 1509–1510, 1513–1514, 1519–1521, 1523–1525, 1555–1558, 1648–1649, 1654–1656, 1672–1674, 1676–1678, 1692–1695, 1731–1732, 1761, 1809–1810, 1831, 1835, 1873, 1892–1893, 1909, 1911–1916, 1919, 1944–1945, 1947–1949, 1966–1967, 1989, 1992, 1998–2000, 2005–2006, 2044, 2097, 2101, 2106, 2179–2180, 2191–2199, 2229–2231
telemetry.py	46	24	47%	45–49, 56, 58–60, 62, 69–82
src/backend/services
title_service.py	75	8	89%	39, 55–56, 75–77, 142–143
src/tests
test_app_title_endpoints.py	222	0	100%
test_llm_token_telemetry.py	102	0	100%
TOTAL	8471	589	93%

Tests	Skipped	Failures	Errors	Time
442	0 💤	0 ❌	0 🔥	12.728s ⏱️

[Copilot]

Pull request overview

Adds end-to-end LLM token usage telemetry for agent/workflow executions in the backend, plus Azure Monitor artifacts (workbook + KQL) to analyze usage by request, agent, model, and stage.

Changes:

• Added TokenUsageAccumulator + extraction helpers to capture token usage from Agent Framework responses/stream updates and emit LLM_*_Token_Usage App Insights custom events.
• Threaded user_id through orchestrator entrypoints and API handlers; added per-request ContextVar propagation to tag telemetry emitted from deeper helpers (e.g., image generation).
• Added standalone Bicep deployments for monitoring add-on resources and a “Token Usage” workbook, plus workbook JSON, KQL query pack, and docs.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/backend/token_usage.py	New module to extract/accumulate token counts and emit App Insights custom events.
src/backend/orchestrator.py	Creates/records/flushes token usage across workflow streaming, brief parsing, generation, and image paths; propagates user_id.
src/backend/app.py	Passes user_id into orchestrator calls for telemetry correlation.
infra/workbook/workbook.bicep	Standalone deployment of the Token Usage workbook targeting an App Insights resource (optional binding).
infra/workbook/README.md	Deployment instructions for the standalone workbook template.
infra/monitoring/monitoring.bicep	Standalone “add monitoring later” deployment (LA + App Insights).
infra/monitoring/README.md	Instructions for post-deploy monitoring enablement and wiring.
infra/dashboards/token-usage-workbook.json	Serialized workbook definition with tiles/queries for token usage analysis.
infra/dashboards/token-usage-queries.kql	KQL query pack for App Insights / Log Analytics.
docs/TokenUsageTelemetry.md	Documentation for emitted events, enabling telemetry, and querying/visualizing usage.
infra/main.bicep	Notes workbook is deployed separately; adds ACI tag hashing to force restart on monitoring config change.
infra/main_custom.bicep	Notes workbook deployed separately; adds ACI tag hashing; changes default gptModelCapacity.
infra/main.json	Recompiled ARM output with additional infra deltas beyond token telemetry.
.gitignore	Fixes rai_results ignore entry and adds Python coverage artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.