fix: Token fix acount Token count Monitor by Ashwal-Microsoft · Pull Request #945 · microsoft/Conversation-Knowledge-Mining-Solution-Accelerator

Ashwal-Microsoft · 2026-06-02T16:41:27Z

Purpose

Implement comprehensive token usage tracking across all LLM call sites

Does this introduce a breaking change?

Yes
No

Golden Path Validation

I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

I have validated the deployment process successfully and all services are running as expected with this change.

…, teams, and models - Add token_usage_utils.py with extraction and emission utilities - Integrate token tracking into chat_service.py streaming flow - Add KQL queries and Azure Monitor workbook for dashboards - Add unit tests (27 tests) for token usage utilities - Add AZURE_OPENAI_MODEL_DEPLOYMENT and TEAM_NAME env vars Tracks per-agent, per-user, per-team, and per-model token consumption to Application Insights for monitoring, cost estimation, and optimization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This pull request introduces a cross-accelerator token-usage telemetry module and wires it into key backend LLM call sites so token usage can be emitted as standardized Application Insights custom events (plus adds supporting sample/config/dashboard assets).

Changes:

Added common.logging.llm_token_telemetry with token extraction helpers, an emitter, and a scope/decorator for consistent event emission.
Introduced a process-wide token_emitter singleton (src/api/telemetry.py) and integrated token tracking into chat streaming and title generation.
Added supporting artifacts for monitoring and sample data (KQL queries, infra parameter, sample transcripts/SQL inserts) and corresponding tests.

Reviewed changes

Copilot reviewed 15 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/tests/api/common/logging/test_llm_token_telemetry.py	New unit tests for token telemetry helpers/emitter/scope.
src/api/telemetry.py	Adds a process-wide `TokenUsageEmitter` singleton configured via env vars.
src/api/services/history_service.py	Emits token usage for the title-generation agent run.
src/api/services/chat_service.py	Tracks/accumulates token usage across streaming agent chunks and emits telemetry.
src/api/common/logging/llm_token_telemetry.py	New core telemetry implementation: extraction, event emission, scope/decorator.
src/api/.env.sample	Adds env placeholders for token-tracking related settings.
infra/scripts/index_scripts/sql_files/processed_new_key_phrases.sql	Adds SQL insert script content for processed key phrases (sample data).
infra/scripts/index_scripts/sql_files/processed_data_batch_insert.sql	Adds batch insert SQL for processed conversation records (sample data).
infra/main.parameters.json	Adds `enableMonitoring` parameter substitution for deployments.
infra/dashboards/token-usage-queries.kql	Adds ready-to-run App Insights KQL queries for token-usage monitoring/cost estimation.
call_transcripts/convo_*.json	Adds sample call transcript JSON files used by data processing flows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    in_details = _get(usage, "input_token_details") or {}
+    out_details = _get(usage, "output_token_details") or {}
+
+    record = TokenUsage(
+        input_tokens=inp,
+        output_tokens=out,
+        total_tokens=tot,
+        input_audio_tokens=_to_int(_get(in_details, "audio_tokens")),
+        input_text_tokens=_to_int(_get(in_details, "text_tokens")),
+        input_cached_tokens=_to_int(_get(in_details, "cached_tokens")),
+        output_audio_tokens=_to_int(_get(out_details, "audio_tokens")),
+        output_text_tokens=_to_int(_get(out_details, "text_tokens")),
+    )


+# Token usage tracking configuration
+AZURE_OPENAI_MODEL_DEPLOYMENT=
+TEAM_NAME=


+        self._log.info(
+            "[TOKEN USAGE] agent=%s model=%s input=%d output=%d total=%d %s",
+            agent_name,
+            model_deployment_name,
+            usage.input_tokens,
+            usage.output_tokens,
+            usage.total_tokens,
+            " ".join(f"{k}={v}" for k, v in dimensions.items() if v),
+        )


- Use TokenUsageScope as context manager (with statement) instead of manual __exit__ call to guarantee emission on all exit paths - Fix extract_realtime_usage to preserve None for missing optional token detail fields instead of coercing to 0 - Remove redundant double extraction in TokenUsageScope.add() since extract_usage_from_stream_chunk already calls extract_usage internally - Hash user_id in emit_all() log statement to prevent leaking raw IDs - Remove unused 'patch' import from test module - Add missing LLM_TOKEN_SAMPLE_RATE, LLM_TOKEN_USER_ID_HMAC_KEY, and LLM_TOKEN_PRICING to .env.sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-03T10:08:18Z

Coverage Report •

File	Stmts	Miss	Cover	Missing
src/api
telemetry.py	46	24	47%	39–43, 50, 52–54, 56, 63–76
src/api/common/logging
llm_token_telemetry.py	422	144	65%	103, 108, 137, 165, 176–182, 210, 229, 248–261, 276, 285–287, 289–293, 295–296, 298, 300–302, 304, 314, 323–324, 332–348, 362–363, 413–414, 419–425, 429, 445, 450–454, 459–461, 468–473, 484, 486–488, 498, 513–514, 523, 530–532, 540–542, 554, 572, 589, 605, 625, 648–650, 676, 722, 790–792, 800, 803–804, 830–831, 857, 859–860, 862–869, 871–877, 879–880, 885, 895
src/api/services
chat_service.py	191	25	86%	64–65, 220–222, 225–234, 261–264, 268, 271, 282–283, 303–304
history_service.py	213	24	88%	110, 241–242, 244, 281–283, 299, 305–307, 324, 340–341, 343, 359, 385–386, 388, 404, 424–425, 427, 446
TOTAL	1808	333	81%

Tests	Skipped	Failures	Errors	Time
191	0 💤	0 ❌	0 🔥	7.071s ⏱️

Copilot

Pull request overview

Copilot reviewed 15 out of 23 changed files in this pull request and generated 9 comments.

+    "enableMonitoring": {
+      "value": "${enableMonitoring}"
    }


+        safe_dims = dict(dimensions)
+        if "user_id" in safe_dims:
+            safe_dims["user_id"] = self._apply_user_id_hash(safe_dims["user_id"])
+


- Fix duplicate/conflicting imports in history_service.py (consolidated to single import line with get_azure_credential_async and build_async_azure_credential, removed unused get_azure_credential) - Fix enableMonitoring parameter to use azd-compatible env var pattern with default value (AZURE_ENV_ENABLE_MONITORING=false) - Strip user_id from logs entirely when HMAC hasher is not configured to prevent PII leakage in application logs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…iles These were accidentally committed alongside the token telemetry feature. They are not part of the token monitoring fix scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

These were accidentally included in commit caabe82 and are not part of the token monitoring fix scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

The fallback hasattr(__iter__) check does accept arbitrary iterables (excluding str/bytes/Mapping), so update the docstring accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ashwal-Microsoft and others added 4 commits May 14, 2026 15:09

Fix token input tracking and commit all pending workspace updates

caabe82

fix:Token count Monitor

d5d79e6

fix: resolve flake8 failures in PyLint workflow

9c48575

Copilot AI review requested due to automatic review settings June 2, 2026 16:41

Ashwal-Microsoft requested review from Avijit-Microsoft, Prajwal-Microsoft, Roopan-Microsoft, Vinay-Microsoft, aniaroramsft, brittneek, dgp10801, nchandhi and toherman-msft as code owners June 2, 2026 16:41

Copilot started reviewing on behalf of Ashwal-Microsoft June 2, 2026 16:41 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Ashwal-Microsoft and others added 2 commits June 2, 2026 22:24

Merge branch 'dev' into Token-fix-Acount

8d7f29e

Copilot AI review requested due to automatic review settings June 3, 2026 10:06

Ashwal-Microsoft temporarily deployed to production June 3, 2026 10:07 — with GitHub Actions Inactive

Copilot started reviewing on behalf of Ashwal-Microsoft June 3, 2026 10:07 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Ashwal-Microsoft temporarily deployed to production June 3, 2026 10:45 — with GitHub Actions Inactive

chore: remove unrelated audio_data, call_transcripts, and dashboard f…

8e86465

…iles These were accidentally committed alongside the token telemetry feature. They are not part of the token monitoring fix scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 3, 2026 12:28

Ashwal-Microsoft temporarily deployed to production June 3, 2026 12:28 — with GitHub Actions Inactive

Copilot started reviewing on behalf of Ashwal-Microsoft June 3, 2026 12:28 View session

chore: remove unrelated SQL seed files from token telemetry PR

ba1f434

These were accidentally included in commit caabe82 and are not part of the token monitoring fix scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ashwal-Microsoft temporarily deployed to production June 3, 2026 12:30 — with GitHub Actions Inactive

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread src/api/common/logging/llm_token_telemetry.py Outdated

fix: update _is_iterable docstring to match implementation

04711b2

The fallback hasattr(__iter__) check does accept arbitrary iterables (excluding str/bytes/Mapping), so update the docstring accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ashwal-Microsoft temporarily deployed to production June 3, 2026 12:37 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Token fix acount Token count Monitor#945

fix: Token fix acount Token count Monitor#945
Ashwal-Microsoft wants to merge 10 commits into
devfrom
Token-fix-Acount

Ashwal-Microsoft commented Jun 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Ashwal-Microsoft commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Does this introduce a breaking change?

Golden Path Validation

Deployment Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ashwal-Microsoft commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 3, 2026 •

edited

Loading