Skip to content

fix(sensor): attribute MCP servers and failed tool calls in Claude Code sessions - #38

Open
aglanio wants to merge 1 commit into
uber:mainfrom
aglanio:fix/claude-parser-mcp-attribution-and-errors
Open

fix(sensor): attribute MCP servers and failed tool calls in Claude Code sessions#38
aglanio wants to merge 1 commit into
uber:mainfrom
aglanio:fix/claude-parser-mcp-attribution-and-errors

Conversation

@aglanio

@aglanio aglanio commented Aug 10, 2026

Copy link
Copy Markdown

Problem

The Claude Code parser loses three signals that matter for downstream detection:

  1. tool_type is hardcoded to "tool_use" for every tool call (claude_parser.py#L279), so terminal commands and MCP calls end up in the same bucket.
  2. server_name is never populated, even though Claude Code namespaces MCP tools as mcp__<server>__<tool> and the server is recoverable from the name alone.
  3. The is_error flag on tool_result is never read, so every call is recorded as success or unknown — failed and blocked tool calls disappear.

Measured on a real corpus of 1,814 sessions / 90,626 tool calls:

before after
tool_type tool_use 90,626 (single bucket) terminal_command 48,008 · tool_use 24,617 · mcp_tool 18,001
MCP servers attributed 0 15 distinct servers
errors detected 0 5,327 (5.9%), led by Bash (2,584)

Fix

Adds ClaudeParser._classify_tool(), mirroring what opencode_parser._classify_tool() already does for opencode — the Claude Code parser had simply not received the same treatment. Also:

  • preserves server_name across the tool_use -> tool_result merge, which previously dropped it even when set;
  • reads is_error from the tool result and maps it to status="error" plus the error field.

Tests

Three new tests in TestClaudeParser: tool classification, server attribution surviving the result merge, and is_error producing status="error".

pytest tests goes from 118 to 121 passing. The 5 TestPlatformPathCoverage failures in my environment are pre-existing and Windows-specific (confirmed unchanged with the patch stashed) — untouched by this PR.

…de sessions

The Claude Code parser hardcoded `tool_type="tool_use"` for every tool call, never
populated `server_name`, and dropped the `is_error` flag from tool results. On a real
corpus of 1,814 sessions / 90,626 tool calls this meant:

- every call landed in a single `tool_use` bucket, so terminal commands were
  indistinguishable from MCP calls;
- no MCP server attribution at all, even though Claude Code namespaces MCP tools as
  `mcp__<server>__<tool>` and the server is recoverable from the name;
- zero errors reported across all 90,626 calls, because `is_error` was never read.

After the fix, the same corpus yields `terminal_command` 48,008 / `tool_use` 24,617 /
`mcp_tool` 18,001, 15 distinct MCP servers, and 5,327 failed calls (5.9%). Failed and
blocked tool calls matter for detection, so losing them silently is a meaningful gap.

This mirrors what `opencode_parser._classify_tool()` already does for opencode; the
Claude Code parser had simply not received the same treatment. Also preserves
`server_name` across the tool_use -> tool_result merge, which previously dropped it.

Adds three tests: tool classification, server attribution surviving the result merge,
and `is_error` producing `status="error"`.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@pengyuzhang pengyuzhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at commit 83bda7c. The premise checks out: is_error really does live on the tool_result content block (658 of 1296 blocks carry the key, 27 of them true in a ~1,300-call sample from ~/.claude/projects), and the mcp__<server>__<tool> split handles names containing extra underscores, including the degenerate mcp____x. Test suite passes (76).

Three inline comments below — one of them a regression this PR introduces, so worth resolving before merge.

Two more findings that fall outside the diff, so they can't be anchored inline:

Sensor/adr_sensor/parsers/claude_desktop_parser.py:382 — the fix is half-applied across sources. Claude Desktop agent mode is the most MCP-heavy source in the repo, yet its tool path still hardcodes tool_type="tool_use" with no server_name, and _attach_tool_result (~line 411) still does status="success" if result else "unknown" with no is_error handling and no error field. Every third-party MCP connector call from Claude Desktop stays in the unattributed bucket, and every failed call is still recorded as a success — the exact two gaps this PR closes for Claude Code.

Sensor/README.md:49 — undocumented behavior change. The README documents the per-source MCP namespacing rule for opencode (<server>_<tool>mcp_tool + server_name) but has no equivalent note for Claude Code, and the new Bash/BashOutput/KillShellterminal_command reclassification is undocumented. Anyone reading the schema comment listing tool_type values has no way to know which ones Claude Code now emits.

Checked and cleared: _classify_tool handles mcp__a_b___tool correctly; toolUseResult is a plain string ("Error: …") on every error case in the sampled logs, so the dict-only override at line 200 doesn't swallow error text; nothing in Detection/ reads tool_type and get_content_hash has no non-test callers, so the terminal_command reclassification breaks no downstream consumer or event dedup.

tools = []

for tool_data in msg_data.get("tools", []):
tool_name = tool_data.get("name", "unknown")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A tool_use block with "name": null now drops the entire session.

.get("name", "unknown") only defaults when the key is absent — a present-but-null name yields None, and _classify_tool(None) raises AttributeError on None.startswith("mcp__"). That escapes to the broad except Exception in _create_entry_from_extracted_session, which prints and returns None, so the whole session — every user prompt and every other tool call in it — is silently discarded from telemetry. On main the same input parsed fine, since tool_name=None was simply stored.

Reproduced locally: the parser prints [CLAUDE] Error creating entry for session s1 and returns [].

Suggested change
tool_name = tool_data.get("name", "unknown")
tool_name = tool_data.get("name") or "unknown"

An isinstance(tool_name, str) guard at the top of _classify_tool would also cover the other call sites.

Comment on lines +275 to +280
if is_error:
status = "error"
elif result:
status = "success"
else:
status = "unknown"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

status="unknown" is provably wrong for empty non-error results.

A tool_result block is the completion signal, and is_error is authoritative — anything with falsy is_error succeeded, whether or not its content normalizes to a non-empty string.

Measured against ~1,300 real tool calls in ~/.claude/projects: 12 come out as unknown, all of them successful ToolSearch calls whose result content normalizes to "", each carrying an explicit is_error: false in the log. Collapsing the branch also matches the stated point of the PR:

Suggested change
if is_error:
status = "error"
elif result:
status = "success"
else:
status = "unknown"
status = "error" if is_error else "success"

Comment on lines 290 to 291
for msg in entry.chat_history:
if msg.role == "assistant":

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The result merge two lines below matches by value, not identity — and this PR makes the misattribution worse.

ToolUsage is @dataclass(frozen=True), so two distinct pending calls with the same name/type/arguments compare equal while both still have result=None. if t == old_tool then hits whichever comes first, not the one this result belongs to.

Concrete failure: an assistant message issues two parallel Bash calls with the same command, and their tool_results complete out of order (the erroring one first). The new status="error" and error=… land on the call that actually succeeded, and the success lands on the one that failed. Before this PR that only swapped result strings; now it mislabels which call failed. Value-identical repeated calls do occur in practice — 4 identical ToolSearch invocations in a ~1,300-call sample.

The sibling parser already fixes exactly this, at claude_desktop_parser.py:428:

# Match on identity: two identical calls in different messages compare
# equal, and only the one this result belongs to should change.
if tool is not old_tool:
    continue

Separately, the inner break exits only the for idx, t in enumerate(msg.tools) loop, not the enclosing for msg in entry.chat_history, so a single result can be applied in more than one message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants