fix(sensor): attribute MCP servers and failed tool calls in Claude Code sessions - #38
fix(sensor): attribute MCP servers and failed tool calls in Claude Code sessions#38aglanio wants to merge 1 commit into
Conversation
…de sessions The Claude Code parser hardcoded `tool_type="tool_use"` for every tool call, never populated `server_name`, and dropped the `is_error` flag from tool results. On a real corpus of 1,814 sessions / 90,626 tool calls this meant: - every call landed in a single `tool_use` bucket, so terminal commands were indistinguishable from MCP calls; - no MCP server attribution at all, even though Claude Code namespaces MCP tools as `mcp__<server>__<tool>` and the server is recoverable from the name; - zero errors reported across all 90,626 calls, because `is_error` was never read. After the fix, the same corpus yields `terminal_command` 48,008 / `tool_use` 24,617 / `mcp_tool` 18,001, 15 distinct MCP servers, and 5,327 failed calls (5.9%). Failed and blocked tool calls matter for detection, so losing them silently is a meaningful gap. This mirrors what `opencode_parser._classify_tool()` already does for opencode; the Claude Code parser had simply not received the same treatment. Also preserves `server_name` across the tool_use -> tool_result merge, which previously dropped it. Adds three tests: tool classification, server attribution surviving the result merge, and `is_error` producing `status="error"`.
|
|
pengyuzhang
left a comment
There was a problem hiding this comment.
Reviewed at commit 83bda7c. The premise checks out: is_error really does live on the tool_result content block (658 of 1296 blocks carry the key, 27 of them true in a ~1,300-call sample from ~/.claude/projects), and the mcp__<server>__<tool> split handles names containing extra underscores, including the degenerate mcp____x. Test suite passes (76).
Three inline comments below — one of them a regression this PR introduces, so worth resolving before merge.
Two more findings that fall outside the diff, so they can't be anchored inline:
Sensor/adr_sensor/parsers/claude_desktop_parser.py:382 — the fix is half-applied across sources. Claude Desktop agent mode is the most MCP-heavy source in the repo, yet its tool path still hardcodes tool_type="tool_use" with no server_name, and _attach_tool_result (~line 411) still does status="success" if result else "unknown" with no is_error handling and no error field. Every third-party MCP connector call from Claude Desktop stays in the unattributed bucket, and every failed call is still recorded as a success — the exact two gaps this PR closes for Claude Code.
Sensor/README.md:49 — undocumented behavior change. The README documents the per-source MCP namespacing rule for opencode (<server>_<tool> → mcp_tool + server_name) but has no equivalent note for Claude Code, and the new Bash/BashOutput/KillShell → terminal_command reclassification is undocumented. Anyone reading the schema comment listing tool_type values has no way to know which ones Claude Code now emits.
Checked and cleared: _classify_tool handles mcp__a_b___tool correctly; toolUseResult is a plain string ("Error: …") on every error case in the sampled logs, so the dict-only override at line 200 doesn't swallow error text; nothing in Detection/ reads tool_type and get_content_hash has no non-test callers, so the terminal_command reclassification breaks no downstream consumer or event dedup.
| tools = [] | ||
|
|
||
| for tool_data in msg_data.get("tools", []): | ||
| tool_name = tool_data.get("name", "unknown") |
There was a problem hiding this comment.
A tool_use block with "name": null now drops the entire session.
.get("name", "unknown") only defaults when the key is absent — a present-but-null name yields None, and _classify_tool(None) raises AttributeError on None.startswith("mcp__"). That escapes to the broad except Exception in _create_entry_from_extracted_session, which prints and returns None, so the whole session — every user prompt and every other tool call in it — is silently discarded from telemetry. On main the same input parsed fine, since tool_name=None was simply stored.
Reproduced locally: the parser prints [CLAUDE] Error creating entry for session s1 and returns [].
| tool_name = tool_data.get("name", "unknown") | |
| tool_name = tool_data.get("name") or "unknown" |
An isinstance(tool_name, str) guard at the top of _classify_tool would also cover the other call sites.
| if is_error: | ||
| status = "error" | ||
| elif result: | ||
| status = "success" | ||
| else: | ||
| status = "unknown" |
There was a problem hiding this comment.
status="unknown" is provably wrong for empty non-error results.
A tool_result block is the completion signal, and is_error is authoritative — anything with falsy is_error succeeded, whether or not its content normalizes to a non-empty string.
Measured against ~1,300 real tool calls in ~/.claude/projects: 12 come out as unknown, all of them successful ToolSearch calls whose result content normalizes to "", each carrying an explicit is_error: false in the log. Collapsing the branch also matches the stated point of the PR:
| if is_error: | |
| status = "error" | |
| elif result: | |
| status = "success" | |
| else: | |
| status = "unknown" | |
| status = "error" if is_error else "success" |
| for msg in entry.chat_history: | ||
| if msg.role == "assistant": |
There was a problem hiding this comment.
The result merge two lines below matches by value, not identity — and this PR makes the misattribution worse.
ToolUsage is @dataclass(frozen=True), so two distinct pending calls with the same name/type/arguments compare equal while both still have result=None. if t == old_tool then hits whichever comes first, not the one this result belongs to.
Concrete failure: an assistant message issues two parallel Bash calls with the same command, and their tool_results complete out of order (the erroring one first). The new status="error" and error=… land on the call that actually succeeded, and the success lands on the one that failed. Before this PR that only swapped result strings; now it mislabels which call failed. Value-identical repeated calls do occur in practice — 4 identical ToolSearch invocations in a ~1,300-call sample.
The sibling parser already fixes exactly this, at claude_desktop_parser.py:428:
# Match on identity: two identical calls in different messages compare
# equal, and only the one this result belongs to should change.
if tool is not old_tool:
continueSeparately, the inner break exits only the for idx, t in enumerate(msg.tools) loop, not the enclosing for msg in entry.chat_history, so a single result can be applied in more than one message.
Problem
The Claude Code parser loses three signals that matter for downstream detection:
tool_typeis hardcoded to"tool_use"for every tool call (claude_parser.py#L279), so terminal commands and MCP calls end up in the same bucket.server_nameis never populated, even though Claude Code namespaces MCP tools asmcp__<server>__<tool>and the server is recoverable from the name alone.is_errorflag ontool_resultis never read, so every call is recorded assuccessorunknown— failed and blocked tool calls disappear.Measured on a real corpus of 1,814 sessions / 90,626 tool calls:
tool_typetool_use90,626 (single bucket)terminal_command48,008 ·tool_use24,617 ·mcp_tool18,001Bash(2,584)Fix
Adds
ClaudeParser._classify_tool(), mirroring whatopencode_parser._classify_tool()already does for opencode — the Claude Code parser had simply not received the same treatment. Also:server_nameacross thetool_use->tool_resultmerge, which previously dropped it even when set;is_errorfrom the tool result and maps it tostatus="error"plus theerrorfield.Tests
Three new tests in
TestClaudeParser: tool classification, server attribution surviving the result merge, andis_errorproducingstatus="error".pytest testsgoes from 118 to 121 passing. The 5TestPlatformPathCoveragefailures in my environment are pre-existing and Windows-specific (confirmed unchanged with the patch stashed) — untouched by this PR.