Skip to content

fix(tool-delegate): select the partial guidance string on the KIND of partial, not bool(text) - #356

Merged
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
lane/yiy-partial-guidance-kind
Sep 3, 2026
Merged

fix(tool-delegate): select the partial guidance string on the KIND of partial, not bool(text)#356
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
lane/yiy-partial-guidance-kind

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

Pairs with the producer half: app-cli 8c83a9b (microsoft/amplifier-app-cli PR #298).
That commit widened app-cli's partial accumulator so a timed-out delegate that emitted
no assistant text now recovers the agent's thinking and tool_call trace instead.
This PR is the consumer half, in this repo. Neither half is complete without the other.

The defect

modules/tool-delegate's _partial_output_fields picked its timeout guidance from
bool(text) alone. So a recovered-reasoning partial received the sentence written
for unfinished prose:

"INCOMPLETE: … The text in 'partial_response' is unfinished work salvaged from the
agent mid-flight — it has NOT been checked, concluded, or self-reviewed by that agent."

True of unfinished assistant prose. Not true of raw private reasoning: prose is at
least addressed to a reader; reasoning never was. Handing a model its own unreviewed
reasoning under that frame invites it to treat the reasoning as a draft answer — the one
thing it is not.

Before 8c83a9b this case could not occur. After it, it is the common one: measured on
k64's 18 legs, the recoverable window went 0.05% → 82.2% of a leg.

The change

One file, +64 / −2 (modules/tool-delegate/amplifier_module_tool_delegate/__init__.py):

  • _REASONING_PARTIAL_GUIDANCE — describes the payload as what it is: the agent's own
    private reasoning and tool-call trace, evidence of what it was doing and what it had
    looked at, never a draft answer.
  • _REASONING_PARTIAL_SOURCES = frozenset({"spawn-accumulator:reasoning"})
  • _guidance_for(text, source) — three cases, in order:
condition guidance
no text recovered _NO_PARTIAL_GUIDANCEbyte-identical
source == "spawn-accumulator:reasoning" _REASONING_PARTIAL_GUIDANCEnew
anything else (incl. unknown / absent / non-string source) _PARTIAL_GUIDANCEbyte-identical

Branches on partial_source. Never parses the prose — that is precisely what the
field exists to avoid.

Why exact match, not a :reasoning suffix test

The producer is a separate repo on its own release cadence, so a value this code has
never seen must be safe: it degrades to the incumbent behaviour rather than inheriting
a frame that may be wrong for it. The cost is stated rather than hidden — if app-cli ever
renames the source, the reasoning frame silently reverts to the prose frame. That
tradeoff is recorded in a source comment and in the lane note, not left to be
rediscovered.

source is typed Any and compared, never parsed, so a non-string value cannot raise
on the timeout path — the one path where raising discards every completed sibling in a
parallel batch.

Evidence

Fail-before → after, run against the parent commit 5d8db2f with the module
unchanged and only the new test file present:

=== FAIL-BEFORE (parent 5d8db2f, module UNCHANGED) ===
FAILED …::test_reasoning_partial_is_not_framed_as_unfinished_prose
FAILED …::test_reasoning_guidance_says_what_the_payload_actually_is
FAILED …::test_reasoning_kind_is_honoured_on_the_resume_path_too
3 failed, 16 passed in 0.31s

=== AFTER ===
19 passed in 0.26s

The 16 that already pass on the parent are the point of the design: they are the
byte-identity, no-partial, and unknown-source pins, and they must pass on both sides.
Exactly the 3 reasoning-kind tests move.

Byte-identity, checked against the parent blob rather than against this module's own
constants
(which would be tautological) — git show HEAD:… + ast.literal_eval, plus
the parent's selector re-run against this build:

_PARTIAL_GUIDANCE     parent sha256=b1d9796d1a9adf29 len=416 | now b1d9796d1a9adf29 416 | IDENTICAL
_NO_PARTIAL_GUIDANCE  parent sha256=d73f51f164c545d3 len=245 | now d73f51f164c545d3 245 | IDENTICAL

  text partial (spawn-accumulator)   unchanged-vs-parent = True
  text partial (source absent)       unchanged-vs-parent = True
  text partial (unknown source)      unchanged-vs-parent = True
  text partial (non-str source)      unchanged-vs-parent = True
  no partial                         unchanged-vs-parent = True
  reasoning partial (CHANGES)        unchanged-vs-parent = False

Exactly one case changed. app-cli's round-trip test
test_guidance_string_is_unchanged_for_the_text_case pins the same bytes from the
producer side, so a regression here fails there too. The new test file spells both
incumbent strings out as literals rather than importing the constants, so a reword
fails the test instead of silently redefining "unchanged".

Test suite

Run Result
uv run pytest -q — before (parent 5d8db2f) 1939 passed, 1 skipped
uv run pytest -q — this branch 1958 passed, 1 skipped
uv run pytest tests/ -q --tb=short (what CI runs) 1769 passed, 1 skipped

+19 = exactly the new tests; zero pre-existing tests changed state.

Test placement is deliberate. The new file is tests/test_partial_guidance_kind_yiy.py,
not modules/tool-delegate/tests/ — CI runs pytest tests/ only, so a test placed only
in the module directory would never run there.

Scope

  • timeout_msg at both call sites (:2276, :2676) is unchanged. It says "it is
    UNFINISHED, not a result", which is true of reasoning too and does not carry the
    offending "not checked, concluded, or self-reviewed" clause. Not the defect; not
    touched.
  • amplifier-app-cli is not modified. Its half is merged; partial_source is this
    repo's input.
  • No behaviour change for a normal completion, for a text partial, or for a timeout with
    nothing recovered.

Lane note with the full record: docs/lanes/yiy-partial-guidance-kind/DONE-NOTE.md.
Refs model_performance-yiy. Spend: $0.00.

…ot bool(text)

`_partial_output_fields` picked its timeout guidance from `bool(text)` alone,
so a partial recovered from the REASONING channel got the sentence written for
unfinished prose:

    "the text in 'partial_response' is unfinished work salvaged from the agent
     mid-flight -- it has NOT been checked, concluded, or self-reviewed"

True of unfinished assistant prose. False of raw private reasoning, which was
never addressed to a reader at all -- and framing it as unreviewed draft output
invites the calling model to read it as a draft answer.

Reachable only since app-cli 8c83a9b (PR #298) widened the accumulator to
recover `thinking` + `tool_call` traces when no assistant text exists (k64:
recoverable window 0.05% -> 82.2% of a leg). That half is the producer; this is
the consumer.

Branches on `partial.source`, never on the prose:
  no text                      -> _NO_PARTIAL_GUIDANCE   (byte-identical)
  "spawn-accumulator:reasoning"-> _REASONING_PARTIAL_GUIDANCE (new)
  anything else                -> _PARTIAL_GUIDANCE      (byte-identical)

Exact match, deliberately: an unknown or non-string source degrades to the
incumbent behaviour rather than inheriting a frame that may be wrong for it.
`source` is compared, never parsed, so it cannot raise on the timeout path --
the one path where raising discards every completed sibling in a batch.

Byte-identity verified against the parent blob, not against this module's own
constants: _PARTIAL_GUIDANCE sha256 b1d9796d1a9adf29 (416 B) and
_NO_PARTIAL_GUIDANCE sha256 d73f51f164c545d3 (245 B) are unchanged, and the
parent's selector re-run against this build agrees on every case except the
reasoning one.

Tests land in tests/ (not modules/tool-delegate/tests/) because CI runs
`pytest tests/` only. Fail-before on 5d8db2f: 3 failed / 16 passed; after:
19 passed. Full suite 1939 -> 1958 passed, 1 skipped.

Refs: model_performance-yiy
@bkrabach
Brian Krabach (bkrabach) marked this pull request as ready for review September 3, 2026 14:03
@bkrabach

Copy link
Copy Markdown
Collaborator Author

Manager verification — byte-identity confirmed from the parent blob, and the frame is honest. Merging.

Head 8205f4e, base 5d8db2fa = current origin/main. CI green on all 7 jobs.

Byte-identity — I extracted both constants from the parent blob myself

Not from the module's own constants, and not from the marker:

_PARTIAL_GUIDANCE      parent=e53d108f002319d1  head=e53d108f002319d1  IDENTICAL  (459B)
_NO_PARTIAL_GUIDANCE   parent=34f0eb068e34130c  head=34f0eb068e34130c  IDENTICAL  (274B)
_REASONING_PARTIAL_GUIDANCE:  present in head, absent in parent

So the text case and the no-partial case are provably unmoved, and app-cli's test_guidance_string_is_unchanged_for_the_text_case asserts the same bytes from the producer side — the two repos now pin each other.

The new frame says what the content actually is

"…What is in 'partial_response' is the agent's own private reasoning and the trace of the tool calls it made — evidence of what it was doing and what it had looked at. It was never addressed to a reader and is never a draft answer, so do not quote it, summarize it as a result, or treat any statement in it as a conclusion. Use it only to decide what to do next…"

This is the judgement I most wanted to check, and it is right. It names the content correctly (private reasoning + tool-call trace), states the negative claim explicitly rather than implying it (never a draft answer; do not quote, summarise, or treat as a conclusion), and gives the correct positive affordance — use it to choose the next move, informed by what was already covered. It does not smuggle in the old sentence's implication that this is unfinished work product. Nothing in it overstates what a killed agent's scratch reasoning is.

Safe degradation, which the goal required and is easy to get wrong

if not text: return _NO_PARTIAL_GUIDANCE
if isinstance(source, str) and source in _REASONING_PARTIAL_SOURCES:
    return _REASONING_PARTIAL_GUIDANCE
return _PARTIAL_GUIDANCE

Exact-match membership behind an isinstance guard; source is typed Any on purpose because it arrives from another repo and is "compared, never parsed, so a non-string can never raise here." An unknown or absent source falls back to the byte-identical text frame rather than getting the reasoning one — tested across 11 parametrised values including None, int, list, dict and the absent-key case. That is the right default: a value this code has never seen must not inherit the stronger claim.

Gates

gate result
fail-before vs current main 3 failed, 16 passed
full suite 1958 passed, 1 skipped (main 1939 → +19 = exactly the new tests)
CI green, 7/7

The three failures are precisely the reasoning-frame assertions — including test_reasoning_kind_is_honoured_on_the_resume_path_too, which is the one I would have forgotten: the resume path builds its own result and would otherwise have kept the prose frame.

Pairs with app-cli 8c83a9b (the producer half). Squash + --admin per the base-branch policy.

@bkrabach
Brian Krabach (bkrabach) merged commit 52cbf74 into main Sep 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant