fix(scoring): BBH exact-match accepts Markdown-emphasised answers (**True** == True) - #463
Open
RealDiligent wants to merge 2 commits into
Open
fix(scoring): BBH exact-match accepts Markdown-emphasised answers (**True** == True)#463RealDiligent wants to merge 2 commits into
RealDiligent wants to merge 2 commits into
Conversation
…ent of James-CUDA#450) main CI is red: tests/test_drop_leading_decimal_point.py pins $.5 / .5. forms that still lose the leading point on the second-chance strip path, so every PR inherits the failure. This commit applies the same minimal repair PR James-CUDA#450 proposes (exclude "." from _STRIP_EDGE, retry a failed parse with rstrip(".")) purely so CI can validate this branch's own fix. Credit for the repair belongs to James-CUDA#450; once it lands on main this resolves to a no-op in the merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…True** == True) Bolding the final answer is a very common model format — common enough that the shared choice extractor grew _strip_choice_md_emphasis for it, and the multiple-choice half of this adapter accepts "Answer: **B**". But _normalize_exact's strip set had no */_, so "Answer: **True**" normalised to "**true**" and every exact-match subtask (boolean_expressions, formal_fallacies, navigate, sports_understanding, object_counting, ...) graded the bolded correct answer 0.0 — an inconsistent false negative inside one adapter. Fix: peel Markdown emphasis only when the SAME marker wraps the whole answer (backreference; nested wrappers collapse via fixpoint loop), once before the dyck bracket-sequence check (so a bolded bracket answer still compares as a sequence) and once after the edge strip (so "**true**." unwraps). Asymmetric or inner-word markers are content and never peeled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
General improvement
What does this PR do?
Fixes #461.
BBH's exact-match half zeroes a Markdown-bolded correct answer while the multiple-choice half of the SAME adapter accepts one:
Answer: TrueTrueAnswer: **True**TrueAnswer: **valid**validAnswer: **False**TrueRoot cause:
_normalize_exact's strip set has no*/_, so**True**normalizes to**true**≠true. The repo already established that bolding the final answer "is a very common model format" (reward._strip_choice_md_emphasis), soAnswer: **B**passes MCQ subtasks whileAnswer: **True**fails all ~10 exact-match subtasks — an inconsistent false negative inside one adapter.Fix approach (minimal): peel Markdown emphasis only when the SAME marker wraps the whole answer (backreference; nested
**_True_**collapses via a fixpoint loop), applied before the dyck bracket-sequence check (a bolded bracket answer still compares as a sequence) and once more after the edge strip (so**True**.unwraps). Asymmetric markers (2*3) and inner-word emphasis are content and never peeled — pinned by tests.Why is it needed?
Systematic false negatives on ~10 of BBH's 27 subtasks (boolean_expressions, formal_fallacies, navigate, sports_understanding, object_counting, ...) for one of the most common model output styles, and intra-adapter consistency with the MCQ path.
Tests:
tests/test_bbh_md_emphasis.py(9 cases): bold/italic/underscore/nested unwrap, emphasis inside terminal punctuation, multiword answers, asymmetric-marker guard, dyck sequences (plain and bolded), and end-to-end grading both directions.Note on CI: current
mainis red by itself —tests/test_drop_leading_decimal_point.py(3 failures) pins wrapped leading-decimal forms; open PR #450 is the dedicated repair. Like #420/#424 did for #404, this branch carries onechore(ci)commit applying the equivalent minimal repair (credit belongs to #450) purely so CI can validate this fix; it resolves to a no-op once #450 lands, and I will rebase it away on request. With it, the full suite passes locally:2148 passed, 2 skipped(+9 new).Checklist
pytest tests/(2148 passed, 2 skipped locally; 3 Windows-only console-encoding failures unrelated to this change pass on Linux)ruff check src/mypy src/(for changed files)🤖 Generated with Claude Code