AssistantBench scorer treats a thousands separator as a decimal point: "3,080,000" scores 0 against gold 3080000, while "1,000" scores 1.0 against gold 1

[shaurya416]

browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluator.py:56-67 (the same replace(",", ".") is duplicated in evaluation/evaluate_utils/evaluate_dicts.py:46):

def fix_number(number):
    if type(number) == str:
        copy_ans = number
        copy_ans = " ".join(
            " ".join(" ".join(copy_ans.split("$")).split("%")).split("sqft")
        ).strip()
        copy_ans = copy_ans.strip()
        copy_ans = copy_ans.replace(",", ".").replace(" square kilometers", "")
        try:
            return float(copy_ans), True
        except:
            return number, False

The comma-to-dot rewrite is applied to both the prediction and the gold. It is correct for a European decimal comma (14,2 -> 14.2, which is dev gold validation.2), but a US thousands separator goes through the same path: 1,000 becomes 1.0, and 3,080,000 becomes 3.080.000, which float() rejects, so the prediction is passed to evaluate_numbers as a string and scored 0.

Measured

Real evaluation/evaluator.py loaded from the repo (only the package __init__ chain bypassed) and question_scorer called directly. numpy 2.3.1, scipy 1.16.0.

input	result	note
question_scorer('1000', '1000')	1.0	control, pass
question_scorer('2000', '1000')	0.307	control, log-distance partial credit as designed
question_scorer('$55', '$55')	1.0	control, dev gold validation.29
question_scorer('14,2', '14.2')	1.0	control, EU decimal comma, dev gold validation.2
question_scorer('3,080,000', '3080000')	0	correct answer, dev gold validation.26
question_scorer('1,010,000', '1010000')	0	correct answer, dev gold validation.3
question_scorer('1,000', '1000')	0	correct answer
question_scorer('1,000', '1')	1.0	wrong by 1000x
question_scorer('1,010', '1.01')	1.0	wrong by 1000x
fix_number('1,000')	(1.0, True)
fix_number('3,080,000')	('3,080,000', False)	falls through to evaluate_numbers -> float() ValueError -> 0

The two dev-split tasks above ("What's the lowest price a Single Family house was sold in Queen Anne in January 2023?", "What's the highest price a high-rise apartment was sold for in Mission Bay, San Francisco, in 2021?") ask for dollar amounts that Zillow and most models render with thousands separators, and neither task text asks for a comma-free number.

Consequence

AssistantBenchTask.validate returns this accuracy as the episode reward. A correctly answered task written as 3,080,000 counts as a 0 in cum_reward, and an answer that is off by three orders of magnitude but happens to carry a single comma counts as a full 1.0. The dict path is affected too: a JSON value "1,000" against gold 1000 scores the pair 0.5 instead of 1.0 through the duplicate in evaluate_dicts.py:46. task.py:140 feeds the raw chat message to question_scorer with no normalisation. None of the 33 rows pinned in tests/assistantbench/test_evaluation.py contain a digit-comma, so the test suite does not cover either direction.

This code is a verbatim copy of the official leaderboard evaluator (AssistantBench/leaderboard, evaluation/evaluator.py), so the hidden test split scored on the leaderboard has the same behaviour; the point of raising it here is that dev-split rewards computed inside BrowserGym are silently wrong for comma-formatted numbers, and a fix should probably be mirrored upstream so the two scorers stay aligned.

Suggested fix

Disambiguate the comma before the float() attempt instead of rewriting it unconditionally, e.g. in fix_number (and the duplicate in evaluate_dicts.py):

• if the string matches ^\d{1,3}(,\d{3})+(\.\d+)?$ (comma groups of exactly three digits, optional dot decimal), drop the commas;
• otherwise, if it contains exactly one comma and no dot, treat the comma as a decimal point (the existing 14,2 case);
• leave everything else as is.

The package already has a small numeric normalizer, evaluate_utils/evaluate_strings.py:_normalize_number / _is_number, so the rule could live there and be shared by fix_number in both files. Adding ('3,080,000', '3080000') -> 1.0, ('1,000', '1') -> 0, and ('14,2', '14.2') -> 1.0 to the pinned evaluation data would lock the behaviour in.

Happy to open the PR.