runner: strict response comparison mode (PR-S)

[dmitriy-b]

Summary

compare gains --strict-response-comparison, for callers replaying recorded traffic against two builds of the same client, where any byte of difference is a finding. It removes three implicit normalizations that no rule could switch off, and records what the block override actually put on the wire. Nothing changes when the flag is absent — the default path is byte-for-byte the behaviour on main, and explicit ignore / numeric_tolerance rules and the built-in eth_estimateGas tolerance behave identically in both modes.

This is "PR-S" in the Nethermind RPC differential-testing plan; NethermindEth/rpc-corpus-tools already holds the executable specification for it (tests/test_strict_comparator.py, tests/test_wire_replay.py), where these cases are xfail(strict=True) against this change.

Default	Under --strict-response-comparison
"0x" equals any all-zero hex string (0x0, 0x00, 0x00…00), in result and inside error.data, before any rule is consulted	They are different values. A client that omits a value and one that returns zero are not giving the same answer
JSON numbers decode as float64, so two integers above 2⁵³ that differ can compare equal	Decoded with UseNumber and compared by their literal digits
--block-override adds fromBlock/toBlock to an eth_getLogs filter carrying blockHash, turning a single-block query into a range query	A filter carrying blockHash is sent unchanged

comparison-provenance.json gains strict_response_comparison and wire_transformations — the original and effective params of every call the block override rewrote, written in both modes. The result artifact carries the params the caller supplied, so without this the request the answers actually belong to was recorded nowhere.

One non-obvious consequence handled here: under UseNumber an error code is a json.Number, not a float64. Left alone, errorCode and classifyError would read every strict-mode error as code 0 and quietly move the environment/real difference split. numericCode reads both shapes and is covered by a test.

Task	Delivered	State
PR-S (a) empty/zero-hex equivalence	comparator/diff.go, comparator/rules.go (diffContext.strict)	Passed
PR-S (b) exact integers above 2⁵³	comparator/transport.go (decodeResponse), diff.go (json.Number case), rules.go (numericCode)	Passed
PR-S (c) eth_getLogs blockHash guard	comparator/block_override.go	Passed
PR-S (d) original + effective wire params	comparator/comparator.go (wireTransform, Provenance)	Passed
Flag wiring + docs	cmd/compare.go, .claude/skills/compare-nodes/references/flags.md	Passed

CI

This repository runs no Go test workflow (build_docker_image.yml and codeql.yml only), so the evidence is the local runs below.

Verification

go build ./...                       # clean
go vet ./runner/...                  # clean
go test ./runner/comparator/         # ok — 62 tests pass (53 on main)
go test ./runner/comparator/ -race   # ok
go test ./runner/...                 # unchanged otherwise: `runner` still fails
                                     # TestP99DataFlow (needs postgres on :5432), on main too

Cross-repo, against the consumer's executable spec (rpc-corpus-tools at 0bc6e31, this branch's binary via RPC_CORPUS_JSON_BENCH_BIN):

# default mode — the whole consumer suite, unchanged
1234 passed, 7 skipped, 5 xfailed in 160.28s

# with --strict-response-comparison
tests/test_strict_comparator.py + tests/test_wire_replay.py:
  5 × XPASS(strict) — the four "0x" vs all-zero-hex mutations and the 2⁵³ case
                      are now caught, which is what this PR is for
  1 failure — TestConsumerHazardsTheProducerGuardsAgainst::
              test_block_hash_only_filter_acquires_a_range_under_override,
              which asserts the hazard this flag removes

Those six are the consumer's signal that the change landed; they are updated on the rpc-corpus-tools side when it bumps its pin to this commit.

🤖 Generated with Claude Code

[dmitriy-b]

Note on the red Build and publish Docker image check — it is not caused by this PR.

The Trivy gate fails on one finding, and it is in a third-party binary this branch does not touch:

usr/bin/k6 (gobinary)
Total: 1 (HIGH: 1, CRITICAL: 0)
google.golang.org/grpc  CVE-2026-84445  HIGH  fixed  v1.81.1  -> 1.82.2, 1.83.2, ...

app/jsonrpc-bench-runner — the binary this PR changes — scans 0, as does the Alpine layer. The diff touches no Dockerfile, no go.mod and nothing that produces usr/bin/k6; it is 11 files under runner/ plus one doc page.

CVE-2026-84445 is a newly published CVE against the same vendored pkg:golang/google.golang.org/grpc@v1.81.1 that .trivyignore.yaml already suppresses twice (CVE-2026-84304, CVE-2026-56854) for exactly this reason — Grafana ships k6 prebuilt and nothing in this repo rebuilds it. The last green run of this workflow was 2026-09-11; the CVE database is refreshed on every run, so main would fail the same way today.

The fix is another entry in .trivyignore.yaml with the same purl scope and expiry, which is a security-policy change for the repo rather than something this feature PR should carry. Happy to open it as a separate one-line PR if you would like.