perf: compare each benchmark in adjacent app processes

[mrousavy]

For a given benchmark, base and head currently run minutes apart across four complete suite passes. Install the two Release apps side by side and measure each case as calibration → base → head before moving to the next case. This reduces a 46-case comparison from 230 fresh app processes to 138 while keeping identical operation counts, five warmups, and twenty samples per measurement.

Base retains its app ID; head uses .head through Gradle and Xcode build settings. Same-SHA checks reuse the exact same binary for both roles. Changed suites retain head-only baselines. The controller owns one case loop. Raw per-process JSON, artifact provenance, and measurement-only reruns remain available. Reporting accepts one pair and explicitly does not claim between-launch repeatability or statistical confidence.

CI results

The PR workflow passed. Compared with the previous cleanup run:

Measurement job	Previous	This PR	Typical base/head separation
Android	27m08s	16m17s	5.7 minutes → 7 seconds
iOS	37m16s	25m47s	7.0 minutes → 9 seconds

The entire workflow took 38m09s, versus 49m10s previously. These are observations from separate hosted runners, not a controlled estimate of repeatable speedup. All 46 calibration/base/head triples on each platform had matching benchmark IDs and work counts, and all 138 process timestamps followed the intended order.

Accuracy remains limited, especially on iOS. This PR does not change the measured function implementations, yet 2/46 Android cases and 23/46 iOS cases showed changes above 5%; the largest iOS difference was about 42%. Some buffer/callback workloads still drift within a process. Closer pairing is not enough to make a 3% regression gate trustworthy.

The separate exact-same-binary A/A diagnostic also passed. The saved base/head app artifacts were verified byte-for-byte identical on both platforms, and every calibration/base/head triple preserved matching work and case order.

A/A diagnostic	Median absolute difference across cases	Cases differing by more than 5%	Largest absolute difference
Android	0.79%	3/46	12.3%
iOS	10.22%	33/46	33.9%

These are descriptive results from one diagnostic run, not calibrated error rates. They demonstrate that this iOS environment remains unsuitable for treating small observed differences as regressions. The change saves time and brings comparisons closer together; it does not solve iOS measurement reliability. Results remain report-only. Download the A/A raw JSON.

Download the PR's raw JSON artifact. The new trusted validator rebuilt Markdown and Bencher JSON locally from that artifact against GitHub's actual run, PR, and artifact metadata. No local publication was performed.

Validation and rollout

74 tooling tests passed, including real controller/receiver fixtures for Android and iOS, app identity and process ordering, A/A reuse, changed suites, malformed results, and artifact packaging. Tooling/app typechecks, app lint, actionlint, shellcheck, and diff checks passed. The original PR run at 77c4d5c3 passed all checks, including both native Release builds and measurements. Both downloaded CI artifacts also passed the measured revision’s trusted report validator against GitHub metadata.

Originally stacked on #1609. The results above measure 77c4d5c3, before the later rebase to 816875fa and merge. The benchmark app and measurement controller are unchanged by that rebase. Its replacement CI run is still running; the superseded run was cancelled before executing jobs. At the time of the measured PR run, main's reporter skipped this internal PR, so the new report was validated locally without adding a temporary publishing path.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nitro-docs	Ready	Preview	Sep 7, 2026 10:54am UTC