fix: bound benchmark scenario wall-clock time to fit CI's job timeout - #1707
Merged
Merged
Conversation
The tsgo sync API spawns a native child process per ts-loader instance, making a "cold build" iteration ~10-20x more expensive than the classic API's cheap in-process instantiation; touching a widely-imported (hub) file is similarly ~35x more expensive per incremental rebuild due to per-dependant recheck round trips over the sync RPC channel. The fixed iteration counts (tuned for the classic API's cost profile) made the cold-typeCheck and hub-touch scenarios alone take ~28 minutes combined, blowing the 20-minute CI job timeout before a single scenario finished. Cap each scenario's wall-clock time in run-side.mts instead of guessing a smaller fixed iteration count, so cheap scenarios keep their full sample size while expensive ones stop once they've collected enough measured samples. Confirmed locally: full default run (300 files) went from never finishing to completing in ~4 minutes.
Benchmark (Ubuntu)
PR branch = |
Benchmark (Windows)
PR branch = |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the wall-clock safety cap from the
copilot/implement-new-tsgo-api-supportbranch's benchmark harness (commit b9efbc4) ontomain, so both branches' benchmark mechanisms match.run-side.mts(60s, after a minimum sample floor) instead of relying solely on a fixed iteration count.maintoday (classic ts-loader, cheap in-process instantiation) this cap essentially never triggers - verified with a local run against this branch's own checkout, where every scenario completed at its full iteration count with no early cutoff. It's here as a safety net: if a future change (here or in a dependency) makes some scenario meaningfully more expensive per iteration, the benchmark degrades gracefully (fewer samples, still reports) instead of timing out with no results at all - exactly what happened on the tsgo branch before this cap was added there.Test plan
yarn lintpassesyarn buildpasses (TypeScript 6.0.2)npx tsc --project test/benchmark-tests/tsconfig.json --noEmitpassesnode test/benchmark-tests/run-benchmark.mtslocally (self-comparison) - all 6 scenarios completed at full iteration count, no early cutoffs, deltas near zero as expected🤖 Generated with Claude Code