ci: a 78s-per-leg Unity step regression stayed green for two days because nothing watches step durations

[wallstop]

What happened

#394 merged on 2026-08-13 and added a test whose scanner was quadratic in file length (#407). It added 78 seconds to the EditMode step on every editor leg -- about 5.2 minutes per Unity Tests run, all of it inside the organization build-lock window.

Every run stayed green. It was found two days later, and only because the scan eventually blew a per-test timeout on a developer's machine during a local MCP run. CI never complained, because nothing in CI looks at how long a step takes.

The signal was already there

Step durations over the last 24 green Unity Tests runs on master (52 successful editor legs), split at the #394 merge:

Step	Before (n=24)	After (n=28)	Delta
Run Unity Test Runner (EditMode)	35.8s	113.8s	+78.0s
Run Unity PlayMode tests	50.7s	48.2s	-2.5s
Run Unity standalone tests	63.3s	57.6s	-5.7s

That is a 3.2x step regression, isolated to one step, obvious in the first run after the merge, and already available from GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs with no new instrumentation. #409 takes it back to a 36.3s mean.

The tension, stated up front

The repository's tooling philosophy is explicitly against bespoke validators, wrappers, and preflight/doctor machinery, and this is exactly the kind of thing that grows into a monitoring subsystem. So this issue records the gap; it does not assume something should be built.

SuiteWallClockBudgetTest already exists and did not catch this, which is worth understanding before adding anything: it asserts a per-version ceiling on the default suite (300s on 2021.3, 180s on 2022.3 and 6000.x), and a 78s regression on a leg that was previously well under the ceiling does not cross it. A budget that only fires at a fixed ceiling cannot see a 3.2x regression with headroom left.

Options, cheapest first

1. Nothing. Accept that step regressions are found when they eventually break something. This was found in two days, and the cost was CI minutes, not correctness. This is a legitimate answer.
2. Tighten the existing budget instead of adding a new mechanism. SuiteWallClockBudgetTest already warns past a 60s soft budget. If the soft warning were visible in the job summary rather than only in the log, Isolate EditMode scene fixtures #394 would have flagged itself on the first run. No new script, no new workflow.
3. A step-duration comparison in an existing job. The aggregate gate already runs; comparing the current run's three Run Unity * step durations against a rolling median of recent green master runs is one API call. This is the option most at odds with the tooling philosophy and should not be taken without the owner agreeing the recurring cost is worth it.

Revisit if

• A second step-duration regression of this size reaches master undetected.
• Unity Tests wall clock regresses without a corresponding correctness failure.
• Work on [Feature]: chore/research: CI Test Optimization #337 resumes, since this is now the cheapest remaining lever there -- the licensed section is test execution, not scheduling, and it moved by five minutes without anyone noticing.

Context and full measurements: #337 (comment)

[wallstop]

Nothing built here, and I do not think anything should be. But two things in the options above do not hold, and one of them changes the answer.

Option 2 could not have flagged #394. Not late - at all.

SuiteWallClockBudgetTest is a [SetUpFixture] declared in DxMessaging.Tests.Runtime, which compiles into WallstopStudios.DxMessaging.Tests.Runtime - the PlayMode assembly. NUnit applies a SetUpFixture per assembly, and the fixture's own remarks say so. The EditMode assemblies (...Tests.Editor, ...Tests.Editor.Allocations, ...Tests.00.Editor.Benchmarks) have no wall-clock fixture at all.

#394's regression was Run Unity Test Runner, the EditMode step: 35.8s -> 113.8s. The soft budget never ran there, so surfacing its warning would have surfaced nothing.

Option 2's "no new script, no new workflow" is also not accurate.

There is no GITHUB_STEP_SUMMARY write anywhere in scripts/unity/run-ci-tests.ps1 or .github/workflows/unity-tests.yml. "Make the soft warning visible in the job summary" means building the job-summary path first.

Current state, for the record.

Step means over the last 12 green Unity Tests runs on master (48 legs):

Step	Mean	Pre-#394	Post-#394
Run Unity Test Runner (EditMode)	35.3s	35.8s	113.8s
Run Unity PlayMode tests	46.7s	50.7s	48.2s
Run Unity standalone tests	58.1s	63.3s	57.6s

#409's fix holds two weeks on.

Why I still did not build.

Adding the missing EditMode fixture is cheap and uses only what exists. It would not have caught this one: EditMode runs at ~35s, and a ceiling loose enough not to flake on the 2021.3 runner (the PlayMode fixture uses 300s there, 180s elsewhere) does not trip at 113.8s. That is this issue's own finding - a fixed ceiling cannot see a 3.2x regression with headroom left - and it applies to the fixture that is missing exactly as much as to the one that exists.

So the EditMode gap is real but it is not this gap. It is worth knowing that the EditMode legs currently have no wall-clock ceiling of any kind, if a blow-out there ever matters on its own terms.

Measured from #438.

[wallstop]

Option 2 is implemented

The issue's second option: "If the soft warning were visible in the job summary rather than only in
the log, #394 would have flagged itself on the first run. No new script, no new workflow."

SuiteWallClockBudgetTest already logs the suite's elapsed time and both budgets on every run.
scripts/unity/run-ci-tests.ps1 now reads that one line out of the Unity log and:

• writes a row into the job summary for every leg, so the three numbers are visible on a green run;
• emits a workflow warning when the elapsed time is over the 60s soft budget.

Leg	Elapsed	Soft budget	Hard budget
editmode	36.30s	60.0s	180.0s

No new script and no new workflow. One regex per leg.

What it does not do

It does not compare against a rolling median of recent green runs. That is option 3, which the
issue calls the one most at odds with the repository's tooling philosophy, and it is not taken.

So this does not catch a regression that stays under the soft budget. It would have caught #394:
113.8s against a 60s soft budget, on the first run after the merge.

One thing that had to be pinned

The number is produced by C# and read by PowerShell, and neither compiles the other. The suite now
formats with the invariant culture so the decimal separator does not follow the runner locale, and
a test asserts the shared line shape from both sides.

Coverage

scripts/__tests__/run-ci-tests-wall-clock-summary.test.js: under budget, over budget, the
once-per-job table header, a log with no wall-clock line, a missing log file, and the shared line
shape. All passing.

[wallstop]

Option 2 is shipped and working. Step durations over the last 20 successful Unity Tests runs on master, 80 legs:

Step	Median	Mean
Run Unity Test Runner (EditMode)	37.0s	47.6s
Run Unity PlayMode tests	51.0s	47.5s
Run Unity standalone tests	56.0s	57.4s

EditMode is back in its pre-#394 band of 35.8s, against the 113.8s the regression caused.

run-ci-tests.ps1 lifts the suite's own wall-clock line into the job summary and warns past the 60s soft budget, so #394 would have flagged itself on its first run at 113.8s.

Option 1 is "do nothing". Option 3 is the one this issue calls most at odds with the tooling philosophy, and it needs the owner to accept a recurring cost.

Recommended for closing. Reopen if a second step-duration regression of this size reaches master undetected.

[wallstop]

Closing as recommended above: option 2 is shipped in run-ci-tests.ps1 (suite wall-clock line in the job summary plus the 60s soft-budget warning), EditMode step medians are back at 37.0s over 80 recent legs, and options 1 and 3 remain recorded in the issue. Reopen if a step-duration regression of this size reaches master undetected again.