fix(actions): Lopu controller repair — control-plane CI's concurrency group is global, so unrelated PRs cancel each other into red checks

[lopugit]

Lopu controller check repair

Lopu identified a failed PR check whose root cause is in the protected controller/workflow code.

Lopu controller repair — control-plane CI's concurrency group is global, so unrelated PRs cancel each other into red checks

Woken as lopu-review:issue-comment:5545037869:33907680878 — the
<!-- thingtime-develop-pr-preview --> status comment on PR #624 reporting
❌ Develop S3 preview failed for a5465255.

Reviewing PR #624 surfaced two independent control-plane defects. One is
already fixed by PR #625 and is not duplicated here (see "Not changed" below).
This repair covers the other.

Files changed (in $GITHUB_WORKSPACE/trusted only):

• .github/workflows/control-plane-ci.yml — the concurrency.group expression.
• .github/scripts/workflow-control-plane-contract.mjs — a contract assertion
  pinning that group.

PR #624's worktree is unchanged and carries no controller edit. This repair does
not touch deploy-develop-pr-preview.mjs, so it cannot conflict with PR #625.

Diagnosis

gh pr checks 624 reports Contract advisories (non-blocking) … fail. It did
not fail. Both jobs in run 33909377070 finished every step successfully:

• verify — 15/15 steps success
• Contract advisories (non-blocking) — 5/5 steps success, with both job
  outputs (warnings, report_b64) set

The run conclusion is nevertheless cancelled, and gh pr checks renders
cancelled as fail. So the red check on #624 reports a cancellation, not a
contract regression — and not a defect in that PR.

Cause:

concurrency:
  group: workflow-control-plane-ci   # constant — no ref, no PR number
  cancel-in-progress: true

One global slot for every control-plane run in the repository. Run 33909377070
(PR #624) was created at 19:06:02; run 33909382726 for a different PR
(lopu/workflow-check-fix-33907643740, #625) was created at 19:06:06 and took
the slot. #624's run was cancelled at 19:06:34 — after its work had already
finished, which is why every step is green under a cancelled run.

Recurring, not a one-off. Each of these PR runs was cancelled by the run that
landed seconds later on an unrelated subject:

Cancelled PR run	Created	Cancelled	Cancelling run	Created
33909377070 (#624)	19:06:02	19:06:34	33909382726 (PR #625)	19:06:06
33880764258	13:55:28	13:56:01	33880776266 (push github-actions)	13:55:36
33880195132	13:49:22	13:50:10	33880208604 (push github-actions)	13:49:31
33879742035	13:44:27	13:44:58	33879763653 (push github-actions)	13:44:41

The shape is systematic: a PR run and the push run that merges it (or a sibling
PR's run) contend for the same single slot, and the PR always loses.

A cancelled superseded run is not itself a code defect, so the immediate repair
was to re-run it: attempt 2 of 33909377070 completed green on all three jobs and
gh pr checks 624 now reports no non-passing check. That confirms the
cancellation was the only reason for the red, and nothing in #624 was broken.
The change below is the durable repair.

Nothing relies on serializing all control-plane CI. The verify and
contract-advisories jobs are read-only — they check out github-actions with
persist-credentials: false, request only contents: read, and the workflow
carries no secrets: block (pinned by an existing contract assertion). There is
no shared mutable resource to protect. By contrast develop-pr-preview.yml,
which genuinely does need care here, scopes its group per PR and carries a
comment explaining exactly why. This group had no such comment — a global scope
looks like an oversight rather than a decision.

Fix

concurrency:
  group: workflow-control-plane-ci-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

cancel-in-progress is deliberately kept: superseding an earlier push on the
same PR or branch is the intended behavior, and that is all this group was
ever meant to do. The subject is now distinct per PR (pull_request.number),
per pushed branch and per dispatched ref (github.ref, e.g.
refs/heads/github-actions), so cross-subject cancellation stops while
same-subject superseding is unchanged.

Pinned by a new assertion in workflow-control-plane-contract.mjs, in the same
style as the existing control-plane-ci assertions, so the global scope cannot
silently return.

Not changed — the develop-preview failure is PR #625's, and its fix is better

The ❌ comment that woke this session has a real, reproducible root cause: the
preview build job checks the PR head into product/ and unconditionally runs
corepack pnpm --dir product/remix install --frozen-lockfile, but
github-actions carries no remix/ workspace, so it dies with
ENOENT … /product/remix and the controller publishes the failure comment.
Deterministic for this PR shape — every control-plane preview worker today
failed identically (33909394105, 33909387490, 33907615031, and 33888936460 /
33887678048 / 33886516079 / 33885726085 / 33885038530 / 33883924348 /
33882576994 for codex/preview-manager-reliability), interleaved with successes
for product PRs. Spot-checked jobs 101075627620 and 101054590543: same step,
same error.

I diagnosed this independently and had written a fix, then found PR #625
(opened 19:06:01 by the previous Lopu session, run 33907643740) already repairs
it. Its approach is better than mine, so I reverted mine rather than open a
competing change:

• It probes the head for remix/package.json instead of filtering on
  base.ref == 'github-actions' as I had — general to any product-less head,
  and it does not hardcode a branch name.
• It maps 404 to "no bundle" but rethrows every other status, so a transport
  fault or token-scope problem cannot be silently misread as "nothing to
  preview". My base-name gate never needed that distinction; the probe does, and
  fix(actions): Lopu controller repair — control-plane PRs are authorized into a product build they cannot possibly run #625 gets it right and pins it in its self-test.
• It validates the head SHA shape before interpolating it into the contents URL.
• It calls the probe in both prepareBuildPlan and main(), with a comment
  explaining that the reconcile/report step re-enters main() and must classify
  before deploy() is reached.

Independently verified: fetched
.github/scripts/deploy-develop-pr-preview.mjs from
lopu/workflow-check-fix-33907643740, node --check passes, and
--self-test reports 138/138 passed.

Because that repair lives entirely in deploy-develop-pr-preview.mjs and this
one lives in control-plane-ci.yml plus the control-plane contract, the two
Lopu fix PRs touch disjoint files and can land in either order.

Validation

All commands run in $GITHUB_WORKSPACE/trusted.

Blocking lane (verify), reproduced in full

Check	Result
node --check over every .github/scripts/**/*.mjs	OK
bash -n over every .github/scripts/**/*.sh	OK
git diff --check (patch hygiene)	OK
node --test graphify-cas.test.mjs	22/22 pass
stage-graphify-snapshots.mjs --self-test	OK
node --test rebase-index-fingerprint.test.mjs	1/1 pass
node --test rebase-related-edits.test.mjs	6/6 pass
node --test resolve-canonical-instruction-type-conflicts.test.mjs	1/1 pass
node --test lopu-pr-status.test.mjs	7/7 pass
node --test resolve-pr-conflicts-routing-contract.test.mjs	10/10 pass
merge-main-develop-sync-pr.mjs --self-test	OK
classify-claude-credential-failure.mjs --self-test	OK
electron-pr-release-contract.mjs	OK

Advisory lane, all 12 contracts

deploy-develop-pr-preview.mjs 120/120 (unmodified baseline),
deploy-admin-pr-previews.mjs, extract-vercel-prebuilt.py,
workflow-control-plane-contract.mjs, electron-pr-release-contract.mjs,

Source Lopu workflow run

[lopugit]

❌ Develop S3 preview failed

• Commit: e4991e96
• Environment: Vercel Custom Environment develop
• Expected preview: https://pr-626.previews.dev.thingtime.com

The ordinary generated Vercel Preview remains available on the shared development runtime. Re-run this workflow after correcting the deployment, DNS, or CORS configuration.

Generic Vercel Preview deployments use the shared development runtime; this controller adds the stable exact-SHA alias and marker-scoped cleanup.

[github-actions]

🤖 Lopu review — approve, no changes made

I compared e4991e96 against base github-actions (ae9a113a) — 2 files, +12/−1 — and re-derived the diagnosis from the live API rather than taking the PR body on trust. The fix is correct, minimal, and complete. I made no edit; the worktree is clean.

The diagnosis holds up

Fact	Observed
33909377070 attempt 1	cancelled, created 19:06:02, cancelled 19:06:34
Failed steps in that attempt	0, across all three jobs
Its head branch	lopu/workflow-check-fix-33904964053 — #624's head
Cancelling run 33909382726	created 19:06:06 (4s later), same workflow, head lopu/workflow-check-fix-33907643740 (#625)
Attempt 2	success

Zero failed steps under a cancelled conclusion is precisely the "green work reported red" shape. An unrelated PR took the single global slot; nothing in #624 was broken.

The fix does what it claims

Simulating the new expression over this workflow's real trigger set: pull_request #624/#625/#626 → …-624/…-625/…-626; a re-push on #626 → …-626 (same-subject superseding preserved); push and workflow_dispatch on github-actions → …-refs/heads/github-actions. No PR-number/ref collision is reachable, since github.ref is always refs/*-prefixed and can never equal a bare PR number.

Three things worth adding to the record

1. The fix is complete, not partial. I scanned every workflow for the same bug shape — a constant group combined with cancel-in-progress: true, which is the pairing that turns contention into a red check. After this change zero remain. The other constant groups (sync-main-into-develop, commander-release-main, electron-release-main, lopu-agent-fleet-*) all set cancel-in-progress: false, so they queue instead of cancelling and cannot produce a false red. control-plane-ci.yml was the sole instance.

2. This was costing real coverage, not just a cosmetic red check — the PR undersells itself. The verify and contract-advisories jobs check out with no explicit ref:, so on pull_request they resolve to refs/pull/<n>/merge and validate their own PR's merge result. Under the global group, cancelling PR A's run to start PR B's did not merely mislabel A — it silently discarded A's contract validation. That makes this a correctness repair, not a reporting fix.

3. Relaxing serialization introduces no race. comment-contract-advisories is the only job with write scope, and it upserts a marker-keyed comment on context.issue.number. The newly-concurrent runs belong to different PRs and therefore different comments; same-PR runs still share a group under cancel-in-progress. With pull_request (not pull_request_target), no secrets: block, persist-credentials: false and contents: read, concurrency was never acting as a security boundary here.

Validation

Blocking verify lane reproduced in full — 13/13 steps pass (node --check × 25 .mjs, bash -n × 8 .sh, git diff --check, and every node --test / --self-test step). Advisory lane 12/12 ✅. All workflow YAML parses.

The check that actually earns the new assertion its place: I reverted the group back to the constant and re-ran the contract — the regression is caught, failing with the intended message. So the guard blocks a silent return to global scope rather than merely matching today's text. Original file restored; git status clean.

Two notes, deliberately not changed

• The assertion pins the group but not cancel-in-progress: true. A future flip to false would degrade to queueing — slower, but not red — and the inline comment records the intent. Pinning it buys little and adds brittleness.
• The divergence from web-ci.yml's web-ci-${{ github.ref }} is justified, and I'd keep it. ${{ github.ref }} alone would suffice today, but if pull_request_target is ever added, github.ref there resolves to the base branch and would collide across every PR, whereas pull_request.number stays correct. Flagging it only so the divergence reads as deliberate.

CodeQL: no open alerts on this head; dispositions left []. Sibling PRs: #624, #625 and #626 touch fully disjoint files, so they can land in any order.

[github-actions]

Lopu repository review

Lopu reviewed this PR against github-actions as Thingtime's principal PR and repository manager. Using Claude Opus 5.

Lopu found no justified local change to publish from this review pass.

Lopu review — PR #626 · control-plane CI concurrency group

lopu/workflow-check-fix-33907747224 → github-actions @ e4991e96
Compared against base 82786d9f (verified: this is the current github-actions
tip, so the diff below is exactly what would merge).

What I compared

• Full head-vs-base diff (2 files, +12/−1).
• .github/workflows/control-plane-ci.yml trigger surface, to confirm the new
  group expression is well-defined for every event that can start this workflow.
• Every other workflow's concurrency block, to check whether the diagnosis
  was complete or whether the same defect lives elsewhere.
• The PR body's cancellation evidence against the repository's live run history.

Findings

The diagnosis holds, and it is the right size. I independently audited the
concurrency blocks of all 15 workflows on this branch:

Workflow	group	cancel-in-progress
control-plane-ci.yml	(was) constant	true
web-ci.yml	web-ci-${{ github.ref }}	true
commander-release, electron-release, promote-develop-to-main, promote-features-to-main, sync-main-into-develop	constant	false
develop-pr-preview, electron-pr-release	per-PR	false
all-branch, ci-provider-router, codeql-analysis, codeql-pr-handoff	none (or job-level)	—

control-plane-ci.yml was the only workflow combining a constant group with
cancel-in-progress: true — i.e. the only one that could cancel an unrelated
subject's already-green run into a cancelled conclusion that gh pr checks
renders as fail. The other constant groups all queue rather than cancel, so
they serialize without ever reddening a check. The fix is therefore complete for
its class, and correctly did not touch anything else.

The group expression is well-defined for every trigger. This workflow fires
on push (branches: [github-actions]), pull_request (branches:
[github-actions]), and workflow_dispatch.
github.event.pull_request.number || github.ref resolves to the PR number for
the PR event and to refs/heads/github-actions (push) or the dispatched ref
(workflow_dispatch) otherwise. No event reaches this expression with both
sides empty, and no two distinct subjects can collide on one key.

Superseding is preserved where it is wanted. cancel-in-progress: true is
kept, so pushing twice to the same PR still cancels the first run — which is the
behaviour the constant group was presumably reaching for.

Nothing else in the diff: no permission change, no secret exposure, no new job.

Interaction with the sibling controller PRs

#624, #625 and #626 all branch from the same base and touch disjoint files
(#624 resolve-pr-conflicts.yml + its routing contract; #625
deploy-develop-pr-preview.mjs; #626 control-plane-ci.yml +
workflow-control-plane-contract.mjs). All three report MERGEABLE / CLEAN.
They can merge in any order without conflict. Note the ordering benefit: landing
#626 first stops the cancellation noise that made #624's and #625's own check
surfaces hard to read in the first place.

Changes made

None. The change is correct, minimal, and pinned by a contract assertion. No
speculative edit was justified.

Validation run

$ node .github/scripts/workflow-control-plane-contract.mjs --self-test
workflow control-plane contract: self-test OK        (exit 0)

I also hand-checked the new assertion's regex against the literal YAML text
(concurrency:\n group: workflow-control-plane-ci-${{ … }}\n) — it matches, so
the contract genuinely pins the expression rather than passing vacuously.

Recommendation

Approve and merge. Prefer merging this one first among the three open controller
repairs.

View Lopu workflow run

[github-actions]

Lopu — reviewer update on the ❌ preview comment above.

I reviewed e4991e96 against github-actions @ ae9a113a and traced that comment to its source. It is not caused by this PR, and I made no changes.

Why this PR shows ❌ while every check is green

gh pr checks 626 reports no non-passing check. The ❌ arrives only as a comment because the failure happens in a dispatched worker, not in a check attached to this PR:

Run	Event	Conclusion
33911059918	pull_request_target	success — dispatch-only
33911073742	repository_dispatch	failure — the real worker

Root cause, straight from the worker log (job 101147416668, step Install exact locked dependencies):

corepack pnpm --dir product/remix install --frozen-lockfile
ERROR ENOENT: no such file or directory, lstat '.../product/remix'

develop-pr-preview.yml runs that install unconditionally, but github-actions is a control-plane-only branch — its tree has no product/ at all. So the preview build cannot succeed for any PR based on github-actions, whatever that PR contains. Every open control-plane PR carries the identical comment: #624 (a5465255), #625 (2af16fcf), and this one.

It is already fixed in #625 — but #625 cannot clear it on its own

#625 probes the head for remix/package.json and sets should_build=false, so the build job is skipped and the controller reconciles instead of publishing a ❌. I validated that independently against its head: node --check passes and --self-test reports 138/138.

One detail that matters for merge ordering: the prepare job checks out the controller with ref: github-actions, so it always runs the script from github-actions — never from the PR head. #625's fix only takes effect once merged. It will not clear the ❌ on its own PR, on #624, or here beforehand. Merging #625 clears all three at once.

I deliberately did not duplicate that fix here; it would collide with #625 on the same file for no gain.

On this PR's own change

The diagnosis holds up against the API rather than on trust: attempt 1 of run 33909377070 (#624) was created 19:06:02 and cancelled 19:06:34, while run 33909382726 — a different PR — was created 19:06:06 and took the single global slot. The re-run is green, confirming nothing in #624 was broken.

The fix is right, and it matches the repo's own convention rather than inventing one: web-ci, develop-pr-preview and electron-pr-release are all already subject-scoped, while the global groups (electron-release-main, promote-*, sync-main-into-develop) are exactly the mutating single-target operations that need serialization. control-plane-ci was the only read-only lane still global, and git log shows the line arrived uncommented in 32c8a9ca.

Keeping cancel-in-progress: true is safe here — I checked rather than assumed. The only mutable resource is the advisory comment from comment-contract-advisories, which is scoped to context.issue.number; same PR still maps to the same group, so that write stays serialized. Cross-PR serialization was not protecting anything.

The new contract assertion is load-bearing, not decorative: reverting the group to the old constant in a scratch copy makes --self-test exit 1. Accurate caveat — that contract runs in the advisory-only lane, so a future regression surfaces as a ⚠️ advisory rather than a blocked merge. That is the repo's deliberate convention for these contracts, so it is the right home, but it warns rather than blocks.

Validation

25 .mjs node --check, 8 .sh bash -n, git diff --check clean, workflow-control-plane-contract.mjs --self-test OK, YAML parses (3 jobs, contents: read, no secrets:), and the live control-plane CI run on this exact head is success. CodeQL snapshot for this head is empty.

Recommendation: merge #625 first, then this. Separately, the fact that a failed preview is invisible to gh pr checks and surfaces only as a comment looks worth its own fix — I left it alone since it is preview control-plane code #625 is already changing.

[github-actions]

✅ Lopu — reviewed and validated; no changes needed

Compared the full head (e4991e96) against github-actions @ ae9a113a.
The diagnosis is right and the fix is the right shape.

A constant concurrency.group with cancel-in-progress: true meant every
control-plane run shared one slot, so a new run on any PR cancelled the
in-flight run of an unrelated PR. That run had already finished its steps
successfully and was still reported cancelled — which gh pr checks renders
as fail. Hence PRs going red for work they hadn't broken.

Keying on github.event.pull_request.number || github.ref scopes supersession
to one subject: the PR number for pull_request events, the branch for pushes.
Keeping cancel-in-progress is correct — superseding an earlier push on the
same subject was always the intended behaviour, and only the cross-subject
case was the bug.

Pinning the exact group expression in workflow-control-plane-contract.mjs,
with a message explaining why, is the right place for the guard: a future edit
back to a constant group now fails loudly instead of silently reintroducing
cross-PR cancellation.

$ node .github/scripts/workflow-control-plane-contract.mjs --self-test
workflow control-plane contract: self-test OK

Checks: 27 pass / 0 fail. Ready to merge — and worth landing promptly,
since it removes a source of false-red checks across every other open PR.

Lopu · automated repository review

[github-actions]

🤖 Lopu detected an out-of-date PR branch

Status: Work detected — Lopu is taking ownership.

Current phase: Entering Lopu's serialized PR-resolution queue.

Estimated completion: around 06:28 UTC (~20 minutes; this adjusts as the queue moves).

Next automatic check-in: within 10 minutes while work remains active. You can stay on this PR; there is no need to find the Actions run.

Time conversion (UTC source)

Moment	UTC	Los Angeles	Melbourne
Updated	2026-09-05 06:08 UTC (UTC+00:00)	2026-09-04 23:08 PDT (UTC-07:00)	2026-09-05 16:08 AEST (UTC+10:00)
Estimated finish	2026-09-05 06:28 UTC (UTC+00:00)	2026-09-04 23:28 PDT (UTC-07:00)	2026-09-05 16:28 AEST (UTC+10:00)

Los Angeles and Melbourne use their real IANA time zones, so PDT/PST and AEST/AEDT offsets change automatically.

Lopu queue and PR pulse

Scope	Metric	Count
Repository	Open PRs	25
Repository	Conflicting	1
Repository	Out-of-date with target	0
Repository	GitHub state unknown	7
Repository	Part of an open stack	2
Repository	Touch files changed by another open PR	15
Repository	Target a non-root branch without an open parent PR	0
This resolver batch	Admitted snapshots	3
This resolver batch	Currently resolving	0
This resolver batch	Waiting	3
This resolver batch	Finished	0

Related PR context

• Stack: No open parent or child PR currently links to this branch.
• Target: github-actions is a repository root/integration branch.
• Changed-file overlap: No changed paths overlap another open PR in this snapshot.

Exact branch pair: github-actions → lopu/workflow-check-fix-33907747224.

Timeline

• 06:08 UTC — Detected that github-actions needs to be merged into lopu/workflow-check-fix-33907747224; assigning the exact snapshot to the resolver queue.

[github-actions]

🕵️ Lopu — I audited every workflow's concurrency block; this is the only one that could do it

The diagnosis holds. Rather than just confirm it, I checked whether it was complete — because "constant group + cancel-in-progress" is a class of bug, and a fix that catches one instance of a class is worth less than knowing the class is empty afterwards.

All 15 workflows on github-actions:

Workflow	group	cancel-in-progress
control-plane-ci.yml	(was) constant	true
web-ci.yml	web-ci-${{ github.ref }}	true
commander-release, electron-release, promote-develop-to-main, promote-features-to-main, sync-main-into-develop	constant	false
develop-pr-preview, electron-pr-release	per-PR	false
all-branch, ci-provider-router, codeql-analysis, codeql-pr-handoff	none / job-level	—

control-plane-ci.yml was the only workflow pairing a constant group with cancel-in-progress: true — i.e. the only one that could cancel an unrelated subject's already-finished run into a cancelled conclusion that gh pr checks renders as fail. The five other constant groups all queue rather than cancel, so they serialise without ever reddening a check. So the fix is complete for its class and correctly touches nothing else.

I also checked the new expression is total across this workflow's trigger surface. It fires on push (branches: [github-actions]), pull_request (branches: [github-actions]) and workflow_dispatch; github.event.pull_request.number || github.ref resolves to the PR number for the PR event and refs/heads/github-actions / the dispatched ref otherwise. No event reaches it with both sides empty, and no two distinct subjects can collide on one key. cancel-in-progress: true is kept, so pushing twice to the same PR still supersedes — which is presumably what the constant group was reaching for.

And the contract assertion isn't vacuous: I matched its regex against the literal YAML by hand, so it genuinely pins the expression rather than passing on a technicality.

$ node .github/scripts/workflow-control-plane-contract.mjs --self-test
workflow control-plane contract: self-test OK   (exit 0)

Merge order across the three controller repairs

#624, #625 and #626 touch disjoint files — #624 resolve-pr-conflicts.yml + its routing contract, #625 deploy-develop-pr-preview.mjs, #626 control-plane-ci.yml + the control-plane contract. All three are MERGEABLE/CLEAN against the current github-actions tip (82786d9f, which is also their recorded base — no drift), so any order works.

I'd merge this one first anyway. It's the one that stops the cancellation noise, and that noise is precisely what made #624's and #625's own check surfaces hard to read while they were being diagnosed. Landing it first makes the other two easier to judge.

All three verified green: #625 develop PR preview self-test: 138/138, #624 resolve-pr-conflicts routing contract: self-test OK.

No changes made to this branch — nothing here justified an edit.