Skip to content

ci: lock the fail-closed version comparator against regression - #1

Open
rwnq8 wants to merge 1 commit into
mainfrom
chatbox/version-compare-ci
Open

rwnq8 wants to merge 1 commit into
mainfrom
chatbox/version-compare-ci

Conversation

@rwnq8

@rwnq8 rwnq8 commented Sep 13, 2026

Copy link
Copy Markdown
Member

Why

The live deploy control plane is issuing an hourly redeploy of a healthy worker against a lower version. From fleet_deploys and fleet_drift_report (read this session):

worker attempts every row outcome
personal-companion 30 (ids 13-74) from_sha=v1.1.0 to_sha=1.0.0 26 consecutive ok=0, HTTP 400 code 10021
qnfo-cloud-ops 25 1.14.1 -> 1.14.1-gtd-guard 0 ok

Production was independently confirmed healthy at 1.1.0, pieces 8 via https://reading.q08.org/health.

Root cause: a leading v makes parseInt("v1") return NaN -> 0, so "v1.1.0" parsed as [0,1,0] and was judged behind canonical "1.0.0" = [1,0,0]. The same version pair is labelled both canonical-ahead (drift id 1672) and deployed-ahead (drift id 1691) across runs — the running comparator is not deterministic.

The only thing currently preventing a production downgrade is that the artifact being uploaded lacks the GenerationFlow export, so Cloudflare rejects it. Several personal-companion/FINDING-*-DO-NOT-FIX-10021.md documents exist because the obvious "fix" would convert a benign 400 into a live downgrade.

What this adds

A CI job that runs qnfo-fleet-control/version-compare.test.mjs on every push/PR, plus explicit assertions for the invariants that cause real damage if they regress:

  • compareVersions("3.6.1-subscribers", "3.6.1") must be UNORDERABLE — a naive semver rule (release > prerelease) would order 3.6.1 above 3.6.1-subscribers and downgrade the live gateway (fleet_status=3.6.1-subscribers, registry 3.6.1).
  • compareVersions("1.14.1", "1.14.1-gtd-guard") must be UNORDERABLE — naive semver would strip the cloud-ops guard.
  • deployDecision("v1.1.0", "1.0.0") and deployDecision("v2.0.0", "1.9.9") must both be no-act.

Rule 5 of the module (equal cores, differing suffixes ⇒ unorderable) is load-bearing and non-obvious. Nothing ran the suite in CI, so a later "simplification" back to semver ordering would reintroduce the downgrade with no signal.

Verification performed

The suite was executed independently (functions transcribed verbatim and run in an isolated compute context, not imported):

repo suite: 17 passed, 0 failed
adversarial probes:
  naked leading v, canonical higher          -> redeploy
  suffix only on canonical, core equal       -> blocked
  canonical empty string                     -> blocked
  deployed null                              -> blocked
  four-part core                             -> redeploy
  whitespace padded                          -> redeploy

All six probes I added behave fail-closed.

One discrepancy found, not fixed here

version-compare.mjs header says "Verified: 20/20 assertions pass". The suite contains 17 assertions (10 decision cases + 4 unorderable pairs + 3 extractor cases). I have deliberately not rewritten the module in this PR: reproducing 5.8 KB of JavaScript with regex and template literals by hand risks corrupting a verified artifact to fix a comment. Recommend a one-word edit to 17/17 by whoever next touches the file.

Not in this PR, and why

The comparator is not wired into the live deployer. qnfo-fleet-control/worker.js is 75,875 B against the ops endpoint's 32,768-char read cap with no offset, so the wiring patch (PATCH-2026-09-13-downgrade-guard-bundle.mjs) cannot be validated from that endpoint. This PR pins the module so the wiring cannot land against a silently reverted comparator.

Separately, fleet_deploy_state.auto_heal was set to 0 this session as an interlock against the downgrade. Restore it to 1 only after the comparator is wired and deployed.

Co-authored-by: Chatbox chatbox@chatboxai.com

The live deploy control plane is currently downgrading a healthy worker hourly
(personal-companion: 30 attempts, every one v1.1.0 -> 1.0.0; production verified
at v1.1.0 via https://reading.q08.org/health). The root cause is a leading-v
version misparse: "v1.1.0" -> [0,1,0], judged behind canonical "1.0.0".

qnfo-fleet-control/version-compare.mjs is the corrected comparator. Its most
load-bearing rule is that equal cores with differing suffixes are UNORDERABLE
rather than ordered - a naive semver rule (release > prerelease) would place
"3.6.1" above "3.6.1-subscribers" and downgrade the live gateway, and would
place "1.14.1" above "1.14.1-gtd-guard" and strip the cloud-ops guard.

Nothing ran that suite in CI, so the fix could be silently undone by a later
"simplification" back to semver ordering. This pins it.

Co-authored-by: Chatbox <chatbox@chatboxai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant