Conversation
The live deploy control plane is currently downgrading a healthy worker hourly (personal-companion: 30 attempts, every one v1.1.0 -> 1.0.0; production verified at v1.1.0 via https://reading.q08.org/health). The root cause is a leading-v version misparse: "v1.1.0" -> [0,1,0], judged behind canonical "1.0.0". qnfo-fleet-control/version-compare.mjs is the corrected comparator. Its most load-bearing rule is that equal cores with differing suffixes are UNORDERABLE rather than ordered - a naive semver rule (release > prerelease) would place "3.6.1" above "3.6.1-subscribers" and downgrade the live gateway, and would place "1.14.1" above "1.14.1-gtd-guard" and strip the cloud-ops guard. Nothing ran that suite in CI, so the fix could be silently undone by a later "simplification" back to semver ordering. This pins it. Co-authored-by: Chatbox <chatbox@chatboxai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The live deploy control plane is issuing an hourly redeploy of a healthy worker against a lower version. From
fleet_deploysandfleet_drift_report(read this session):personal-companionfrom_sha=v1.1.0to_sha=1.0.0ok=0, HTTP 400 code 10021qnfo-cloud-ops1.14.1 -> 1.14.1-gtd-guardProduction was independently confirmed healthy at
1.1.0, pieces 8 viahttps://reading.q08.org/health.Root cause: a leading
vmakesparseInt("v1")returnNaN -> 0, so"v1.1.0"parsed as[0,1,0]and was judged behind canonical"1.0.0"=[1,0,0]. The same version pair is labelled bothcanonical-ahead(drift id 1672) anddeployed-ahead(drift id 1691) across runs — the running comparator is not deterministic.The only thing currently preventing a production downgrade is that the artifact being uploaded lacks the
GenerationFlowexport, so Cloudflare rejects it. Severalpersonal-companion/FINDING-*-DO-NOT-FIX-10021.mddocuments exist because the obvious "fix" would convert a benign 400 into a live downgrade.What this adds
A CI job that runs
qnfo-fleet-control/version-compare.test.mjson every push/PR, plus explicit assertions for the invariants that cause real damage if they regress:compareVersions("3.6.1-subscribers", "3.6.1")must beUNORDERABLE— a naive semver rule (release > prerelease) would order3.6.1above3.6.1-subscribersand downgrade the live gateway (fleet_status=3.6.1-subscribers, registry3.6.1).compareVersions("1.14.1", "1.14.1-gtd-guard")must beUNORDERABLE— naive semver would strip the cloud-ops guard.deployDecision("v1.1.0", "1.0.0")anddeployDecision("v2.0.0", "1.9.9")must both beno-act.Rule 5 of the module (equal cores, differing suffixes ⇒ unorderable) is load-bearing and non-obvious. Nothing ran the suite in CI, so a later "simplification" back to semver ordering would reintroduce the downgrade with no signal.
Verification performed
The suite was executed independently (functions transcribed verbatim and run in an isolated compute context, not imported):
All six probes I added behave fail-closed.
One discrepancy found, not fixed here
version-compare.mjsheader says "Verified: 20/20 assertions pass". The suite contains 17 assertions (10 decision cases + 4 unorderable pairs + 3 extractor cases). I have deliberately not rewritten the module in this PR: reproducing 5.8 KB of JavaScript with regex and template literals by hand risks corrupting a verified artifact to fix a comment. Recommend a one-word edit to17/17by whoever next touches the file.Not in this PR, and why
The comparator is not wired into the live deployer.
qnfo-fleet-control/worker.jsis 75,875 B against the ops endpoint's 32,768-char read cap with no offset, so the wiring patch (PATCH-2026-09-13-downgrade-guard-bundle.mjs) cannot be validated from that endpoint. This PR pins the module so the wiring cannot land against a silently reverted comparator.Separately,
fleet_deploy_state.auto_healwas set to0this session as an interlock against the downgrade. Restore it to1only after the comparator is wired and deployed.Co-authored-by: Chatbox chatbox@chatboxai.com