test: five failure-class harnesses from the corruption-hunt postmortems - #42
Merged
Conversation
Each harness targets a failure CLASS the three audit rounds kept finding, so the next bug of that shape is caught structurally: 1. Repo model checker (TestModelCheck_*): seeded random op sequences (plant/delete/undelete/gc/rotate/hold/replicate) through the real CLI paths, global invariants after EVERY step: every live manifest restorable, every live incremental's chain live, audit chain verifying, replica never lying about restorability. Fixed seeds in CI; 2000-op randomized run added to the nightly chaos-soak workflow (seed logged for exact replay). 2. Idempotency sweep (TestIdempotencySweep): every maintenance command twice; second run must succeed AND change no durable repo bytes (audit chain excluded). Immediately caught a wart, fixed here: shared-DEK rotation migration rewrote key slots with fresh nonces on every resume run -- sharedkey.Rewrap now no-ops when the destination already holds the same DEK under the new KEK. 3. Fault sweep (TestFaultSweep): one injected storage failure at every call position (test-only "faultfile" scheme wrapping fs); faulted runs must fail loudly, and a clean re-run must converge to the fault-free reference state. 4. Golden repo (TestGoldenRepo_StillReadable): frozen miniature repo committed under internal/backup/testdata/ (signed manifests, encrypted chunks, shared-DEK object, audit chain); every build must open, verify, decrypt, and restore it byte-exact. Regenerate only via PGHS_REGEN_GOLDEN=1 for deliberate format changes. 5. Encryption path parity (TestEncryptionResolutionParity_CLIvsAgent): CLI and agent must resolve backup encryption identically for every keyring state (empty / present / corrupt). Fixture hygiene: harnesses that generate keystore material now scope a per-test config dir so a generated kek.bin can't flip other tests' no-KEK branches. Full repo suite green: 162/162 packages, -race -p 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the 23-bug corruption-hunt postmortem into structural test coverage: each harness targets a failure class, so the next bug of that shape is caught by CI instead of the next audit.
TestModelCheck_*)TestIdempotencySweep)TestFaultSweep)faultfile://scheme injects one failure at every storage-call position; faulted runs must fail loudly and a clean re-run must converge to the fault-free reference stateTestGoldenRepo_StillReadable)PGHS_REGEN_GOLDEN=1for deliberate format changes onlyTestEncryptionResolutionParity_CLIvsAgent)Also in this PR:
sharedkey.Rewrapgains anunwrapNewparameter and no-ops when the destination slot already holds the same DEK (the idempotency wart harness Bump go.mongodb.org/mongo-driver from 1.8.3 to 1.17.7 #2 found).operations/integrity-testing.md; CHANGELOG entry.kek.binwas flipping other tests' no-KEK branches in the full run).Verification: full repo suite 162/162 packages green under
-race -p 1; model checker additionally validated with a 300-op randomized run.