Give each TestScheduler test its own scheduler and project directory - #1009
Open
calvinp0 wants to merge 1 commit into
Open
Give each TestScheduler test its own scheduler and project directory#1009calvinp0 wants to merge 1 commit into
calvinp0 wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1009 +/- ##
==========================================
- Coverage 64.60% 64.58% -0.02%
==========================================
Files 119 119
Lines 39785 39785
Branches 10307 10307
==========================================
- Hits 25703 25696 -7
- Misses 11105 11110 +5
- Partials 2977 2979 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Aug 22, 2026
calvinp0
force-pushed
the
fix_scheduler_test_shared_fixtures
branch
from
August 23, 2026 08:56
e08c54a to
9f8c763
Compare
…ther TestScheduler built its schedulers, species and jobs once in setUpClass and shared them across every test in the class, while eleven of those tests mutate what they are handed: they replace scheduler.output, inject species into species_dict and job_dict, flip job_types['rotors'], overwrite job_status and output paths, and append to unique_species_labels. All of them also shared a single fixed project directory that tearDownClass deleted. Under pytest-xdist the class is split across workers, each running its own class-level setup and teardown, so one worker's tearDownClass removed the project directory another worker was still writing into, and a test that had implicitly relied on an earlier test dirtying the shared scheduler saw a clean one instead. Roughly half of the runs failed, with a different set of tests each time. Build the fixtures per test in setUp instead, and give every test its own project directory removed via addCleanup, which also lets tearDownClass go. Constructing the fixtures costs about 60 ms, so the serial run of the file grows from ~5.0 s to ~6.7 s and the -n 4 run is unchanged. test_initialize_output_dict opened by asserting that the shared output dict already contained information, which only held because an earlier test had put it there; it failed when run on its own. It now writes that information itself before asserting.
calvinp0
force-pushed
the
fix_scheduler_test_shared_fixtures
branch
from
August 23, 2026 12:14
9f8c763 to
320e542
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestSchedulerdoes not pass reliably onmain. Measured over 12 runs ofpytest arc/scheduler_test.py::TestScheduler -n 4 --dist worksteal: 0 clean, 12 failing. The whole file failed 5 of 5.The failures rotate, which is why this reads as noise rather than a bug:
test_check_rxn_e0_by_spc(7 of 12),test_initialize_output_dict(2),test_conformers(1), plusERRORcascades of up to 11 tests at once.The cleanest demonstration
That test only passes when something else runs first. Its opening line asserted that the shared
outputdict already contained information — true only because an earlier test had dirtied it, and testing nothing about the method under test. It now writes that state itself.Two causes, not one
1. A shared project directory deleted by
tearDownClass. Every fixture usedARC_PATH/Projects/arc_project_for_testing_delete_after_usage3, andtearDownClassrmtree'd it.--dist workstealcan split a singleTestCaseclass across workers, so each worker independently runssetUpClassandtearDownClass— worker A's teardown deletes the tree worker B is still writing into. This is the wholeERRORcascade, and it explainstest_check_rxn_e0_by_spc: PIL writingts_guesses.pnginto a directory that had just vanished.2. Tests mutating shared class state. Twelve of them, including
test_non_rotorpermanently flippingsched1.job_types['rotors'],test_add_label_to_unique_species_labelsappending to a list after asserting its exact initial value, andtest_troubleshoot_ess_under_max_attemptsreading whattest_troubleshoot_ess_max_attemptswrote.The change
setUpClass→setUp, plus amake_project_directory()helper giving each test atempfile.mkdtempunderARC_PATH/Projectsreleased byaddCleanup.tearDownClassdeleted.Construction cost was measured before choosing:
setUpClasstakes 62 ms, so per-test construction is cheap and the simple answer was the right one — no need to split read-only from mutating tests.Verification
TestScheduler,-n 4 --dist worksteal-n 4-n 4 --dist worksteal-n 4runtimeMutation-verified: restoring the shared fixed-name project directory and its
tearDownClassbrings the failures back in 3 of 10 runs. Re-sharing the scheduler object alone did not reproduce within 10 runs, because xdist dispatches in collection order so the polluting test usually still precedes its victim on the same worker — the order dependency is instead proven directly by the solo-run result above.Also fixed:
test_save_e_electcleaned up with a trailingrmtreethat leaked its directory on any failure; nowaddCleanup.One commit, one file.
Relationship to other PRs
Same defect class as #1008, in a different file — #1008 converts 19 test modules writing scratch into
ARC_TESTING_PATH. This one isARC_PATH/Projectsand is independent of it. Fixing this also unblocks CI on #1001, which has been failing ontest_initialize_output_dictby luck of the draw rather than on anything in its own diff.🤖 Generated with Claude Code