Skip to content

Give each TestScheduler test its own scheduler and project directory - #1009

Open
calvinp0 wants to merge 1 commit into
mainfrom
fix_scheduler_test_shared_fixtures
Open

Give each TestScheduler test its own scheduler and project directory#1009
calvinp0 wants to merge 1 commit into
mainfrom
fix_scheduler_test_shared_fixtures

Conversation

@calvinp0

Copy link
Copy Markdown
Member

TestScheduler does not pass reliably on main. Measured over 12 runs of pytest arc/scheduler_test.py::TestScheduler -n 4 --dist worksteal: 0 clean, 12 failing. The whole file failed 5 of 5.

The failures rotate, which is why this reads as noise rather than a bug: test_check_rxn_e0_by_spc (7 of 12), test_initialize_output_dict (2), test_conformers (1), plus ERROR cascades of up to 11 tests at once.

The cleanest demonstration

$ pytest arc/scheduler_test.py::TestScheduler::test_initialize_output_dict     # on main
1 failed        AssertionError: False is not true

That test only passes when something else runs first. Its opening line asserted that the shared output dict already contained information — true only because an earlier test had dirtied it, and testing nothing about the method under test. It now writes that state itself.

Two causes, not one

1. A shared project directory deleted by tearDownClass. Every fixture used ARC_PATH/Projects/arc_project_for_testing_delete_after_usage3, and tearDownClass rmtree'd it. --dist worksteal can split a single TestCase class across workers, so each worker independently runs setUpClass and tearDownClass — worker A's teardown deletes the tree worker B is still writing into. This is the whole ERROR cascade, and it explains test_check_rxn_e0_by_spc: PIL writing ts_guesses.png into a directory that had just vanished.

2. Tests mutating shared class state. Twelve of them, including test_non_rotor permanently flipping sched1.job_types['rotors'], test_add_label_to_unique_species_labels appending to a list after asserting its exact initial value, and test_troubleshoot_ess_under_max_attempts reading what test_troubleshoot_ess_max_attempts wrote.

The change

setUpClasssetUp, plus a make_project_directory() helper giving each test a tempfile.mkdtemp under ARC_PATH/Projects released by addCleanup. tearDownClass deleted.

Construction cost was measured before choosing: setUpClass takes 62 ms, so per-test construction is cheap and the simple answer was the right one — no need to split read-only from mutating tests.

Verification

TestScheduler, -n 4 --dist worksteal 22/22 pass (was 0/12)
solo -n 4 6/6
whole file, -n 4 --dist worksteal 5/5
serial test count 45 before, 45 after
serial runtime 4.98 s → 6.70 s (+35 %)
-n 4 runtime ~4.0 s → ~3.9 s (unchanged)

Mutation-verified: restoring the shared fixed-name project directory and its tearDownClass brings the failures back in 3 of 10 runs. Re-sharing the scheduler object alone did not reproduce within 10 runs, because xdist dispatches in collection order so the polluting test usually still precedes its victim on the same worker — the order dependency is instead proven directly by the solo-run result above.

Also fixed: test_save_e_elect cleaned up with a trailing rmtree that leaked its directory on any failure; now addCleanup.

One commit, one file.

Relationship to other PRs

Same defect class as #1008, in a different file — #1008 converts 19 test modules writing scratch into ARC_TESTING_PATH. This one is ARC_PATH/Projects and is independent of it. Fixing this also unblocks CI on #1001, which has been failing on test_initialize_output_dict by luck of the draw rather than on anything in its own diff.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings August 22, 2026 08:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.58%. Comparing base (45d73a0) to head (320e542).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1009      +/-   ##
==========================================
- Coverage   64.60%   64.58%   -0.02%     
==========================================
  Files         119      119              
  Lines       39785    39785              
  Branches    10307    10307              
==========================================
- Hits        25703    25696       -7     
- Misses      11105    11110       +5     
- Partials     2977     2979       +2     
Flag Coverage Δ
functionaltests 64.58% <ø> (-0.02%) ⬇️
unittests 64.58% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ther

TestScheduler built its schedulers, species and jobs once in setUpClass and shared
them across every test in the class, while eleven of those tests mutate what they are
handed: they replace scheduler.output, inject species into species_dict and job_dict,
flip job_types['rotors'], overwrite job_status and output paths, and append to
unique_species_labels. All of them also shared a single fixed project directory that
tearDownClass deleted.

Under pytest-xdist the class is split across workers, each running its own class-level
setup and teardown, so one worker's tearDownClass removed the project directory another
worker was still writing into, and a test that had implicitly relied on an earlier test
dirtying the shared scheduler saw a clean one instead. Roughly half of the runs failed,
with a different set of tests each time.

Build the fixtures per test in setUp instead, and give every test its own project
directory removed via addCleanup, which also lets tearDownClass go. Constructing the
fixtures costs about 60 ms, so the serial run of the file grows from ~5.0 s to ~6.7 s
and the -n 4 run is unchanged.

test_initialize_output_dict opened by asserting that the shared output dict already
contained information, which only held because an earlier test had put it there; it
failed when run on its own. It now writes that information itself before asserting.
@calvinp0
calvinp0 force-pushed the fix_scheduler_test_shared_fixtures branch from 9f8c763 to 320e542 Compare August 23, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants