Skip to content

Make Lewis-structure perception independent of PYTHONHASHSEED - #989

Closed
calvinp0 wants to merge 2 commits into
mainfrom
fix_hash_seed_dependent_perception
Closed

Make Lewis-structure perception independent of PYTHONHASHSEED#989
calvinp0 wants to merge 2 commits into
mainfrom
fix_hash_seed_dependent_perception

Conversation

@calvinp0

Copy link
Copy Markdown
Member

ARC perceives a different Lewis structure depending on the process's PYTHONHASHSEED. Same molecule, same code, same commit — the answer depends on which process it runs in.

seed=1    bonds = B:36  S:116          embeds fine
seed=35   bonds = B:24  D:6  S:122     Explicit valence for atom # 5 C greater than permitted

That is COC1=C(CN[C@H]2C3CCN(CC3)[C@H]2C(C2=CC=CC=C2)C2=CC=CC=C2)C=C(C=C1)C(C)C built from the same xyz both times. Roughly 2% of seeds give the wrong structure.

The chain

arc/molecule/graph.pyx::get_all_edges de-duplicated through a set and returned list(edge_set):

edge_set = set()
for vertex in self.vertices:
    for edge in vertex.edges.values():
        edge_set.add(edge)
return list(edge_set)

So the returned order was the set's iteration order. Atom and Bond take content-derived hashes — hash(('Atom', self.symbol)), hash(('Bond', order, sym1, sym2)) — while __eq__ is identity. Every carbon therefore collides into one bucket, and the resulting order is governed by Python's per-process randomized string hash.

arc/species/perceive.py::generate_lewis_structure consumes that order directly:

bond_pairs = [(base_mol.atoms.index(e.vertex1), base_mol.atoms.index(e.vertex2))
              for e in base_mol.get_all_edges()]

and runs an A* that expands equal-cost states in bond_pairs order. The molecule above has two degenerate minimum-cost Lewis structures, so the tie is broken by the hash seed.

Seed 35 places the double bond on the methoxy C–O rather than inside the anisole ring, leaving ring carbon #5 with three single bonds and a lone pair — a carbene. to_rdkit_mol's carbene handling then sets two radical electrons on it, giving explicit valence 5, and RDKit refuses to sanitize.

Why this matters beyond the crash

Perception decides bond orders. Bond orders feed bond_corrections, which feed Arkane's BAC, which feed energies and rates. A calculation is therefore not guaranteed to be reproducible across processes — including between a run and its own restart.

No downstream numerical impact has been measured. That is stated as an open question, not a claim.

The fix

Return edges in the graph's vertex order, de-duplicating on edge identity rather than on hash-and-equality:

edges = []
seen = set()
for vertex in self.vertices:
    for edge in vertex.edges.values():
        if id(edge) not in seen:
            seen.add(id(edge))
            edges.append(edge)
return edges

Same edges, same complexity, order identical in every process.

This is deliberately at the producer. get_all_edges also feeds py_rdl SSSR ring perception (graph.pyx:962, :995), so fixing the order where it is manufactured fixes every consumer at once, rather than sorting at each call site.

How it was found

A CI run of an unrelated PR failed on one xdist worker with IndexError: list index out of range from conformers_test.py::test_embed_rdkit. It looked like flakiness, then like cross-test contamination — both wrong. Replaying that worker's exact 90-test prefix node-by-node passes. Fourteen full-suite runs and ~8,300 per-test canary checks came back clean because they varied test order; the variable was the seed.

Each xdist worker is a separate process with its own hash seed, which is why parallel CI surfaced it and serial runs never did.

Checks

  • 600 PYTHONHASHSEED values against the affected molecule: 588/600 pass before, 600/600 after, all reporting one identical bond census. Known-bad seeds were 35, 79, 87, 91, 166.
  • bond_pairs order for seeds 1 and 35: different before, identical after.
  • arc/molecule/graph_test.py: 41 → 43 passed. The new test asserts the vertex-order contract and spawns two subprocesses at different PYTHONHASHSEED values, asserting identical edge order — it fails against the unfixed code.
  • arc/species/conformers_test.py serial: 48 passed, unchanged.
  • The failing worker's exact prefix, replayed: 118 passed.
  • Full suite -n 6 --dist worksteal: 2745 → 2747 passed, same 5 pre-existing torch_ani failures either side.

Deliberately not addressed

  • The wrong structure passes is_mol_valid. A neutral carbene carbon with three single bonds and a lone pair should not survive get_octet_deviation. This change makes the affected molecule always take the good branch, but the validity gate remains permissive, and another molecule's degenerate optimum could land on a carbene deterministically. That wants its own change.
  • Atom.__hash__ / Bond.__hash__ are content-derived while __eq__ is identity. Legal, but it makes every atom or bond set degenerate to a single bucket per element — an O(n²) hazard as well as an ordering one. Changing it would trade a seed-dependent order for an address-dependent one, so it fixes nothing on its own. get_all_edges is where that order escaped into a chemical decision.

Graph.get_all_edges() de-duplicated the edges through a set of Edge objects
and returned list(edge_set), so the order came out of the set's iteration.
Atom and Bond hash on their symbols and bond order rather than on identity,
and those hashes are derived from string hashes, which Python randomises per
process. Every carbon in a molecule therefore lands in one hash bucket and
the resulting edge order differs from one process to the next.

Molecule perception consumes that order: generate_lewis_structure() walks the
bond list in an A* search whose equal-cost states are explored in the order
the bonds are listed, so a molecule with several equal-cost Lewis structures
could be perceived differently in different processes. Diphenylprolinol
methyl ether was perceived with a methoxy C=O double bond and a carbene ring
carbon in about 2% of hash seeds, which then failed RDKit's valence check.

Return the edges in the graph's vertex order instead, de-duplicating on the
edge identities, so the order is the same in every process.
@calvinp0
calvinp0 force-pushed the fix_hash_seed_dependent_perception branch from f5adf52 to 872d8e2 Compare August 16, 2026 22:19
The edge order returned by get_all_edges() used to follow the iteration order of
a set of Edge objects, which is governed by the per-process randomized string
hash, so this asserts the property directly: two subprocesses started at
different PYTHONHASHSEED values must report the same order.

The child processes are given PYTHONPATH explicitly. A subprocess inherits the
parent's working directory but not pytest's sys.path, so without it the child
imports whichever ARC `import arc` resolves to -- which, with an editable
install present, is not necessarily the tree under test. The test then either
fails spuriously when run from outside the repository root, or passes while
having validated a different checkout.
@codecov

codecov Bot commented Aug 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.11%. Comparing base (adf58ad) to head (872d8e2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #989      +/-   ##
==========================================
- Coverage   64.15%   64.11%   -0.05%     
==========================================
  Files         119      119              
  Lines       39564    39564              
  Branches    10265    10265              
==========================================
- Hits        25383    25366      -17     
- Misses      11206    11221      +15     
- Partials     2975     2977       +2     
Flag Coverage Δ
functionaltests 64.11% <ø> (-0.05%) ⬇️
unittests 64.11% <ø> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@calvinp0

Copy link
Copy Markdown
Member Author

Superseded by #992, which combines this PR with #991.

The two cannot land separately. get_all_edges (this PR) is upstream of the cycle methods #991 fixes: on a build of #991 alone, get_polycycles and get_disparate_cycles remained hash-seed dependent because the list of cycles still followed the hash-ordered edge list. Conversely this PR alone does not make symmetry numbers reproducible. #992 carries both changes plus the review findings from each.

@calvinp0 calvinp0 closed this Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant