Skip to content

[BUG] check_dead_modules reports success while five of its roots import a module that was removed #150

Description

@RobertSigmundsson

Description

scripts/check_dead_modules.py (added in #137, run in CI as --strict) computes reachability
from the entry points where execution begins — console scripts, __main__ modules, string
literals, and anything an example, benchmark or script imports. Those last ones are roots.

Five of those roots import surreal_memory.storage.sqlite_store, a module #141 removed. The
guard walks the import graph purely from the AST, so it never asks whether an import target
exists. It reports No unreachable modules. and exits 0 while five of the files that define its
reachability set cannot run.

Measured, so the claim is not louder than the evidence: one of the five fails at import
(benchmarks/stress_at_scale.py, whose import is at module level), and the other four fail the
moment the function holding the import is called. All five are dead either way; only one of them
is dead in a way that a plain import would reveal.

#137 is one of the nicer things in this repo and its PR body puts the principle better than we
could:

A guard that reports nothing proves nothing — a broken one behaves identically.

This is that same principle one rung up, and not a flaw in the check's design: the guard itself is
sound, but the roots it trusts are not, and nothing tells it so.

To Reproduce

On a clean v3.0.3 checkout with the package installed:

$ python scripts/check_dead_modules.py
No unreachable modules.
$ echo $?
0
$ python -c "import importlib.util as u; s=u.spec_from_file_location('b','benchmarks/stress_at_scale.py'); m=u.module_from_spec(s); s.loader.exec_module(m)"
ModuleNotFoundError: No module named 'surreal_memory.storage.sqlite_store'

The other four import inside a function, so they load and then raise when that function runs —
rehearsal_coverage_overhead.py on the branch it takes when SURREALDB_URL is unset,
e2e_gemini_recall.py in step 2, and both comparison benchmarks in their nm_setup.

$ grep -rn "SQLiteStorage" benchmarks/ scripts/ | wc -l
18
$ python -c "import surreal_memory.storage as s; print(hasattr(s, 'SQLiteStorage'))"
False

Expected Behavior

Two separable things:

  1. The guard should not pass while a root imports something that is not there. Resolving each root's
    surreal_memory.* imports against the modules actually present in the tree is enough — no
    code has to be executed, so the check keeps the "reads the source tree only, no install step"
    property #137 deliberately gave it.
  2. The five files should stop referencing the removed backend.

Actual Behavior

No unreachable modules., exit 0, in CI (.github/workflows/ci.yml:37) and locally
(make audit-dead).

Code Sample

The 18 references, all of them real imports and uses — not prose:

benchmarks/stress_at_scale.py:33                    from …sqlite_store import SQLiteStorage   ← module level
benchmarks/stress_at_scale.py:136,180,214,232       type annotations
benchmarks/stress_at_scale.py:294                   SQLiteStorage(db_path=str(db_path))
benchmarks/stress_at_scale.py:379,482               report text: "Real SQLiteStorage … WAL mode"
benchmarks/rehearsal_coverage_overhead.py:117,121   the else branch taken when SURREALDB_URL is unset
scripts/benchmark_mem0_vs_nm.py:121,286,294
scripts/benchmark_cognee_vs_nm.py:116,276,284
scripts/e2e_gemini_recall.py:63,66

benchmarks/stress_at_scale.py is the only one with a module-level import, so it is the only
one an import test catches; the other four fail at call time. For contrast, the 16 occurrences
of the same name
under src/ and tests/ are all historical prose in docstrings ("used to reach into a private
SQLiteStorage attribute") — those are accurate and should stay.

Error Message

ModuleNotFoundError: No module named 'surreal_memory.storage.sqlite_store'

Environment

  • OS: Linux (x86-64)
  • Python version: 3.12.13
  • Surreal-Memory version: 3.0.3 (ac20df41)
  • Installation method: source, pip install -e ".[dev,server,surrealdb]"

Additional Context

We did not open a pull request for the second half, because the fix is not mechanical and the
choice is yours. The symbol is used, not merely imported — as a constructor and in type
annotations — so deleting the import does not leave working code. The options we can see:

  • Delete. benchmarks/stress_at_scale.py is SQLite-specific end to end (its own report says
    "Real SQLiteStorage benchmarks", "aiosqlite, WAL mode"), so with the backend gone it measures
    something that no longer ships.
  • Port. #140 made InMemoryStorage implement the full interface, which is a drop-in for
    scripts/e2e_gemini_recall.py and the two comparison benchmarks — but for a storage
    benchmark it changes what is being measured, and the prose claims about WAL mode would have to
    go with it.
  • Require SurrealDB. benchmarks/rehearsal_coverage_overhead.py already calls the SurrealDB
    path "the production-accurate path" in its own message; dropping the fallback branch is the
    smallest change there.

We are happy to send whichever of these you prefer as a PR, including the guard change.

Related: #137 (the guard), #141 (backend removal), #140 (InMemoryStorage completeness).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions