Skip to content

fix(tests): add pytest.importorskip guards to fix 9 collection errors - #30

Open
pt-act wants to merge 1 commit into
JailbrokenAI:mainfrom
pt-act:upstream-contrib/ac4-test-collection-fix
Open

fix(tests): add pytest.importorskip guards to fix 9 collection errors#30
pt-act wants to merge 1 commit into
JailbrokenAI:mainfrom
pt-act:upstream-contrib/ac4-test-collection-fix

Conversation

@pt-act

@pt-act pt-act commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

wallbreaker_mcp/server.py:

  • wb_seed_list now includes builtin static categories as fallback when any source-specific query returns 0 external categories (was only on source='all'). Fixes AC4 for source='harmbench' when HarmBench data isn't cached.

pyproject.toml:

  • Add hypothesis>=6.0 to [dependency-groups] dev (was only in [project.optional-dependencies] dev, not installed by uv sync)

tests/ (9 files):

  • Add pytest.importorskip guards for optional deps (hypothesis, fastapi, PIL/Pillow) so tests skip gracefully instead of erroring during collection
  • Affected: test_security_properties, test_audit_remediation, test_tg1..tg7, test_typographic

Result: 0 collection errors (was 9), 60 skipped, 1173 passed, 16 pre-existing failures (test_session_card, test_tg5_harden)

Summary

Adds pytest.importorskip(...) guards to three test modules so the suite collects cleanly when optional deps are absent, instead of erroring during collection. Small, self-contained test-infrastructure fix — no production code changes.

Problem

When the test environment is missing an optional dependency (hypothesis, fastapi, or Pillow), the affected modules import it at module top-level. Pytest then fails during collection (before any test runs), reporting errors rather than skips. On a minimal install this produced 9 collection errors.

Fix

Three one-line additions / one cleanup (+3 / -1):

  • tests/pbt/test_security_properties.py — guard importorskip('hypothesis') before from hypothesis import ...
  • tests/test_audit_remediation.py — guard importorskip('fastapi') and importorskip('hypothesis')
  • tests/test_tg2_context.py — drop a redundant bare import pytest that preceded an existing importorskip line

Verification

Fresh venv, uv sync --frozen + uv run pytest on main:

Before After
9 collection errors 0 collection errors

Full suite after the fix (fresh venv, optional deps installed): 1 failed, 1485 passed, 55 skipped, 31 xfailed. The single remaining failure (test_ac08_wb_attack_dispatches_existing_strategy_engine in test_wallbreaker_mcp.py) is pre-existing on main — it asserts endpoint.model == "gpt-4o" against a dispatch path whose mock returns a different model, unrelated to this change. Confirmed it fails identically on pristine main without this patch.

Scope note

The original internal commit also carried a wallbreaker_mcp/server.py static-fallback tweak and a pyproject.toml dev-dep entry; those pieces are already present on main, so the only net change required for the collection fix is the test-file guards shipped here. Kept to a single focused commit to make review trivial.

Checklist

  • Single focused commit on top of main
  • No production code touched (test files only)
  • Verified on a clean fresh venv (rm -rf .venv && uv sync --frozen)
  • Pre-existing test_ac08 failure confirmed unrelated

wallbreaker_mcp/server.py:
- wb_seed_list now includes builtin static categories as fallback
  when any source-specific query returns 0 external categories
  (was only on source='all'). Fixes AC4 for source='harmbench'
  when HarmBench data isn't cached.

pyproject.toml:
- Add hypothesis>=6.0 to [dependency-groups] dev (was only in
  [project.optional-dependencies] dev, not installed by uv sync)

tests/ (9 files):
- Add pytest.importorskip guards for optional deps (hypothesis,
  fastapi, PIL/Pillow) so tests skip gracefully instead of
  erroring during collection
- Affected: test_security_properties, test_audit_remediation,
  test_tg1..tg7, test_typographic

Result: 0 collection errors (was 9), 60 skipped, 1173 passed,
16 pre-existing failures (test_session_card, test_tg5_harden)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants