FEAT Add VisualLeakBench dataset loader (arXiv:2603.13385)

[Copilot]

Adds PyRIT support for the VisualLeakBench / MM-SafetyBench dataset — a multimodal benchmark of 1,000 adversarial images testing LVLMs against OCR injection (harmful instructions embedded in images) and PII leakage (social engineering to extract SSNs, passwords, API keys, etc.).

Fixes #1530

New: _VisualLeakBenchDataset

• Fetches metadata.csv from YoutingWang/MM-SafetyBench on GitHub and downloads images with local caching
• Produces image+text prompt pairs per example, linked by prompt_group_id (image at sequence=0, category-specific query at sequence=1)
• Maps harm categories: ocr_injection for OCR entries; pii_leakage + normalized PII type (e.g. ssn, api_key) for PII entries
• Supports filtering via categories, pii_types, and max_examples
• Registered with tags={"default", "safety", "privacy"}, modalities=["image", "text"] for SeedDatasetFilter discovery

New enums

Enum	Values
VisualLeakBenchCategory	OCR_INJECTION, PII_LEAKAGE
VisualLeakBenchPIIType	EMAIL, DOB, PHONE, PASSWORD, PIN, API_KEY, SSN, CREDIT_CARD

Usage

from pyrit.datasets.seed_datasets.remote import (
    _VisualLeakBenchDataset,
    VisualLeakBenchCategory,
    VisualLeakBenchPIIType,
)

# Load only PII leakage examples for SSN and Password
loader = _VisualLeakBenchDataset(
    categories=[VisualLeakBenchCategory.PII_LEAKAGE],
    pii_types=[VisualLeakBenchPIIType.SSN, VisualLeakBenchPIIType.PASSWORD],
)
dataset = await loader.fetch_dataset()

Test coverage

28 unit tests covering init validation, OCR/PII pair creation, harm category mapping, category/PII-type filtering, max_examples, failed image handling, and metadata correctness. Integration test updated to cap image downloads at max_examples=6 (same pattern as _VLSUMultimodalDataset).

Additionally: refactoring VisualLeakBench + VLSU loaders

Applied during code review to both the new loader and the existing _VLSUMultimodalDataset:

• Moved prompt constants from module-level to class-level (OCR_INJECTION_PROMPT, PII_LEAKAGE_PROMPT)
• Made class metadata attributes immutable (frozenset for tags, tuple for modalities/harm_categories)
• Extracted _matches_filters() and _build_prompt_pair_async() helpers from long fetch_dataset methods
• Replaced manual setup_memory fixture with @pytest.mark.usefixtures("patch_central_database") in tests
• Renamed test_failed_image_download_skips_example → test_all_images_fail_produces_empty_dataset for clarity

Fix: strict enum validation across all dataset loaders

Replaced leaky set-based validation with strict isinstance checks in all dataset loaders that accept enum-typed filter parameters. Previously, raw strings matching enum values (e.g. "PII Leakage") would silently pass __init__ validation but crash downstream when .value was called on them.

Added _validate_enums / _validate_enum helpers to _RemoteDatasetLoader base class. Applied to:

• visual_leak_bench_dataset.py (categories, pii_types)
• vlsu_multimodal_dataset.py (categories)
• harmbench_multimodal_dataset.py (categories)
• promptintel_dataset.py (severity, categories)

Non-enum values are now rejected immediately with a clear error listing valid enum members.

[adrian-gavrila]

All looks good except for one potential minor bug