Skip to content

Add SORRY-Bench dataset loader - #17

Open
panditdhamdhere wants to merge 1 commit into
JailbrokenAI:mainfrom
panditdhamdhere:feat/sorrybench-loader
Open

Add SORRY-Bench dataset loader#17
panditdhamdhere wants to merge 1 commit into
JailbrokenAI:mainfrom
panditdhamdhere:feat/sorrybench-loader

Conversation

@panditdhamdhere

Copy link
Copy Markdown
Contributor

Summary

Adds a sorrybench dataset source (roadmap Phase 1) so sweeps, campaigns, and leaderboards can pull from the SORRY-Bench 44-class safety taxonomy via source="sorrybench".

Changes

  • wallbreaker/datasets/sorrybench.py — loader + normalize to {id, behavior, category, source, benign}
  • wallbreaker/datasets/__init__.py — register sorrybench
  • wallbreaker/prompts.py — document in agent source= battery list
  • tests/test_datasets.py — registry, normalize, and cached battery tests

Notes

  • Uses the public SillyTilly/SorryBench CSV mirror (sorry_bench_202406.csv).
  • The official sorry-bench/sorry-bench-202503 HuggingFace dataset is gated and requires access approval.
  • Categories are prefixed sorry-{1..44} for stratified sampling.

Test plan

  • pytest tests/test_datasets.py -q (9 passed)
  • Download URL returns 200 (~490 prompts)

@twaai

twaai commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

thanks for the contribution! merging.

@Gitanuj993

Copy link
Copy Markdown
Contributor

👍 No harmful and No unwanted code detected !,
It adds a new dataset called sorrybench

@Gitanuj993 Gitanuj993 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 No harmful and No unwanted code detected !,
It adds a new dataset called sorrybench

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants