FEAT: Adding AttackTechniqueRegistry#1611
Conversation
Introduce a factory-based registry for reusable attack technique configurations. AttackTechniqueFactory captures technique-specific kwargs at registration time and calls the real attack constructor at create() time with scenario-specific objective_target and scorer. Key design decisions: - Signature validation at construction time catches kwarg typos early - Deep-copy of kwargs at both construction and create() prevents shared mutable state between factory calls - No changes to any existing attack class New files: - pyrit/scenario/core/attack_technique_factory.py - pyrit/registry/instance_registries/attack_technique_registry.py - tests/unit/scenario/test_attack_technique_factory.py (19 tests) - tests/unit/registry/test_attack_technique_registry.py (17 tests) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tests - _validate_kwargs now rejects constructors with **kwargs (VAR_KEYWORD) since parameter validation requires explicitly named params - _build_identifier includes serialized kwarg values (not just keys) so factories with different values produce different hashes - _serialize_value handles primitives, Identifiable objects, and collections recursively; falls back to type name for others - Added tests: VAR_KEYWORD rejection, real PromptSendingAttack validation through @apply_defaults, Identifiable value serialization, same-keys-different-values hash divergence - Rewrote test_create_without_optional_configs to use sentinel pattern instead of **kwargs spy (which is now rejected) - Cleaned up dead code in registry test_create_technique_passes_scoring_config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All registry subclasses had identical get_registry_singleton() overrides that existed solely for return type narrowing. Using Self in the base classes (RegistryProtocol, BaseClassRegistry, BaseInstanceRegistry) gives the same type safety and eliminates 69 lines of boilerplate. Self is imported under TYPE_CHECKING for 3.10 compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| objective_target: PromptTarget, | ||
| attack_scoring_config: AttackScoringConfig, | ||
| attack_adversarial_config: AttackAdversarialConfig | None = None, | ||
| attack_converter_config: AttackConverterConfig | None = None, |
There was a problem hiding this comment.
We'll may want to add more params later, but this covers a bunch to start with; but keep in mind this is only for adjusting existing things. So if we have a factory with WHATEVER, we can simply add another converter to the end without a whole new factory
Remove identical _build_metadata overrides from all 4 instance registry subclasses (converter, scorer, target, attack_technique) by providing a concrete default in the base class. Drop unused MetadataT TypeVar and bound T to Identifiable so the default can call get_identifier() without type: ignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
I think the AttackTechniqueRegistry breaks the BaseInstanceRegistry because we're not storing the AttachTechnique object so we should either rename to AttachTechniqueFactoryRegistry which I think is cheating a little but then we actually have a factory instance which follows the other instance registries. Otherwise, why not create another BaseFactoryRegistry class that AttackTechniqueRegistry inherits from. It's unclear to me whether there would be more factory registries but I don't think as is the AttackTechniqueRegistry fits into the instance registry definition. To me, it's similar to the workflow executor we have which is just xpia but makes more sense to have as a workflow than trying to make it fit in a different type of executor(benchmark/promptgen/attack)
There was a problem hiding this comment.
This is a really good point, but it's a decent sized refactor. It'll make the PR bigger, and I think worth it.
Extract shared infrastructure (singleton, registration, tags, metadata, container protocol) into BaseItemRegistry. BaseInstanceRegistry extends it with get(), get_entry(), get_all_instances() for direct-retrieval registries. AttackTechniqueRegistry now extends BaseItemRegistry directly, so factory registries don't inherit misleading instance-retrieval methods. Hierarchy: BaseItemRegistry (shared core) ├── BaseInstanceRegistry (+ get/get_entry/get_all_instances) │ ├── ConverterRegistry │ ├── ScorerRegistry │ └── TargetRegistry └── AttackTechniqueRegistry (factory — no get()) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2 participants
This PR introduces a factory-based registry for attack techniques, enabling PyRIT components to register reusable attack configurations and instantiate them with scenario-specific
objective_targetand scoring config at execution time. This will be used to shareAttackTechniques between scenarios and leverage metrics (e.g., ASR) to select techniques dynamically.The Challenge
Attack techniques are more complex than other registry items because their constructors mix two kinds of parameters: technique-specific config (converters, tree depth, max turns) that is fixed at registration time, and scenario-specific config (objective target, scorer) that varies per execution. This PR separates the two concerns with a factory pattern.
Solution
AttackTechniqueFactory— Captures anAttackStrategysubclass and its technique-specific kwargs at registration time.create()calls the real constructor with the frozen kwargs plus scenario-provided target/scorer, producing independent instances via deep-copy. Validates kwargs against the constructor signature at factory construction time (rejects typos,**kwargsconstructors, andobjective_targetin frozen kwargs).AttackTechniqueRegistry— Singleton registry (extendsBaseInstanceRegistry) for named factories with tag-based lookup.create_technique()delegates to the stored factory.Where This Fits
Also Included
typing.Selfforget_registry_singleton()return types, eliminating 6 identical boilerplate overrides (~69 lines removed)