{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Custom Scenario Parameters\n", "\n", "Sometimes a scenario needs a custom parameter that a user can set without\n", "editing source code (`max_turns`, dataset names, feature flags, etc.).\n", "Scenarios can declare typed parameters that flow from CLI flags into\n", "`self.params`.\n", "\n", "This is different from [Common Scenario Parameters](./1_common_scenario_parameters.ipynb),\n", "which covers the framework-level configuration surface (datasets, techniques,\n", "scorers, baseline). This guide is about parameters that scenario authors add\n", "on their own classes.\n", "\n", "## Declaring a parameter\n", "\n", "`Parameter` is the unified declaration shared by initializers and scenarios.\n", "To declare one on a scenario, override the `additional_parameters()` classmethod\n", "and return a list of just your extras — the base composes these with the common\n", "run inputs, so you never repeat (or accidentally drop) them. Here's the actual\n", "declaration on [`Scam`](../../../pyrit/scenario/scenarios/airt/scam.py):\n", "\n", "```python\n", "@classmethod\n", "def additional_parameters(cls) -> list[Parameter]:\n", " \"\"\"Declare custom parameters this scenario accepts from the CLI.\"\"\"\n", " return [\n", " Parameter(\n", " name=\"max_turns\",\n", " description=\"Maximum conversation turns for the persuasive_rta technique.\",\n", " param_type=int,\n", " default=5,\n", " ),\n", " ]\n", "```\n", "\n", "At runtime the framework calls `supported_parameters()` to inspect declarations\n", "(the common inputs plus your `additional_parameters()`). It's a classmethod, so\n", "this works without instantiating the scenario (which would wire up memory and\n", "scorers):" ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "./AppData/Local/miniconda3/Lib/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.5.0) or chardet (7.4.3)/charset_normalizer (3.3.2) doesn't match a supported version!\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[pyrit:alembic] No new upgrade operations detected.\n", "Parameter(name='max_turns', description='Maximum conversation turns for the persuasive_rta technique.', default=5, param_type=, destination=)\n" ] } ], "source": [ "from pyrit.scenario.airt.scam import Scam\n", "from pyrit.setup import initialize_pyrit_async\n", "from pyrit.setup.initializers.techniques import TechniqueInitializer\n", "\n", "await initialize_pyrit_async(memory_db_type=\"InMemory\") # type: ignore [top-level-await]\n", "await TechniqueInitializer().initialize_async() # type: ignore [top-level-await]\n", "\n", "for param in Scam.supported_parameters():\n", " print(param)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "Each declaration lives inside the scenario class body, in the\n", "`additional_parameters()` classmethod. End users don't construct `Parameter`\n", "objects themselves; they pass values via CLI flags.\n", "\n", "Each `Parameter` carries:\n", "\n", "- **name**: dict key in `self.params`, converted to `--kebab-case` for the CLI\n", "- **description**: shown in `list-scenarios` and `--help`\n", "- **default**: value used when not supplied; deep-copied per run\n", "- **param_type**: `str`, `int`, `float`, `bool`, a `Literal[...]`/`Enum` (a\n", " constrained scalar that carries its own allowed set), a `list[...]` of any of\n", " those, or `None` (raw passthrough)\n", "\n", "A more complete declaration list might look like:" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parameter(name='objective', description='Goal the attack pursues', default=None, param_type=, destination=)\n", "Parameter(name='max_turns', description='Conversation cap', default=5, param_type=, destination=)\n", "Parameter(name='mode', description='Speed mode', default='fast', param_type=typing.Literal['fast', 'slow'], destination=)\n", "Parameter(name='tags', description='Tag list', default=['default'], param_type=list[str], destination=)\n" ] } ], "source": [ "from typing import Literal\n", "\n", "from pyrit.models import Parameter\n", "\n", "# What a scenario author would return from additional_parameters():\n", "example_declarations = [\n", " # Scalar with no default — author must guard against None at run time\n", " Parameter(name=\"objective\", description=\"Goal the attack pursues\", param_type=str),\n", " # Scalar with default\n", " Parameter(name=\"max_turns\", description=\"Conversation cap\", default=5, param_type=int),\n", " # Constrained scalar: a Literal behaves like an enum (the type *is* the allowed set)\n", " Parameter(\n", " name=\"mode\",\n", " description=\"Speed mode\",\n", " default=\"fast\",\n", " param_type=Literal[\"fast\", \"slow\"],\n", " ),\n", " # List parameter\n", " Parameter(name=\"tags\", description=\"Tag list\", default=[\"default\"], param_type=list[str]),\n", "]\n", "\n", "for p in example_declarations:\n", " print(p)" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## Reading the value\n", "\n", "After the framework calls `set_params_from_args` (which `pyrit_scan` and\n", "`pyrit_shell` do automatically), `self.params[\"max_turns\"]` returns the\n", "user's value, or the declared default if no value was supplied. There's\n", "no need for a `.get()` fallback. Mutable defaults like `[\"a\", \"b\"]` are\n", "deep-copied on each run, so changes in one scenario instance don't leak\n", "into another.\n", "\n", "Here's how Scam reads the parameter, in `_get_atomic_attack_from_technique`:\n", "\n", "```python\n", "attack_strategy = RedTeamingAttack(\n", " objective_target=self._objective_target,\n", " attack_scoring_config=self._scorer_config,\n", " attack_adversarial_config=self._adversarial_config,\n", " max_turns=self.params[\"max_turns\"],\n", ")\n", "```\n", "\n", "Programmatic users (constructing the scenario in Python rather than going\n", "through the CLI) get the same behavior: `initialize_async()` materializes\n", "declared defaults the first time it runs, so `self.params[\"max_turns\"]`\n", "is populated even when no explicit `set_params_from_args` call was made." ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Setting a parameter from the CLI\n", "\n", "`pyrit_scan` adds one flag per declared parameter, converting the name from\n", "`snake_case` to `--kebab-case`. Scenario flags go after the scenario name\n", "and can be mixed with built-in flags:\n", "\n", "```bash\n", "# Use the declared default (5)\n", "pyrit_scan run airt.scam --target my_target --initializers target\n", "\n", "# Override\n", "pyrit_scan run airt.scam --target my_target --initializers target --max-turns 10\n", "```\n", "\n", "The same flags work in `pyrit_shell`:\n", "\n", "```text\n", "pyrit_shell> run airt.scam --target my_target --initializers target --max-turns 10\n", "```\n", "\n", "Scenario-declared flags don't appear in `pyrit_scan run --help`\n", "(that shows only the built-in run options); use `list-scenarios` to discover\n", "the full list of a scenario's parameters.\n", "\n", "## Discovering parameters via `list-scenarios`\n", "\n", "`pyrit_scan list-scenarios` prints declared parameters alongside each scenario's\n", "other metadata (description, techniques, datasets). The same formatter the\n", "CLI uses is callable programmatically:" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'adversarial_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'adversarial_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'adversarial_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Available Scenarios:\n", "================================================================================\n", "\u001b[1m\u001b[36m\n", " airt.scam\u001b[0m\n", " Class: Scam\n", " Description:\n", " Scam scenario evaluates an endpoint's ability to generate scam-related\n", " materials (e.g., phishing emails, fraudulent messages) with primarily\n", " persuasion-oriented techniques.\n", " Aggregate Techniques:\n", " - all, single_turn, multi_turn\n", " Available Techniques (3):\n", " context_compliance, role_play, persuasive_rta\n", " Default Technique: all\n", " Default Datasets (1, max 4 per dataset):\n", " airt_scams\n", " Supported Parameters:\n", " - max_turns (int) [default: '5']: Maximum conversation turns for the persuasive_rta technique.\n", "\u001b[1m\u001b[36m\n", " foundry.red_team_agent\u001b[0m\n", " Class: RedTeamAgent\n", " Description:\n", " RedTeamAgent is a preconfigured scenario that automatically generates\n", " multiple AtomicAttack instances based on the specified attack\n", " techniques. It supports both single-turn attacks (with various\n", " converters) and multi-turn attacks (Crescendo, RedTeaming), making it\n", " easy to quickly test a target against multiple attack vectors. The\n", " scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into\n", " their constituent attack techniques, or you can specify individual\n", " techniques directly. This scenario is designed for use with the Foundry\n", " AI Red Teaming Agent library, providing a consistent PyRIT contract for\n", " their integration.\n", " Aggregate Techniques:\n", " - all, easy, moderate, difficult\n", " Available Techniques (25):\n", " ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,\n", " character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,\n", " suffix_append, string_join, unicode_confusable, unicode_substitution,\n", " url, jailbreak, tense, multi_turn, crescendo, pair, tap\n", " Default Technique: easy\n", " Default Datasets (1, max 4 per dataset):\n", " harmbench\n", "\n", "================================================================================\n", "\n", "Total scenarios: 2\n" ] } ], "source": [ "from pyrit.backend.services.scenario_service import get_scenario_service\n", "from pyrit.cli._output import print_scenario_list\n", "\n", "# Show scam (declares a parameter) and red_team_agent (none), so the\n", "# Supported Parameters section is visible in one and absent in the other.\n", "demo_names = {\"airt.scam\", \"foundry.red_team_agent\"}\n", "response = await get_scenario_service().list_scenarios_async(limit=200) # type: ignore\n", "print_scenario_list(items=[s for s in response.items if s.scenario_name in demo_names])" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "Notice the `Supported Parameters:` section under `airt.scam`. It's absent\n", "from `foundry.red_team_agent` because that scenario doesn't declare any\n", "custom parameters. Existing scenarios that don't opt in to this feature\n", "render exactly as before.\n", "\n", "## Resume validation\n", "\n", "When you ask to resume by passing `scenario_result_id` to a Scenario constructor,\n", "PyRIT verifies that the stored result is an exact match for the current\n", "configuration. Any deviation aborts with a `ValueError` rather than silently\n", "starting a fresh scenario, so original progress is never orphaned without the\n", "caller noticing. Mismatch axes:\n", "\n", "- **Stored id not found** in memory (typo, wiped DB, never persisted)\n", "- **Scenario name differs** (e.g., a Scam id passed to a Cyber constructor)\n", "- **Scenario version differs** (release drift between save and resume)\n", "- **Effective parameters differ** from those persisted with the original run\n", "\n", "A typical param-mismatch error message:\n", "\n", "```text\n", "Scenario result id '7c3f...' has mismatched parameters (changed: max_turns).\n", "Drop scenario_result_id to start a new scenario, or pass matching parameters to resume.\n", "```\n", "\n", "The diff names changed/added/removed keys but never prints values, so sensitive\n", "parameters don't leak into exception output. To start fresh, drop the\n", "`scenario_result_id` argument; to resume, pass the same params used originally.\n", "\n", "A dedicated `pyrit_scan --resume` CLI flag that loads stored params for you\n", "(so you can't supply mismatching ones in the first place) is tracked as a\n", "separate follow-up." ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "`Scam.max_turns` was previously hardcoded to `5` in\n", "`_get_atomic_attack_from_technique`. Replacing it with a `Parameter` of\n", "`default=5` keeps the original behavior (no new flag is required to run\n", "Scam as before) while making the value overridable for users who need it." ] } ], "metadata": { "jupytext": { "main_language": "python" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }