{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Common Scenario Parameters\n",
    "\n",
    "This guide covers the key parameters for configuring scenarios programmatically: datasets,\n",
    "techniques, baseline execution, and custom scorers. All examples use `RedTeamAgent` but the\n",
    "patterns apply to any scenario.\n",
    "\n",
    "> **Two selection axes**: *Techniques* select attack techniques (*how* attacks run — e.g., prompt\n",
    "> sending, role play, TAP). *Datasets* select objectives (*what* is tested — e.g., harm categories,\n",
    "> compliance topics). Use `--dataset-names` on the CLI to filter by content category.\n",
    "\n",
    "> **Running scenarios from the command line?** See the [Scanner documentation](../../scanner/0_scanner.md).\n",
    "\n",
    "## Setup\n",
    "\n",
    "Initialize PyRIT and create the target we want to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
      "Loaded environment file: ./.pyrit/.env\n",
      "Loaded environment file: ./.pyrit/.env.local\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[pyrit:alembic] No new upgrade operations detected.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'many_shot', 'pair', 'red_teaming', 'role_play', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from pyrit.output import output_scenario_async\n",
    "from pyrit.registry import TargetRegistry\n",
    "from pyrit.scenario.foundry import FoundryTechnique, RedTeamAgent\n",
    "from pyrit.setup import initialize_from_config_async\n",
    "\n",
    "await initialize_from_config_async(config_path=Path(\"../../scanner/pyrit_conf.yaml\"))  # type: ignore\n",
    "\n",
    "objective_target = TargetRegistry.get_registry_singleton().instances.get(\"openai_chat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Dataset Configuration\n",
    "\n",
    "`DatasetAttackConfiguration` controls which prompts (objectives) are sent to the target.\n",
    "The simplest approach uses `dataset_names` to load datasets by name from memory.\n",
    "By default, `RedTeamAgent` loads four random objectives from HarmBench [@mazeika2024harmbench]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyrit.scenario import DatasetAttackConfiguration\n",
    "\n",
    "dataset_config = DatasetAttackConfiguration(dataset_names=[\"harmbench\"], max_dataset_size=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "For more control, use `SeedDatasetProvider` to fetch datasets and pass explicit `seed_groups`.\n",
    "This is useful when you need to filter, combine, or inspect the prompts before running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from pyrit.datasets import SeedDatasetProvider\n",
    "from pyrit.models import SeedGroup\n",
    "\n",
    "datasets = await SeedDatasetProvider.fetch_datasets_async(dataset_names=[\"harmbench\"])  # type: ignore\n",
    "seed_groups: list[SeedGroup] = datasets[0].seed_groups  # type: ignore\n",
    "\n",
    "# Pass explicit seed_groups instead of dataset_names\n",
    "dataset_config = DatasetAttackConfiguration(seed_groups=seed_groups, max_dataset_size=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "## Technique Selection and Composition\n",
    "\n",
    "`FoundryTechnique` is an enum that defines which attack techniques the scenario runs. There are\n",
    "three ways to specify techniques:\n",
    "\n",
    "**Individual techniques** — a single converter or multi-turn attack:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": [],
   "source": [
    "single_technique = [FoundryTechnique.Base64]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "**Aggregate techniques** — tag-based groups that expand to all matching techniques. For example,\n",
    "`EASY` expands to all techniques tagged as easy (Base64, Binary, CharSwap, etc.):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9",
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregate_technique = [FoundryTechnique.EASY]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10",
   "metadata": {},
   "source": [
    "**Composite techniques** — pair an attack with one or more converters using `FoundryComposite`.\n",
    "For example, to run Crescendo with Base64 encoding applied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyrit.scenario.foundry import FoundryComposite\n",
    "\n",
    "composite_technique = [FoundryComposite(attack=FoundryTechnique.Crescendo, converters=[FoundryTechnique.Base64])]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "You can mix all three types in a single list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13",
   "metadata": {},
   "outputs": [],
   "source": [
    "scenario_techniques = [\n",
    "    FoundryTechnique.Base64,\n",
    "    FoundryTechnique.Binary,\n",
    "    FoundryComposite(attack=FoundryTechnique.Crescendo, converters=[FoundryTechnique.Caesar]),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "## Baseline Execution\n",
    "\n",
    "The baseline sends each objective directly to the target without any converters or multi-turn\n",
    "techniques. It is included automatically when `include_baseline=True` (the default for\n",
    "scenarios that support a baseline). This is useful for:\n",
    "\n",
    "- **Measuring default defenses** — how does the target respond to unmodified harmful prompts?\n",
    "- **Establishing comparison points** — compare baseline refusal rates against attack-enhanced runs\n",
    "- **Calculating attack lift** — how much does each technique improve over the baseline?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "583be1eaf4264bdd8af1d2a9392deda8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Executing RedTeamAgent:   0%|          | 0/21 [00:00<?, ?attack/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\u001b[1m\u001b[36m                                  📊 SCENARIO RESULTS: RedTeamAgent                                  \u001b[0m\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
      "\u001b[36m    • Name: RedTeamAgent\u001b[0m\n",
      "\u001b[36m    • Scenario Version: 1\u001b[0m\n",
      "\u001b[36m    • PyRIT Version: 0.15.0.dev0\u001b[0m\n",
      "\u001b[36m    • Description:\u001b[0m\n",
      "\u001b[36m        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on\u001b[0m\n",
      "\u001b[36m        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn\u001b[0m\n",
      "\u001b[36m        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The\u001b[0m\n",
      "\u001b[36m        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or\u001b[0m\n",
      "\u001b[36m        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red\u001b[0m\n",
      "\u001b[36m        Teaming Agent library, providing a consistent PyRIT contract for their integration.\u001b[0m\n",
      "\n",
      "\u001b[1m  🎯 Target Information\u001b[0m\n",
      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
      "\u001b[36m    • Target Model: gpt-4o-japan-nilfilter\u001b[0m\n",
      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
      "\n",
      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
      "\u001b[36m      • Scorer Type: FloatScaleThresholdScorer\u001b[0m\n",
      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
      "\u001b[36m            • Scorer Type: AzureContentFilterScorer\u001b[0m\n",
      "\u001b[36m            • scorer_type: float_scale\u001b[0m\n",
      "\n",
      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
      "\u001b[31m      • Accuracy: 59.24%\u001b[0m\n",
      "\u001b[36m      • Accuracy Std Error: ±0.0247\u001b[0m\n",
      "\u001b[31m      • F1 Score: 0.5306\u001b[0m\n",
      "\u001b[31m      • Precision: 0.5987\u001b[0m\n",
      "\u001b[31m      • Recall: 0.4764\u001b[0m\n",
      "\u001b[32m      • Average Score Time: 0.04s\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📈 Summary\u001b[0m\n",
      "\u001b[32m    • Total Techniques: 21\u001b[0m\n",
      "\u001b[32m    • Total Attack Results: 42\u001b[0m\n",
      "\u001b[36m    • Overall Success Rate: 28%\u001b[0m\n",
      "\u001b[32m    • Unique Objectives: 2\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: baseline\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ansi_attack\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ascii_art\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ascii_smuggler\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: base64\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: atbash\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: caesar\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: binary\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: character_space\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: char_swap\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: diacritic\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: flip\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: leetspeak\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: morse\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: suffix_append\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: rot13\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[31m    • Success Rate: 100%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: string_join\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: unicode_confusable\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: unicode_substitution\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: url\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: jailbreak\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "baseline_scenario = RedTeamAgent()\n",
    "baseline_scenario.set_params_from_args(  # type: ignore\n",
    "    args={\n",
    "        \"objective_target\": objective_target,\n",
    "        \"scenario_techniques\": None,  # Uses default techniques; baseline is prepended automatically\n",
    "        \"dataset_config\": dataset_config,\n",
    "    }\n",
    ")\n",
    "await baseline_scenario.initialize_async()  # type: ignore\n",
    "baseline_result = await baseline_scenario.run_async()  # type: ignore\n",
    "await output_scenario_async(baseline_result)  # type: ignore [top-level-await]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16",
   "metadata": {},
   "source": [
    "### Sorting the Per-Group Breakdown by Success Rate\n",
    "\n",
    "By default, the **Per-Group Breakdown** lists groups in the order they were executed. The baseline\n",
    "run above produces a row for every default technique, which makes it hard to spot the most\n",
    "successful ones at a glance. Pass `sort_groups_by_success_rate=True` to `output_scenario_async` to\n",
    "re-render the same result with the highest success rates at the top (groups with equal rates keep\n",
    "their original relative order):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\u001b[1m\u001b[36m                                  📊 SCENARIO RESULTS: RedTeamAgent                                  \u001b[0m\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
      "\u001b[36m    • Name: RedTeamAgent\u001b[0m\n",
      "\u001b[36m    • Scenario Version: 1\u001b[0m\n",
      "\u001b[36m    • PyRIT Version: 0.15.0.dev0\u001b[0m\n",
      "\u001b[36m    • Description:\u001b[0m\n",
      "\u001b[36m        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on\u001b[0m\n",
      "\u001b[36m        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn\u001b[0m\n",
      "\u001b[36m        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The\u001b[0m\n",
      "\u001b[36m        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or\u001b[0m\n",
      "\u001b[36m        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red\u001b[0m\n",
      "\u001b[36m        Teaming Agent library, providing a consistent PyRIT contract for their integration.\u001b[0m\n",
      "\n",
      "\u001b[1m  🎯 Target Information\u001b[0m\n",
      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
      "\u001b[36m    • Target Model: gpt-4o-japan-nilfilter\u001b[0m\n",
      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
      "\n",
      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
      "\u001b[36m      • Scorer Type: FloatScaleThresholdScorer\u001b[0m\n",
      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
      "\u001b[36m            • Scorer Type: AzureContentFilterScorer\u001b[0m\n",
      "\u001b[36m            • scorer_type: float_scale\u001b[0m\n",
      "\n",
      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
      "\u001b[31m      • Accuracy: 59.24%\u001b[0m\n",
      "\u001b[36m      • Accuracy Std Error: ±0.0247\u001b[0m\n",
      "\u001b[31m      • F1 Score: 0.5306\u001b[0m\n",
      "\u001b[31m      • Precision: 0.5987\u001b[0m\n",
      "\u001b[31m      • Recall: 0.4764\u001b[0m\n",
      "\u001b[32m      • Average Score Time: 0.04s\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📈 Summary\u001b[0m\n",
      "\u001b[32m    • Total Techniques: 21\u001b[0m\n",
      "\u001b[32m    • Total Attack Results: 42\u001b[0m\n",
      "\u001b[36m    • Overall Success Rate: 28%\u001b[0m\n",
      "\u001b[32m    • Unique Objectives: 2\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: rot13\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[31m    • Success Rate: 100%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: baseline\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: caesar\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: binary\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: character_space\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: char_swap\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: diacritic\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: flip\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: leetspeak\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: morse\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: suffix_append\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ansi_attack\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ascii_art\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: ascii_smuggler\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: base64\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: atbash\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: string_join\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: unicode_confusable\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: unicode_substitution\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: url\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: jailbreak\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "await output_scenario_async(baseline_result, sort_groups_by_success_rate=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {},
   "source": [
    "To disable the automatic baseline entirely (e.g., when you only want attack techniques with no\n",
    "comparison), set `include_baseline=False` in the run params:\n",
    "\n",
    "```python\n",
    "scenario = RedTeamAgent()\n",
    "scenario.set_params_from_args(\n",
    "    args={\n",
    "        \"objective_target\": objective_target,\n",
    "        \"scenario_techniques\": [FoundryTechnique.Base64],\n",
    "        \"include_baseline\": False,\n",
    "    }\n",
    ")\n",
    "await scenario.initialize_async()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "## Custom Scorers\n",
    "\n",
    "By default, `RedTeamAgent` uses a composite scorer with Azure Content Filter and SelfAsk Refusal\n",
    "scorers. You can override this by passing your own `AttackScoringConfig` with a custom\n",
    "`objective_scorer`.\n",
    "\n",
    "For example, to use an inverted refusal scorer (where \"True\" means the target refused):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d526b73348c54dc9b88765f31ff4ee78",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Executing RedTeamAgent:   0%|          | 0/2 [00:00<?, ?attack/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\u001b[1m\u001b[36m                                  📊 SCENARIO RESULTS: RedTeamAgent                                  \u001b[0m\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
      "\u001b[36m    • Name: RedTeamAgent\u001b[0m\n",
      "\u001b[36m    • Scenario Version: 1\u001b[0m\n",
      "\u001b[36m    • PyRIT Version: 0.15.0.dev0\u001b[0m\n",
      "\u001b[36m    • Description:\u001b[0m\n",
      "\u001b[36m        RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on\u001b[0m\n",
      "\u001b[36m        the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn\u001b[0m\n",
      "\u001b[36m        attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The\u001b[0m\n",
      "\u001b[36m        scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or\u001b[0m\n",
      "\u001b[36m        you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red\u001b[0m\n",
      "\u001b[36m        Teaming Agent library, providing a consistent PyRIT contract for their integration.\u001b[0m\n",
      "\n",
      "\u001b[1m  🎯 Target Information\u001b[0m\n",
      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
      "\u001b[36m    • Target Model: gpt-4o-japan-nilfilter\u001b[0m\n",
      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
      "\n",
      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
      "\u001b[36m      • Scorer Type: TrueFalseInverterScorer\u001b[0m\n",
      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
      "\u001b[36m            • Scorer Type: SelfAskRefusalScorer\u001b[0m\n",
      "\u001b[36m            • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m            • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m            • model_name: gpt-4o-japan-nilfilter\u001b[0m\n",
      "\n",
      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
      "\u001b[33m      Official evaluation has not been run yet for this specific configuration\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📈 Summary\u001b[0m\n",
      "\u001b[32m    • Total Techniques: 2\u001b[0m\n",
      "\u001b[32m    • Total Attack Results: 4\u001b[0m\n",
      "\u001b[32m    • Overall Success Rate: 0%\u001b[0m\n",
      "\u001b[32m    • Unique Objectives: 2\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: baseline\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: base64\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyrit.executor.attack import AttackScoringConfig\n",
    "from pyrit.prompt_target import OpenAIChatTarget\n",
    "from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer\n",
    "\n",
    "refusal_scorer = SelfAskRefusalScorer(chat_target=OpenAIChatTarget())\n",
    "inverted_scorer = TrueFalseInverterScorer(scorer=refusal_scorer)\n",
    "\n",
    "custom_scenario = RedTeamAgent(\n",
    "    attack_scoring_config=AttackScoringConfig(objective_scorer=inverted_scorer),\n",
    ")\n",
    "custom_scenario.set_params_from_args(  # type: ignore\n",
    "    args={\n",
    "        \"objective_target\": objective_target,\n",
    "        \"scenario_techniques\": [FoundryTechnique.Base64],\n",
    "        \"dataset_config\": dataset_config,\n",
    "    }\n",
    ")\n",
    "await custom_scenario.initialize_async()  # type: ignore\n",
    "\n",
    "custom_result = await custom_scenario.run_async()  # type: ignore\n",
    "await output_scenario_async(custom_result)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}