{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Adaptive Scenarios\n",
    "\n",
    "Adaptive scenarios select attack techniques for each objective from prior success data. They can\n",
    "spend more attempts on techniques that have worked well while retaining some exploration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "## TextAdaptive\n",
    "\n",
    "`TextAdaptive` uses an epsilon-greedy selector over text-compatible attack techniques. Each\n",
    "objective stops after the first successful technique or after `max_attempts_per_objective`.\n",
    "The direct prompt is available as a baseline comparison.\n",
    "\n",
    "```bash\n",
    "pyrit_scan run adaptive.text_adaptive \\\n",
    "  --initializers target \\\n",
    "  --target openai_chat \\\n",
    "  --dataset-names airt_hate \\\n",
    "  --max-dataset-size 2 \\\n",
    "  --max-attempts-per-objective 2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.\n",
      "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
      "Loaded environment file: ./.pyrit/.env\n",
      "Loaded environment file: ./.pyrit/.env.local\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[pyrit:alembic] No new upgrade operations detected.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TextAdaptive: _EXCLUDED_TECHNIQUES entries ['prompt_sending'] are not in the current scenario-techniques catalog ['best_of_n', 'bijection', 'code_attack', 'context_compliance', 'crescendo_history_lecture', 'crescendo_journalist_interview', 'crescendo_movie_director', 'crescendo_simulated', 'flip', 'many_shot', 'pair', 'red_teaming', 'role_play_movie_script', 'role_play_persuasion', 'role_play_persuasion_written', 'role_play_trivia_game', 'role_play_video_game', 'skeleton_key', 'split_payload', 'tap', 'violent_durian']; the exclusion is a no-op for those entries. Remove stale entries or update the catalog.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d6f5316dd9b4f35bc5746e7ea5009ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Executing TextAdaptive:   0%|          | 0/3 [00:00<?, ?attack/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from pyrit.output import output_scenario_async\n",
    "from pyrit.registry import TargetRegistry\n",
    "from pyrit.scenario import DatasetAttackConfiguration\n",
    "from pyrit.scenario.adaptive import TextAdaptive\n",
    "from pyrit.setup import initialize_from_config_async\n",
    "\n",
    "await initialize_from_config_async(config_path=Path(\"pyrit_conf.yaml\"))  # type: ignore\n",
    "\n",
    "objective_target = TargetRegistry.get_registry_singleton().instances.get(\"openai_chat\")\n",
    "\n",
    "dataset_config = DatasetAttackConfiguration(dataset_names=[\"airt_hate\"], max_dataset_size=2)\n",
    "\n",
    "scenario = TextAdaptive()\n",
    "scenario.set_params_from_args(  # type: ignore\n",
    "    args={\n",
    "        \"objective_target\": objective_target,\n",
    "        \"dataset_config\": dataset_config,\n",
    "        \"max_attempts_per_objective\": 2,\n",
    "    }\n",
    ")\n",
    "await scenario.initialize_async()  # type: ignore\n",
    "\n",
    "scenario_result = await scenario.run_async()  # type: ignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\u001b[1m\u001b[36m                                  📊 SCENARIO RESULTS: TextAdaptive                                  \u001b[0m\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
      "\u001b[36m    • Name: TextAdaptive\u001b[0m\n",
      "\u001b[36m    • Result ID: 1d3c86d4-53b0-4943-9def-085206a8d38e\u001b[0m\n",
      "\u001b[36m    • Scenario Version: 1\u001b[0m\n",
      "\u001b[36m    • PyRIT Version: 1.1.0.dev0\u001b[0m\n",
      "\u001b[36m    • Description:\u001b[0m\n",
      "\u001b[36m        Selects an attack technique for each objective using an epsilon-greedy strategy informed by prior success rates.\u001b[0m\n",
      "\u001b[36m        The scenario stops after a successful attack or after ``max_attempts_per_objective`` attempts.\u001b[0m\n",
      "\n",
      "\u001b[1m  🎯 Target Information\u001b[0m\n",
      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
      "\u001b[36m    • Target Model: gpt-4o\u001b[0m\n",
      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
      "\n",
      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
      "\u001b[36m      • Scorer Type: FloatScaleThresholdScorer\u001b[0m\n",
      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
      "\u001b[36m            • Scorer Type: AzureContentFilterScorer\u001b[0m\n",
      "\u001b[36m            • scorer_type: float_scale\u001b[0m\n",
      "\n",
      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
      "\u001b[31m      • Accuracy: 59.24%\u001b[0m\n",
      "\u001b[36m      • Accuracy Std Error: ±0.0247\u001b[0m\n",
      "\u001b[31m      • F1 Score: 0.5306\u001b[0m\n",
      "\u001b[31m      • Precision: 0.5987\u001b[0m\n",
      "\u001b[31m      • Recall: 0.4764\u001b[0m\n",
      "\u001b[32m      • Average Score Time: 0.04s\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📈 Summary\u001b[0m\n",
      "\u001b[32m    • Total Techniques: 2\u001b[0m\n",
      "\u001b[32m    • Total Attack Results: 6\u001b[0m\n",
      "\u001b[33m    • Overall Success Rate: 50%\u001b[0m\n",
      "\u001b[32m    • Unique Objectives: 2\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: airt_hate\u001b[0m\n",
      "\u001b[33m    • Number of Results: 4\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: baseline\u001b[0m\n",
      "\u001b[33m    • Number of Results: 2\u001b[0m\n",
      "\u001b[33m    • Success Rate: 50%\u001b[0m\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "await output_scenario_async(scenario_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "For more details, see the [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb) and\n",
    "[Configuration](../getting_started/configuration.md)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}