{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Resiliency and Retry\n", "\n", "PyRIT provides multiple layers of retry and resiliency mechanisms to handle failures gracefully during security testing. This notebook explains the different retry mechanisms, how they work together, and how to configure them for your use case.\n", "\n", "## Overview: Different Levels of Retry Mechanisms\n", "\n", "PyRIT implements retries at multiple levels, each serving a different purpose:\n", "\n", "1. Low-level Retries (e.g. `pyrit_target_retry`). These automatically attempt to retry certain target errors under known conditions. For example, if there is an HTTP `RateLimitError` on an LLM endpoint, the target can use this attribute to automatically make the HTTP request again with an exponential backoff.\n", "2. Mid-level Retries (e.g. `pyrit_json_retry`). These automatically attempt to retry under known conditions. As an exmaple, if a scorer expects an LLM response to have specific JSON, and that is JSON malformed, the scorer can use this attribute to ask the LLM to try scoring again. This does not have an exponential backoff.\n", "3. High-level/Scenario-Level Retries: High-level workflow that retries the entire scenario when anything goes wrong, including unknown exceptions. If you are sending thousands of prompts, this can happen! It has logic to pick up where it left off in the scenario execution.\n", "\n", "Understanding when and how to use each is key to building robust security testing workflows. It's also important to understand these retries are stacked on top of each other. For example, if a target is unresponsive for five requests (a low level retry), a higher level still counts that as a \"success\"." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## 1. Low-level Retries: API Call Resilience\n", "\n", "### What is Target-Level Retry?\n", "\n", "The `pyrit_target_retry` decorator is a **low-level** retry mechanism that handles transient API failures when communicating with prompt targets (e.g., OpenAI, Azure, custom endpoints).\n", "\n", "### What It Handles\n", "\n", "Target-level retry automatically retries when it encounters:\n", "\n", "- **Rate limit errors** (`RateLimitError`, `RateLimitException`): API rate limits exceeded\n", "- **Empty responses** (`EmptyResponseException`): Target returns no content\n", "\n", "Note if other unknown errors happen, there is no retry. For example, we don't generally want to retry 10 times if there is an auth failure :)\n", "\n", "### Configuration\n", "\n", "Target-level retries are configured via environment variables:\n", "\n", "```python\n", "RETRY_MAX_NUM_ATTEMPTS=10 # Maximum retry attempts (default: 10)\n", "RETRY_WAIT_MIN_SECONDS=5 # Minimum wait between retries (default: 5)\n", "RETRY_WAIT_MAX_SECONDS=220 # Maximum wait between retries (default: 220)\n", "```\n", "\n", "The decorator uses **exponential backoff** - wait time increases exponentially with each retry attempt.\n", "\n", "### When to Use\n", "\n", "Target-level retry is **automatically applied** to target implementations. You typically don't need to think about it - it's built into PyRIT's target infrastructure.\n", "\n", "It's most valuable when:\n", "- Working with rate-limited APIs (OpenAI, Azure, etc.)\n", "- Network instability causes occasional empty responses\n", "- You want automatic handling of transient communication failures" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "## 2. Mid-level Retries: Response Format Resilience\n", "\n", "### What is JSON-Level Retry?\n", "\n", "The `pyrit_json_retry` decorator handles failures when parsing JSON responses from targets. Due to the probabilistic nature of LLMs, they can occasionally return malformed JSON (or whatever format we're expecting).\n", "\n", "### What It Handles\n", "\n", "JSON-level retry automatically retries when:\n", "\n", "- **Invalid JSON** (`InvalidJsonException`): Target returns non-parseable JSON, a JSON\n", " value other than a top-level object, or (for true/false scorers) a `score_value`\n", " outside the `\"true\"`/`\"false\"` domain (for example, `\"refusal\"`)\n", "\n", "### Configuration\n", "\n", "JSON retries use the same environment variables as target retries:\n", "\n", "```python\n", "RETRY_MAX_NUM_ATTEMPTS=10 # Maximum retry attempts\n", "```\n", "\n", "One difference is it does not have an exponential backoff, it is usually retried immediately.\n", "\n", "### When to Use\n", "\n", "Like target-level retry, JSON retry is **automatically applied** where needed. It's particularly useful when:\n", "- Working with targets that return structured data\n", "- LLMs occasionally generate malformed JSON\n", "- You need automatic recovery from parsing errors\n", "\n", "\n", "## 3. Scenario-Level Retries: High-Level Workflow Resiliency\n", "\n", "### What is Scenario-Level Retry?\n", "\n", "Scenario-level retry is the **highest-level** retry mechanism in PyRIT. When enabled via the `max_retries` parameter, it allows an entire scenario execution to automatically retry if an exception occurs during the workflow.\n", "\n", "But also note, if you rerun the same scenario manually, that follows the same logic and is always an option.\n", "\n", "### Key Features\n", "\n", "- **Picks up where it left off**: On retry, the scenario skips already-completed objectives and continues from the point of exception\n", "- **Broad exception handling**: Catches any exception during scenario execution (network issues, target failures, scoring errors, etc.)\n", "- **Configurable attempts**: Set `max_retries` to control how many additional attempts are allowed\n", "- **Progress tracking**: The `number_tries` field in `ScenarioResult` tracks total attempts\n", "\n", "### How It Works\n", "\n", "When you call `scenario.run_async()`, PyRIT:\n", "\n", "1. **Initial Attempt**: Executes all atomic attacks in sequence\n", "2. **On Exception**: If an exception occurs, checks if retries remain\n", "3. **Retry with Resume**: On retry, queries memory to identify completed objectives and skips them\n", "4. **Continue from Exception Point**: Executes only the remaining objectives\n", "5. **Repeat**: Continues retrying until success or `max_retries` exhausted\n", "\n", "### Example: Basic Scenario with Retries" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "No new upgrade operations detected.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'adversarial_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TargetRegistry entry 'objective_scorer_chat' not found. Falling back to default OpenAIChatTarget.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using fallback default objective scorer: TrueFalseInverterScorer with chat target: OpenAIChatTarget\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "085f263e4205496285a45b08cbed15a2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Executing RedTeamAgent: 0%| | 0/2 [00:00