{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# WebSocket Copilot Target\n", "\n", "The `WebSocketCopilotTarget` is an alternative to the `PlaywrightCopilotTarget` that is designed to be more reliable by minimizing dependence on browser automation. Instead of driving the Copilot UI, it communicates directly with Copilot over a WebSocket connection.\n", "\n", "By default, this target uses automated authentication which requires:\n", "- `COPILOT_USERNAME` and `COPILOT_PASSWORD` environment variables\n", "- Playwright installed: `pip install playwright && playwright install chromium`\n", "\n", "Some environments are not suited for automated authentication (e.g. they have security policies with retrieving tokens or have MFA). See the [Alternative Authentication](#alternative-authentication-with-manualcopilotauthenticator) section below." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## Basic Usage with `PromptSendingAttack`\n", "\n", "The simplest way to interact with the `WebSocketCopilotTarget` is through the `PromptSendingAttack` class." ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 1 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m Tell me a joke about AI\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Here’s a lighthearted one for you:\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m **Why did the AI go broke?**\u001b[0m\n", "\u001b[33m Because it kept working for *exposure*!\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m 😄 Want me to share a few more AI jokes, or maybe some clever tech humor?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n" ] } ], "source": [ "# type: ignore\n", "from pyrit.executor.attack import PromptSendingAttack\n", "from pyrit.output import output_attack_async\n", "from pyrit.prompt_target import WebSocketCopilotTarget\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)\n", "\n", "target = WebSocketCopilotTarget()\n", "attack = PromptSendingAttack(objective_target=target)\n", "\n", "objective = \"Tell me a joke about AI\"\n", "\n", "result = await attack.execute_async(objective=objective)\n", "await output_attack_async(result)" ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## Multi-Turn Conversations\n", "\n", "The `WebSocketCopilotTarget` supports multi-turn conversations by leveraging Copilot's server-side conversation management. It automatically generates consistent `session_id` and `conversation_id` values for each PyRIT conversation, enabling Copilot to maintain context across multiple turns.\n", "\n", "However, this target does not support setting a system prompt nor modifying conversation history. As a result, it cannot be used with attack strategies that require altering prior messages (such as PAIR, TAP, or flip attack) or in contexts where a chat-style target (one that declares `supports_multi_turn=True` and `supports_editable_history=True`) is required.\n", "\n", "Here is a simple multi-turn conversation example:" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 1 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m I'm thinking of a number between 1 and 10.\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Alright, I’ll take a guess… is it **7**?\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m (That’s a classic favorite!)\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m Do you want me to keep guessing until I get it, or should I try to guess it in one shot?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 2 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m It's greater than 5.\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Thanks for the hint! That narrows it down to **6, 7, 8, 9, or 10**.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m My next guess is… **8**.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m Am I getting warmer? Or should I try again?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 3 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m It's an even number.\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Great clue! So now the possibilities are **6, 8, or 10**.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m I’ll go with **10** this time.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m Did I nail it? Or should I keep guessing?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 4 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m What number am I thinking of?\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Based on your clues:\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m - It’s **greater than 5**.\u001b[0m\n", "\u001b[33m - It’s an **even number**.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m That leaves **6, 8, or 10**.\u001b[0m\n", "\u001b[33m Since my last guess was 10 and you didn’t confirm, I’ll go with **8** this time.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m Is your number **8**?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n" ] } ], "source": [ "from pyrit.executor.attack import MultiPromptSendingAttack\n", "from pyrit.models import Message\n", "from pyrit.output import output_attack_async\n", "from pyrit.prompt_target import WebSocketCopilotTarget\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)\n", "\n", "target = WebSocketCopilotTarget()\n", "\n", "prompts = [\n", " \"I'm thinking of a number between 1 and 10.\",\n", " \"It's greater than 5.\",\n", " \"It's an even number.\",\n", " \"What number am I thinking of?\",\n", "]\n", "\n", "messages = [Message.from_prompt(prompt=p, role=\"user\") for p in prompts]\n", "multi_turn_attack = MultiPromptSendingAttack(objective_target=target)\n", "\n", "result = await multi_turn_attack.execute_async(\n", " objective=\"Engage in a multi-turn conversation about a number guessing game\",\n", " user_messages=messages,\n", ")\n", "\n", "await output_attack_async(result)" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Alternative Authentication with `ManualCopilotAuthenticator`\n", "\n", "If browser automation is not suitable for your environment, you can use the `ManualCopilotAuthenticator` instead. This authenticator accepts a pre-obtained access token that you can extract from your browser's DevTools.\n", "\n", "How to obtain the access token:\n", "\n", "1. Open the Copilot webapp (e.g., https://m365.cloud.microsoft/chat) in a browser.\n", "2. Open DevTools (F12 or Ctrl+Shift+I).\n", "3. Go to the Network tab.\n", "4. Filter by \"Socket\" connections or search for \"Chathub\".\n", "5. Start typing in the chat to initiate a WebSocket connection.\n", "6. Look for the latest WebSocket connection to `substrate.office.com/m365Copilot/Chathub`.\n", "7. You may find the `access_token` in the request URL or in the request payload.\n", "\n", "You can either pass the token directly or set the `COPILOT_ACCESS_TOKEN` environment variable." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 1 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m Hello! Who are you?\u001b[0m\n", "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m Hi Richard! I’m Copilot, your conversational AI assistant here to help you think, build, debug,\u001b[0m\n", "\u001b[33m research, create, or just chat.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m You can ask me about:\u001b[0m\n", "\u001b[33m - Technical deep dives or architecture brainstorming\u001b[0m\n", "\u001b[33m - Writing or refactoring code\u001b[0m\n", "\u001b[33m - Productivity help and summarization\u001b[0m\n", "\u001b[33m - Creative projects\u001b[0m\n", "\u001b[33m - Or anything else you’re curious about\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m What would you like to do today?\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n" ] } ], "source": [ "from pyrit.auth import ManualCopilotAuthenticator\n", "from pyrit.executor.attack import PromptSendingAttack\n", "from pyrit.output import output_attack_async\n", "from pyrit.prompt_target import WebSocketCopilotTarget\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)\n", "\n", "# Option 1: Pass the token directly\n", "# auth = ManualCopilotAuthenticator(access_token=\"eyJ0eXAi...\")\n", "\n", "# Option 2: Use COPILOT_ACCESS_TOKEN environment variable\n", "auth = ManualCopilotAuthenticator()\n", "\n", "target = WebSocketCopilotTarget(authenticator=auth)\n", "attack_manual = PromptSendingAttack(objective_target=target)\n", "\n", "result_manual = await attack_manual.execute_async(objective=\"Hello! Who are you?\")\n", "await output_attack_async(result_manual)" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "## Multimodal Support (Text and Images)\n", "\n", "The `WebSocketCopilotTarget` supports multimodal input, allowing you to send both text and images in a single message. Images are automatically uploaded to Copilot's file service and referenced in the conversation using the same process as the Copilot web interface.\n", "\n", "Here's an example of sending an image along with a text prompt:" ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[34m🔹 Turn 1 - USER\u001b[0m\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[34m Answer the question from the image\u001b[0m\n", "\u001b[34m ..\\converters\\benign_cake_question.jpg\u001b[0m\n" ] }, { "data": { "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCACWAJYDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD0iiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACjIornlultNXnldSRuYYH1oA6HNFYFpOs+tiYAhWyef92rLaw5LvFbFokPLE/wCcUAa1FZz6ph7YpHmObHzE8jnBpi6vmWdTGNqKxU564oA1KQsFUsxAUdSe1Q2kzXFssrqFLdhWddak8v2iGKDdGqkM+envQBrJIkq7o3Vx0ypzTqxLC8Wy0x3ZdzGUhR68CrMGps1wsM8BjL/dOfyoA0qKp6r/AMg2b/gP/oQpNJ/5BsX4/wAzQBaE0TSFFkQuOqhhkU+ufWWSHV7h44zI+Wwoq/BqbT2jyLDmRCAV3YHPfNAGjTI5o5c+XIj467WBxVC31Nprg28kaBiDgo24dKzbC++xxSkLuZiuB+dAHSUVnT6jJAIozEvnuuSpbAX8ansb0XiN8u10OGGc0AWqKKKACs63sJItSluGKFGLYAPPJ+laNFAGcunyDVDckp5Zzxk56Y9KrHSrqNZIYpk8mQ856/yraooAzJ9LLWUMMTLvjOdzcZ9agk0eYxxBHjDKpDkk88k+nvW1RQA2JBFEkY6KoArKOl3KSzCCVFilBBz1+nSteigDIGkSGxMLOgkDl1IzjoBinQ6dcNdRzXUqt5eNoX26Vq0UAV72BrmzkhQgM2ME9OoNFjA1taJE5BZc5I6dasUUAZL6ddi7mnhljUvnHJzg/hSf2RIti0QkXzWYMfTjt+ta9FAGVBpk8d5HMzRYVcEIMdiPSq40Obymy8fmZG3BOMd+30rdooAybnTJ5xFKXjMyqFcHOGx3q1YWrWyP5gjDsf4OmKuUUAFFFFABRRRQAUUUUAFVrm8+zuEWCWQ7dx2DgCrNZd/a3M16hCCSHGNpbAB96ALlpdx3kRdARg4INMu9QitGCEM8h/hWodJtprZJRMm3cRjkH+VMvbS4F8t3bqHI6qTQBOdR/dRsLeZncn5AORj1qvcXyXenXIVWR0xlW+tNuob65SFniGATviDYHXvz6VFHp1ykd0vlAbxhQGGD831oAfZX6W1pBFsaR3JwB9atz6nHDK0YjkkZRl9g4Ws+PTruBoJo4suv3l3D1+vcU6406f7XJIsRkSQk4D4Iz60ATXx+0Qi7S6aOHZ8qjOS2TVnSmmayDTEkk/KT1xVeSzmXTookt1Zt+5kZun61qLnYuQAccgdqAFooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//9k=", "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAACWCAIAAACzY+a1AAAbcUlEQVR4AWI8ceIEwygYyoBpKDt+1O0gMBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAH5EchIxjQLQD+/fv3//9/RkZGBgaGf//+MTAwMDIywhkQZzAyMkLU/P//HyKClQQ7HERglYWYDJIG2/X//38mJlAoQUxmYGCAM7Bqh7gQWTvEwP8wwMTE9PfvX4heiGIIm2wS5DjyNEOcRJ5eMnSxsrIyMjL+/fv3////zMzMEM8zMzNDYvTfv3+QgIbEK0QWly0Ql+OJZmQFLCwskNQDsYiRkZGJiQmSdLCaDzEWQsIjkoGBgQkMIIawsLBAXAtRhtUc4gVxRiGy9cjGYQ0drILIuihnQ1IuJAQh8fTnz5+/f/8yMjIyMzNDkvafP3/gsUu5jRATkG2Be5OYoEdWA0kTkCCFiP8DA4gVFJIsePRDLENTQLwgmkYKuZAC7f///5B4AqdpJki+hKTxf//+QQpS/BZBogGrL9A0QlQyMzNDciEkDUFKUUgeQlMP4SK7AdmW////g2MNVFrADUFWDNFOBokzFyJbDzcXkpogUpA0BakbICJwZbRgMDExQWyHBB8kRP7+/QsRZ2RkZGFhYWVl/fPnD55SjiTXQqz7/fs3hMHIyMjKysrCwgLh4vIjJHog0Q9XCSkqIFqYmZkhhQcDAwMkWUDEySbx5UJMQyEuQxPHKoimhnIuJM6YmJggQQCp/CDlJzMz858/fyCtBkhgwcMO015IEGOKYxWBKIZUfv/+/YNEJ8QKXAkFkprhYQJP6HAXQhRAtEOSI1ariRfEmQvhCRYeHBDXQBwHCVBICQMJO+KtJE8lxF5Iyv379y8zGPz+/RsSoH///mVjYwNFKbit8fv3b4K2wA1E8wuad1hYWCB5BdJ0YmZmhovgsQIST5AQ+/cPWlpA6unfv39zcHBQpQiFOABnLoRED8QpkMoG4iBI1EKKdYg4VZISxDV4SEiyhfgcElUfPnx48ODB79+/IYWbjo4OhPH37192dnZIuGMaCPcFRAoSfxBvwttKyLH469evx48ff/jw4d+/f2xsbBISEmJiYhC9WElIaMANhATXnz9/rl27BjGWhYVFQ0MDEp1YTSBVkAWXBiYmpmvXrn379g3iJkVFRUFBQUiR8unTpzt37kDY7Ozsurq6kGIN0uKAt/shrock9n///sHbBcjNdGQtrKyskHiCmAZxGKQogxgCEYHE4t+/fyE13/////+CAQsLyC+QTAkxB6IeEpqQIhduNaRNC4nOixcvQlQyMDDo6+szMYFaSRBdkECHt6F+/vzJysoKr8kgjoeUUpDQgDSvILZA3AkJvT9gAMm+TExM7Ozsv379YmVlhZcif/78YWJi+vPnDwcHx69fvyARDBGEuBySvJD9BXczzoIUEnYQF0BUQ3wCMQvSiIBEEiQvQlwMKcEgoQmxD+5hCJeJiQlSo0B0QfpJbGxs8PiABCtEHNIFhJgAsRfSWoGEMsRGSMpgZWWFBOWvX78groXEOiQm2NjYILZDKiSIAiJJSNULiRUGBgZIoMOjChJK////Z2VlhRjIwsLy8+dPSKn779+/P3/+QMoGSNxDnAEJJUjTmpGR8c+fP+zs7AwMDJCohcQZJP4gFQfEcFyOxxeFEDdBXAlpRzAzg6qfP39AaZ+BgYGFhQWSnCFVPaSegItAIgOStyBsSLKFB+7fv38hkf3r1y9IlwCSMiDpBtLbg7gBUm5DSkhmZmaIk/78+QOJRXiZyczMzMHBAQluiIchEf/jxw+IUyFhATGTeBIS7hBLIXHz588fSB6CJDUmJqafP39CDPz79y8HBwektobYzsAAGoKAGALx4P///9nY2CCOh4TMjx8/IPENT5EQiyBBAQlwSFKA2IJMggofZD6cDdEMz2fgkgCU2dnZ2b9//w7J9ZBSC+IIiPrfv39/AwNGRkZ+fn4ODg54nvv48SMknhgYGISFhVlZWZ8+fQpx6P///8XExJiYmF69egUpYTg4OHh4eCC2QKIE0uz8/v3758+fP336xM/Pz8nJCcl5bGxskCrtz58/bGxsP3/+/Pv3769fvz58+MDHx8fMzMzNzQ1RCSmj4H4khgEp1SEkJPF9+PDh169fkEJPWFgY4jBIXH779u3nz5+Q8OHi4hISEoJUKxDbIQEFKVQ+fvz47ds3SFZjZmYWERGBxO7Xr19//Pjx+fNnXjCApDlIKQ3J0JCkgOxynFEIqWMg+YaZmfnnz59fvnyBhOn3798hBRTEZRDjGBkZHz9+/Pr1a3g76NmzZ////1dRUREWFv4FbhRAMisrK6uwsPCPHz+ePn0KSdqsrKx8fHz///9/9OgRxJ/y8vIQwyEKGBgY/vz58+PHj7t37/748YONje3169fc3NwQb0PSLxMTEycn57dv3548eQJJLv/+/Xv16hWkP6eqqsrDwwPJmhAHE0lC3AN3xrt3775///77929I7f7u3TslJSVmZuYPHz7cu3cPEj3///+HZPpHjx5pampycHBAghFiO6See/bs2devXyH+EhAQEBYW/vv379OnT9+8eQMpPN+8ecPKyioOBpDyBpKUMZ3NgikEEYHXNwwMDD9//nwJBhCfQApriFMgPvz79+/Dhw8/ffoEKd8h9TYk1dy6dUtBQUFMTIyTk/P79++Q0gCSWSGJGuJtSB0GKVv+/fvHy8sLyaAsLCwQDzAxMT169OjHjx/wau/r16+QRAqJpP///3/58uXRo0cfP36E5AlkL9y9e1dOTo6HhwdiKcSPxJCQegHiMEZGRnjigHj/06dPL168kJKSgqR4VlZWeGkEybgPHz5UUVGBFI+QBPf///9v3759/fr19+/fTExMLCwscnJyzMzMjx49evHiBaTKhKj8+vXrs2fPmJiYIG1gSHhi5kKcdSGk+oE0OiA1FiRb/AP3ciABAclwkKT04cMHSAXAxMTEz88vLCwMcQczM/Pbt2///v0rICAAiScmJiZwWfuNATzbAPHzDzCAuI+bmxsS05AUA0kl79+///r1Kzs7+8+fPxkZGeXk5ISEhCCVBKSwYmRkhBSzkBKCk5MTkm7+/fv348eP379/f/jwAeJsYmIOrgZiGsSpDAwMHBwckpKSIiIiEF8zMDB8/vwZEkpSUlJqamo6OjpaWlqcnJyQjPjjxw9IIQ/JiJAi8dmzZ5A6hZGRUVJSElLdvHnzhoWF5devX8LCwqamphISElxcXP/+/fv06RMklcNLbLjbIAycuZAZPHYMSc6QoIR4A5IqIZohTv/37x9y0crFxaWoqPjv37/Pnz9DCpyvX7/++/dPQEAA4nRIRfLjxw9I8+T///8/f/789u0bJHsxMTEJCAjAfQhJen///oV0b379+sXBwSEsLCwuLv7r16/3799DhtYgnvz06RO8mygiIgJRc+PGDUhy/Pjxo4yMDKmxCGkVQyLg79+/wsLCkpKSnz59gmRHSET++fOHj4/vz58/L1++/PLlC6T19OvXL0jRB3EbpHiEtAE/fPgA6dpC0hkDA8O3b98g+YGVlfXr16/Xrl2DhA8DA8OXL18gZQAktOEhD2fgjEK4lZAWjYyMjKCgIKRJ/ePHj/v370MKCgj5/ft3SGRDKiS4a379+gWJpy9fvnBzc7Oxsf348YORkRFSYzMzM4uKij579oyZmfnXr1+Qhs/fv38hHVBIcQ1JtkxMTN+/f4fk8p8/f/Ly8kIae5CMDqmW/v//DylmIQOn7OzskBYHpL0K78xA0jKklP737x8klCHmQMp/ZBFIfENISCxC2lCQ9uSfP384OTkhPdRnz569ePEC4kKQs5iYID0ZiO2QPABRCalrIMELqSP+/fsHiSdIIQRpEEGiDdL4gDTTIA6AqIHHHyhNIHOQ2ZAWP7yIZ2FhYWZm5uLi+vPnz8+fPyHhC8kr////5+bmhgQKExPT169fIS7+9u3b79+/IY1vDg4OVlZWTk5OSO/43bt3kCiHVE5///799OkTJycnZJAMUs1Aym1I8EFKMEhrE1KWQvIuxJ+Q4uHv379cXFwfPnyAVJ9fvnwREhJ6/fo1pCMEiS1IiEBqJggJj36IURDrIJUoxO8Qd0LSMTMz85cvX/j5+d++ffvv3z+IS4SFhSH5D1IeSEtLi4iIvHjx4vnz55BhP0goQcyExOKvX78gPv358+fTp09lZWXZ2dkh6f7///9cXFyQhAKpTSAFMiTiIS1Y5GjCF4WQAhPSY4N3+CBtKkj1AAloiN8gLoDUSd+/f79x4wZc4////wUFBSEKBAUF3759C0nskOKFj4+Pl5cXEtk/f/789++fkJAQvOqFxA3ECh4eHnht8eDBgy9fvnz8+BGS6iGVDQsLC7ysZmJievHixffv3798+QLpqP3//x/SwYCYCUkfkLCAp2tIWEN8ByHh4c7Gxgbx0atXr96/f//t2zdICmBkZIQP5kGy1Js3byAlPKRdCikSIJOOkFqfgYFBSkoK0k6BuFNWVpabm5uRkZGTk/PHjx+fPn0SEhJiYmL68uXLt2/fFBUV2djYID6FOBiNxNmcgbieg4MDoh/iFEhxDGk1QUIBklk5ODhkZWUhk9H//v37+vUrJLIhkQGp/BkZGSElJMQ/kIwFGXj89esXpCiD9CYhbEhYQ+KPgYFBQEAA0jpnAwNIVQQpHtnY2CBm8vDwCAsLQ2ovRkbGz58/Q9iQtp+ysvKPHz8g5QekfIbIQsoMCAkJZYjXIJ6FlNiQggeSVn78+AFJkczMzHx8fOLi4mxsbBCfQnpfr1+/hkQnZOQFUj5BKgsmJiZWVlYpKSmIFkjWv3v3Lg8Pj6SkJKQSZWVlfQAGkNQAcTAkHCChSmwUQmomSHsE0rSB2AcRhzgLUjpDKjxRUVF5eXl4CQDJatzc3Orq6oKCgpBW5Z8/fyCtTUh65+HhYWBg4OHhgaRoSCDy8vLCyy54XEI8oK6uDql7IBWwiooKJAVAwhQy6KWoqCgvLw8pNiEhzsDAICoqqqmpCSn6II1YZvCEAyR0ICUVZJgCYjIkLUIGFCEkvPciKirKysr68+dPFhYWdnZ2GRkZyPCTtLQ0xMu/fv0SERGBdzMgDoa4hIODA267uLj4nz9/IOMY79+///nzp5SUlJycHCQ5gqdhQINQkBCApA/IkBBa/IESOq7rmJmYmCAWQCpeyLAWXD88/f779w9S20FCEJLK3r59y8TExMfHBwlKSF8CUvAyMjL+/v0bkrMhsn///oVUSJDRSEhtBCn6IXkFYikkAUFc9evXL0irHTLYCElPEJMhAQppMUEMgdTfEMMhRQvEEEiqYmFhgYwVQLq/XFxcENdCsiDEAZA6BdIOglR4X79+ZWJigpQKkDqMjY3t9+/fX758gTS1IJ5iZWWFFDDw0IP4DpIiGRgYIBEGb0VDkjKk+c3CwsLFxQUJVUj4QBwML/YhwYIvCkFy4CVikJIEohOS7SBJFdJShzgOYjpEFpJCIY1PSAKHFMIQEyBdH3hAQyIGkkog4QvxHiRHQsIOkhIhGRcyoAxp5kHCFxKykIwLSSUQ50EaI5AWByRRQ0bX4OkGUmZCKgJIcEBcCPE4xEB4iEOqecjgJyTOILbDowHiJEiVCUnHkOwLCRmI+ZDyGe5ISGvg////kNoUUvtCsg3c+xA1kKiFaIc4G2IghMRZF0KCEpIYITUNJD7gAQQZ+IZ4GDmsIeEFGa2HhCMkDiDB8fv3b3Z2dkj5DKkqICECiVRIgQOxFBKIEPMhTQNI/EGqGUiHDGICpAyE50WIRoggpHMGqTIhYQeJckgEQEIK4khIloXHIoQBiQCIvZDogaQkiFGQjiw8lCEZkY2NDeI2SP6DWIGc3P///w/J6ExgAPHC79+/4YkA7mVoDIFXnECyAcSPEHE4OXqv/ZAH+HLhkPfcyACjUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55gG++kNaeg3R7IYMykI4tZNwOMuAEGS6A9HMh3WdIRxsyFAAZM4KMP0CmkCDmQEQgLofP9UBGDG7evAlZZ/b//381NTUeHh7IeAdkzAEyOAIZEICMHkDGcSBqIKMNX79+/fnz5+/fv5mZmdnZ2Xl4eCA9fUjfHzL+CTEN4gD6kAMWhZA4gKxcgrAhg23wES/IqDxkPAkyygwZ74BMyUJCCjKkAhmCgQzdgebPwEvoGRkZIZOu8HEWyMQhZIwJMqQHNwo++gMZh4JYChmOgbD//Pnz4MGDT58+QRIZJGExMTGpq6uzsbFBTIYMF0Oso0/kQWwBJWQIi84kZCAYkuohkQfPXhAuMzMzZLDq9+/fkBXAkGEtyHJIyKAlJM9B5tAhmQOyWhcyEPX379+fP3/Co+fnz5/wSIVoh+ROiCGQ2UqIFRDtkJkpSBZnZ2f/9OkTvBiAOOnPnz/37t2DjERDxkXpH3+g4oHOMQe3DjLACC+mIKEJKTwhg5Z///6FT0RAZoUgIcgEXtMAUQmJPEghDBmJZQUDSAwxMDBwcnJCxoghm4Qhs96QApydnR1SeEKW6kImmCAFIyQ/QQaBIdkOkoYgs8dgG0BrtxkZGb9+/QopG+ADs3AP0o0xYAUppJyEzJFCFnP8/PkTstqFm5sbEkyQqVrIwDRkSgQS+r/B4OvXr9++gZbBcXBw8PPzc3FxQVadfP78GZKhWVlZeXl5IevDIJNEkPFlSLn9+/fvN2/eQFT+//+fl5cXMnXOzMz88ePHDx8+cHJyQkyA1LIcHBwKCgqQCaC7d+++f/8eIg7JuJBCBTKxR7fIg1g0YFEICce/f//ev3//8+fPkEUl4P0toG1m/Pz8cnJykBCB5BXIdCBkovHJkyfwNb7//////fu3srIyGxvblStXIKUlJF9CijhxcXH4wmLIeD8kv7579+7p06eQWQXIQqx///79/Pnz1q1bkOVGP3/+hCxaUFNT+/v3r46ODqQKhKyghJTJXFxckAoVkpUhToWELN3IAa4L////D1nAyMrKCtl1B5kF/fDhA2RVHCSGIDMyHBwc3759u3v37suXL1lYWCCroSAtGkg5DIkhyI4FSJHIyMj49u1byAoaSCqBhOy/f/8+fPgAKR5ZWFjExcUhraF79+5BZtMgk2W/fv369u3bzZs3ITNokOWWt27dghTyjIyMCgoK8NYyRCPEfHqSA5YLIVHFyMjIw8MjJCTEy8vLxsb25s2be/fuQWqv79+/c3FxQapMSIj8////8+fPX758gShgZGQUFxfn5eWFLGz5+/cvPz+/mJgYJOu8e/fuzp07kA1TP378gCyVg4QypJkDWcILWY0hISHx////N2/eQGLuz58/JiYm379/v3nz5t+/f3/8+PH9+3dOTk4GBobHjx9DuhzMzMzy8vLs7OyQ5iikzQyJTohr6UYOWBRCcsn////l5eWfP3/+5s2bT58+gdpX4G26kNWxIiIikMoP3lX48uULJGgg66AhK9Uh5dv////l5OSeP3/+/fv3T58+wasoJiYmyGpuyBoLSBv4yZMnEAX///+XlpaGNEEhyykgnbxbt25ByltIDH3//p2DgwOyCBFinaKiIg8PD2TBI6SehiyPoH9ZOmBRCGmL/v///8aNG5C9Fv/+/YM0SSAtT4gCSNsS0rKHzHdDGg6QVamQdiCkvfrz58+bN29CylVI1/v79++QhAKvqCDlLWSBCKQUhaxFg7SbIKvOIaYxMDBA1kBAIubjx4+Q9aKQlRYMDAz8/PyQuhbSQ4VYRP/4AyVuSKKmPwnJDc+fP//69SvksApdXV1GRsZLly5BWomQPj6k8wCp3iCbKCFbv/78+fPp0ydeXl5I15uVlfXNmzeQVgwnJ6e6uvrv378vXrwI6VnCi0fI4lpmZmZBQcGXL19C9N6/f9/c3PzXr1+QtU+QLQDCwsKQUheyLo2DgwMSRHp6ep8+feLm5oYUGJDaF9KzhKywgsQoRDF9yAHLhZAuGqQXAVmO9+rVqx8/fsBrGkjvEJK6IYUhZA8bfAXw8+fPP3z4wM3N/ebNGxUVFUjdBlm7BtnbABlegXQnIMZCGkf//v0TERGBrMiGrC5/+PChtLQ0BwfH169fITHx798/fn7+b9++ffz4kY2NDbIZ4/Pnz3fv3oW4XFZWVkREBJLCIMUGZESC/lE4YC1SFhiAjGj8////xYsXHz58gFRLkIwI2XUGyTqQnjhkKx4kdUO2RL958waSoSH7QCGtJMhSbkg2giyChbAhxSBk7RMkn0G2l75+/RqyyBpSt0EaVnfu3Hn16hVkLwekS/P+/XvIWA9knSCknoa7H5LaIG6jJzlgUcjAwPDjxw9JSUnIclNIbKmpqUEKJUiTAT5wDFnQx8DAAFkKLSkpCRnqhAQoZFBNRkZGSEjo27dvkNFRNTU1yOAOJPThBkI210Hak5B8A9nr8/z5c3Z2dk1NTSEhIUglysICWqcK6e1A1hyLi4tDBlc5ODj4+Pgg3VZIBxciDimZ6Rl/IH/hWgpMa3dAWnqQUICMqkAWekNKPEjgwpfSQsorSA8Bkuf+/v0L2fcL2dUAyWoMDAyQkpCVlRXe8YCMncJPIoC0ZSCtJMgiZvjqbEgtCClIIeM+nJyckOwLaU9BxvPevn0L2T4HsQJiIKTxBWnE0jro0MwfsEWIkHFISA6A1B+QaIOQkEIJUuJBQgrS/IMUqpBghaiBxC5EJShJws6zhORmiCzEBHgDBFJawnMMZPscpCkEabJCOgmQaIP0PSC7UiDLQSERD8mj8GIfwoC3btBCmabcAYtCmvpqRIGBrAtHVEDTDoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFQx6MRuGQB6NROOTBaBQOeTAahUMejEbhkAejUTjkwWgUDnkwGoVDHoxG4ZAHo1E45MFoFA55MBqFgA35KAQARapF2LLnHxkAAAAASUVORK5CYII=", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m\u001b[33m🔸 ASSISTANT\u001b[0m\n", "\u001b[33m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[33m The image contains only text that says:\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m **“How do I bake a cake?”**\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m Here’s a simple, beginner‑friendly way to bake a basic cake:\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m **Ingredients**\u001b[0m\n", "\u001b[33m - 1 and 1/2 cups flour\u001b[0m\n", "\u001b[33m - 1 cup sugar\u001b[0m\n", "\u001b[33m - 1/2 cup butter (softened)\u001b[0m\n", "\u001b[33m - 2 eggs\u001b[0m\n", "\u001b[33m - 1/2 cup milk\u001b[0m\n", "\u001b[33m - 1 and 1/2 teaspoons baking powder\u001b[0m\n", "\u001b[33m - 1 teaspoon vanilla extract\u001b[0m\n", "\u001b[33m - A pinch of salt\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m **Steps**\u001b[0m\n", "\u001b[33m 1. Preheat your oven to **180°C / 350°F**.\u001b[0m\n", "\u001b[33m 2. Grease a cake pan with butter or line it with baking paper.\u001b[0m\n", "\u001b[33m 3. In a bowl, cream together the butter and sugar until smooth.\u001b[0m\n", "\u001b[33m 4. Add the eggs one at a time and mix well.\u001b[0m\n", "\u001b[33m 5. Add the vanilla.\u001b[0m\n", "\u001b[33m 6. In a separate bowl, mix flour, salt, and baking powder.\u001b[0m\n", "\u001b[33m 7. Add the dry ingredients to the wet ingredients gradually, mixing gently.\u001b[0m\n", "\u001b[33m 8. Pour in the milk and stir until the batter is smooth.\u001b[0m\n", "\u001b[33m 9. Pour the batter into the pan.\u001b[0m\n", "\u001b[33m 10. Bake for **25–35 minutes**, or until a toothpick comes out clean.\u001b[0m\n", "\u001b[33m 11. Let it cool, then enjoy or decorate.\u001b[0m\n", "\u001b[33m \u001b[0m\n", "\u001b[33m If you want, I can help you with flavors, frosting, or a more advanced recipe.\u001b[0m\n", "\n", "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "from pyrit.executor.attack import PromptSendingAttack\n", "from pyrit.models import Message, MessagePiece\n", "from pyrit.output import output_attack_async\n", "from pyrit.prompt_target import WebSocketCopilotTarget\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)\n", "\n", "target = WebSocketCopilotTarget()\n", "attack = PromptSendingAttack(objective_target=target)\n", "\n", "# Replace with the path to your actual image file\n", "image_path = Path(\"../converters/benign_cake_question.jpg\")\n", "\n", "# Create a multimodal message with both text and image pieces\n", "multimodal_message = Message(\n", " message_pieces=[\n", " MessagePiece(\n", " role=\"user\",\n", " original_value=\"Answer the question from the image\",\n", " converted_value=\"Answer the question from the image\",\n", " conversation_id=\"test_conversation\",\n", " original_value_data_type=\"text\",\n", " converted_value_data_type=\"text\",\n", " ),\n", " MessagePiece(\n", " role=\"user\",\n", " original_value=str(image_path),\n", " converted_value=str(image_path),\n", " conversation_id=\"test_conversation\",\n", " original_value_data_type=\"image_path\",\n", " converted_value_data_type=\"image_path\",\n", " ),\n", " ]\n", ")\n", "\n", "result = await attack.execute_async(objective=\"Answer the question from the image\", next_message=multimodal_message)\n", "await output_attack_async(result)" ] } ], "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.18" } }, "nbformat": 4, "nbformat_minor": 5 }