{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Why Instance Registries?\n", "\n", "Some components need configuration that can't easily be passed at instantiation time. For example, scorers often need:\n", "- A configured `chat_target` for LLM-based scoring\n", "- Specific prompt templates\n", "- Other dependencies\n", "\n", "Instance registries let initializers register fully-configured instances that are ready to use." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## Listing Available Instances\n", "\n", "Use `instances.get_names()` to see registered instances, or `instances.list_metadata()` for details." ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "No new upgrade operations detected.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Registered scorers: ['SelfAskRefusalScorer::5f719b8e']\n" ] } ], "source": [ "from pyrit.prompt_target import OpenAIChatTarget\n", "from pyrit.registry import ScorerRegistry\n", "from pyrit.score import SelfAskRefusalScorer\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY) # type: ignore\n", "\n", "# Get the registry singleton\n", "registry = ScorerRegistry.get_registry_singleton()\n", "\n", "# Register a scorer instance for demonstration\n", "chat_target = OpenAIChatTarget()\n", "refusal_scorer = SelfAskRefusalScorer(chat_target=chat_target)\n", "registry.instances.register(refusal_scorer)\n", "\n", "# List what's available\n", "names = registry.instances.get_names()\n", "print(f\"Registered scorers: {names}\")" ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## Getting an Instance\n", "\n", "Use `instances.get()` to retrieve a pre-configured instance by name. The instance is ready to use immediately." ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieved scorer: \n", "Scorer type: SelfAskRefusalScorer\n" ] } ], "source": [ "# Get the first registered scorer\n", "if names:\n", " scorer_name = names[0]\n", " scorer = registry.instances.get(scorer_name)\n", " print(f\"Retrieved scorer: {scorer}\")\n", " print(f\"Scorer type: {type(scorer).__name__}\")" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Inspecting Metadata\n", "\n", "Scorer metadata includes the scorer type and identifier for tracking." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "SelfAskRefusalScorer::5f719b8e:\n", " Class: SelfAskRefusalScorer\n", " Type: true_false\n", "\n", "\u001b[1m 📊 Scorer Information\u001b[0m\n", "\u001b[37m ▸ Scorer Identifier\u001b[0m\n", "\u001b[36m • Scorer Type: SelfAskRefusalScorer\u001b[0m\n", "\u001b[36m • scorer_type: true_false\u001b[0m\n", "\u001b[36m • score_aggregator: OR_\u001b[0m\n", "\u001b[36m • model_name: gpt-4o-japan-nilfilter\u001b[0m\n", "\n", "\u001b[37m ▸ Performance Metrics\u001b[0m\n", "\u001b[33m Official evaluation has not been run yet for this specific configuration\u001b[0m\n" ] } ], "source": [ "from pyrit.output import output_scorer_async\n", "\n", "# Get metadata for all registered scorers\n", "metadata = registry.instances.list_metadata()\n", "for item in metadata:\n", " print(f\"\\n{item.unique_name}:\")\n", " print(f\" Class: {item.class_name}\")\n", " print(f\" Type: {item.params.get('scorer_type', 'unknown')}\")\n", "\n", " await output_scorer_async(scorer_identifier=item)" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "## Filtering\n", "\n", "Use `list_metadata()` with `include_filters` and `exclude_filters` dictionaries to filter scorers by any metadata property. `include_filters` requires ALL criteria to match (AND logic). `exclude_filters` excludes items matching ANY criteria. Filters use exact match for simple types and membership check for list types." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True/False scorers: ['SelfAskRefusalScorer::5f719b8e']\n", "Refusal scorers: ['SelfAskRefusalScorer::5f719b8e']\n", "True/False refusal scorers: ['SelfAskRefusalScorer::5f719b8e']\n" ] } ], "source": [ "# Filter by scorer_type (based on isinstance check against TrueFalseScorer/FloatScaleScorer)\n", "true_false_scorers = registry.instances.list_metadata(include_filters={\"scorer_type\": \"true_false\"})\n", "print(f\"True/False scorers: {[m.unique_name for m in true_false_scorers]}\")\n", "\n", "# Filter by class_name\n", "refusal_scorers = registry.instances.list_metadata(include_filters={\"class_name\": \"SelfAskRefusalScorer\"})\n", "print(f\"Refusal scorers: {[m.unique_name for m in refusal_scorers]}\")\n", "\n", "# Combine multiple filters (AND logic)\n", "specific_scorers = registry.instances.list_metadata(\n", " include_filters={\"scorer_type\": \"true_false\", \"class_name\": \"SelfAskRefusalScorer\"}\n", ")\n", "print(f\"True/False refusal scorers: {[m.unique_name for m in specific_scorers]}\")" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## Using Target Initializer\n", "\n", "You can optionally use the `TargetInitializer` to automatically configure and register targets that use commonly used environment variables (from `.env_example`). This initializer does not strictly require any environment variables - it simply registers whatever endpoints are available." ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Registered targets after initialization: ['adversarial_chat', 'azure_content_safety', 'azure_foundry_deepseek', 'azure_foundry_mistral_large', 'azure_foundry_phi4', 'azure_gpt4o_unsafe_chat', 'azure_gpt4o_unsafe_chat2', 'azure_gpt4o_unsafe_chat_temp9', 'azure_ml_phi', 'azure_openai_gpt35_chat', 'azure_openai_gpt4_chat', 'azure_openai_gpt4o', 'azure_openai_gpt4o_temp9', 'azure_openai_gpt5_1', 'azure_openai_gpt5_4', 'azure_openai_gpt5_responses', 'azure_openai_gpt5_responses_high_reasoning', 'azure_openai_integration_test', 'azure_openai_realtime', 'azure_openai_responses', 'azure_openai_video', 'google_gemini', 'ollama', 'openai_chat', 'openai_completion', 'openai_image_platform', 'openai_tts_azure', 'openai_tts_platform', 'platform_openai_chat', 'platform_openai_responses']\n" ] } ], "source": [ "from pyrit.registry import TargetRegistry\n", "from pyrit.setup import initialize_pyrit_async\n", "from pyrit.setup.initializers import TargetInitializer\n", "\n", "# Using built-in initializer\n", "await initialize_pyrit_async( # type: ignore\n", " memory_db_type=\"InMemory\", initializers=[TargetInitializer()]\n", ")\n", "\n", "# Get the registry singleton\n", "registry = TargetRegistry.get_registry_singleton()\n", "# List registered targets\n", "target_names = registry.instances.get_names()\n", "print(f\"Registered targets after initialization: {target_names}\")" ] } ], "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }