--- title: ModelAudit - Static Security Scanner for ML Models description: Scan AI/ML models for security vulnerabilities, malicious code, and backdoors. Supports PyTorch, TensorFlow, ONNX, Keras, and 30+ model formats. keywords: [ model security, AI security, ML security scanning, static analysis, malicious model detection, pytorch security, tensorflow security, model vulnerability scanner, ] sidebar_label: Overview sidebar_position: 1 --- # Model Scanning ## Overview ModelAudit is a lightweight static security scanner for machine learning models accessible through Promptfoo. It scans AI/ML models for potential security risks before deployment. Promptfoo provides a wrapper command `promptfoo scan-model` that integrates ModelAudit scanning capabilities. ![example model scan results](/img/docs/modelaudit/modelaudit-result.png) Promptfoo also includes a UI that allows you to set up a scan: ![model scan](/img/docs/modelaudit/model-audit-setup.png) And displays the results: ![model scan results](/img/docs/modelaudit/model-audit-results.png) ## Purpose AI/ML models can introduce security risks through: - Malicious code embedded in pickled models - Suspicious TensorFlow operations - Potentially unsafe Keras Lambda layers - Dangerous pickle opcodes - Encoded payloads hidden in model structures - Risky configurations in model architectures - Malicious content in ZIP archives - Embedded executables in binary model files - Hidden credentials (API keys, tokens, passwords) - Network communication patterns (URLs, IPs, sockets) - JIT/Script execution in TorchScript and ONNX models ModelAudit helps identify these risks before models are deployed to production environments, ensuring a more secure AI pipeline. ## Installation ### Using Promptfoo The easiest way to use ModelAudit is through Promptfoo: ```bash # Install Promptfoo globally npm install -g promptfoo # Install modelaudit dependency pip install modelaudit ``` ### Standalone Installation You can also install ModelAudit directly: ```bash # Basic installation pip install modelaudit # With optional dependencies for specific model formats pip install modelaudit[tensorflow,h5,pytorch] # For all dependencies pip install modelaudit[all] # Or install specific components: pip install modelaudit[tensorflow,h5,pytorch] # Core ML frameworks pip install modelaudit[cloud,mlflow] # Remote model access pip install modelaudit[numpy1] # NumPy 1.x compatibility ``` ### Docker ```bash # Pull from GitHub Container Registry docker pull ghcr.io/promptfoo/modelaudit:latest # Use specific variants docker pull ghcr.io/promptfoo/modelaudit:latest-full # All ML frameworks docker pull ghcr.io/promptfoo/modelaudit:latest-tensorflow # TensorFlow only # Run with Docker docker run --rm -v $(pwd):/data ghcr.io/promptfoo/modelaudit:latest scan /data/model.pkl ``` ## Usage ### Basic Command Structure ```bash promptfoo scan-model [OPTIONS] PATH... ``` ### Examples ```bash # Scan a single model file promptfoo scan-model model.pkl # Scan a model directly from HuggingFace without downloading promptfoo scan-model https://huggingface.co/bert-base-uncased promptfoo scan-model hf://microsoft/resnet-50 # Scan from cloud storage promptfoo scan-model s3://my-bucket/model.pt promptfoo scan-model gs://my-bucket/model.h5 # Scan from MLflow registry promptfoo scan-model models:/MyModel/1 # Scan multiple models and directories promptfoo scan-model model.pkl model2.h5 models_directory # Export results to JSON promptfoo scan-model model.pkl --format json --output results.json # Export results to SARIF for security tool integration promptfoo scan-model model.pkl --no-write --format sarif --output results.sarif # Add custom blacklist patterns promptfoo scan-model model.pkl --blacklist "unsafe_model" --blacklist "malicious_net" # Enable verbose output promptfoo scan-model model.pkl --verbose # Set file size limits promptfoo scan-model models/ --max-size 1GB # Generate Software Bill of Materials promptfoo scan-model model.pkl --sbom sbom.json # Enable strict mode for security-critical scans promptfoo scan-model model.pkl --strict # List scanner IDs and run only selected scanners promptfoo scan-model --list-scanners promptfoo scan-model models/ --scanners pickle,tf_savedmodel # Preview scan without actually processing promptfoo scan-model model.pkl --dry-run ``` See the [Advanced Usage](./usage.md) guide for detailed authentication setup for cloud storage, JFrog, and other remote sources. :::info Alternative Installation and Usage - **Standalone**: Install modelaudit directly using `pip install modelaudit`. `modelaudit scan` behaves the same as `promptfoo scan-model`. - **Web Interface**: For a GUI experience, use `promptfoo view` and navigate to `/model-audit` for visual scanning and configuration. ::: ### Options | Option | Description | | ------------------- | --------------------------------------------------------------- | | `--blacklist`, `-b` | Additional blacklist patterns to check against model names | | `--format`, `-f` | Output format (`text` \| `json` \| `sarif`) [default: text] | | `--output`, `-o` | Output file path (prints to stdout if not specified) | | `--timeout`, `-t` | Scan timeout in seconds [default: 300] | | `--verbose`, `-v` | Enable verbose output | | `--max-size` | Maximum total size to scan (e.g., `500MB`, `1GB`) | | `--sbom` | Generate CycloneDX Software Bill of Materials with license info | | `--strict` | Fail on warnings; enable stricter validation | | `--dry-run` | Preview scan without processing files | | `--scanners` | Only run selected scanners | | `--exclude-scanner` | Exclude scanners from the default scanner set | | `--list-scanners` | Print available scanner IDs and class names | | `--quiet` | Suppress non-critical output | | `--progress` | Force-enable progress reporting | | `--no-cache` | Disable caching of downloaded files | | `--no-write` | Skip writing results to database | ## Web Interface Promptfoo includes a web interface for ModelAudit at `/model-audit` with visual path selection, real-time progress tracking, and detailed results visualization. **Access:** Run `promptfoo view` and navigate to `http://localhost:15500/model-audit` **Key Features:** - Visual file/directory selection with current working directory context - GUI configuration for all scan options (blacklist patterns, timeouts, file limits) - Live scanning progress and tabbed results display with severity color coding - Scan history and automatic installation detection ## Supported Formats ModelAudit supports scanning 30+ specialized file format scanners across major ML frameworks: ### Model Formats | Format | Extensions | Description | | ------------------------- | ---------------------------------------------------- | ------------------------------------------------------- | | **PyTorch** | `.pt`, `.pth`, `.bin` | PyTorch model files and checkpoints | | **TensorFlow SavedModel** | `.pb`, directories | TensorFlow's standard model format | | **TensorFlow Lite** | `.tflite` | Mobile-optimized TensorFlow models | | **TensorRT** | `.engine`, `.plan` | NVIDIA GPU-optimized inference engines | | **Keras** | `.h5`, `.keras`, `.hdf5` | Keras/TensorFlow models in HDF5 format | | **ONNX** | `.onnx` | Open Neural Network Exchange format | | **SafeTensors** | `.safetensors` | Hugging Face's secure tensor format | | **GGUF/GGML** | `.gguf`, `.ggml`, `.ggmf`, `.ggjt`, `.ggla`, `.ggsa` | Quantized models (LLaMA, Mistral, etc.) | | **Flax/JAX** | `.msgpack`, `.flax`, `.orbax`, `.jax` | JAX-based model formats | | **JAX Checkpoints** | `.ckpt`, `.checkpoint`, `.orbax-checkpoint` | JAX training checkpoints | | **Pickle** | `.pkl`, `.pickle`, `.dill` | Python serialization (includes Dill) | | **Joblib** | `.joblib` | Scikit-learn and general ML serialization | | **NumPy** | `.npy`, `.npz` | NumPy array storage formats | | **PMML** | `.pmml` | Predictive Model Markup Language (XML) | | **ZIP Archives** | `.zip` | Compressed model archives with recursive scanning | | **Container Manifests** | `.manifest` | OCI/Docker layer scanning | | **Binary Files** | `.bin` | Auto-detected format (PyTorch, ONNX, SafeTensors, etc.) | ### Remote Sources | Source | URL Format | Example | | ------------------------ | ---------------------------------------------------- | ------------------------------------------------------- | | **HuggingFace Hub** | `https://huggingface.co/`, `https://hf.co/`, `hf://` | `hf://microsoft/resnet-50` | | **Amazon S3** | `s3://` | `s3://my-bucket/model.pt` | | **Google Cloud Storage** | `gs://` | `gs://my-bucket/model.h5` | | **Cloudflare R2** | `r2://` | `r2://my-bucket/model.safetensors` | | **MLflow Registry** | `models:/` | `models:/MyModel/1` | | **JFrog Artifactory** | `https://*.jfrog.io/` | `https://company.jfrog.io/artifactory/models/model.pkl` | | **DVC** | `.dvc` files | `model.pkl.dvc` | ## Security Checks Performed The scanner looks for various security issues, including: - **Malicious Code**: Detecting potentially dangerous code in pickled models - **Suspicious Operations**: Identifying risky TensorFlow operations and custom ONNX operators - **Unsafe Layers**: Finding potentially unsafe Keras Lambda layers - **Blacklisted Names**: Checking for models with names matching suspicious patterns - **Dangerous Serialization**: Detecting unsafe pickle opcodes, nested pickle payloads, and decode-exec chains - **Enhanced Dill/Joblib Security**: ML-aware scanning with format validation and bypass prevention - **Encoded Payloads**: Looking for suspicious strings that might indicate hidden code - **Risky Configurations**: Identifying dangerous settings in model architectures - **XML Security**: Detecting XXE attacks and malicious content in PMML files - **Embedded Executables**: Detecting Windows PE, Linux ELF, and macOS Mach-O files - **Container Security**: Scanning model files within OCI/Docker container layers - **Compression Attacks**: Detecting zip bombs and decompression attacks - **Weight Anomalies**: Statistical analysis to detect potential backdoors - **Format Integrity**: Validating file format structure - **License Compliance**: Detecting AGPL obligations and commercial restrictions - **DVC Integration**: Automatic resolution and scanning of DVC-tracked models - **Secrets Detection**: Finding embedded API keys, tokens, and credentials - **Network Analysis**: Detecting URLs, IPs, and socket usage that could enable data exfiltration - **JIT Code Detection**: Scanning TorchScript, ONNX custom ops, and other JIT-compiled code ## Interpreting Results The scan results are classified by severity: - **CRITICAL**: Definite security concerns that should be addressed immediately - **WARNING**: Potential issues that require review - **INFO**: Informational findings, not necessarily security concerns - **DEBUG**: Additional details (only shown with `--verbose`) Some issues include a "Why" explanation to help understand the security risk: ``` 1. suspicious_model.pkl (pos 28): [CRITICAL] Suspicious module reference found: posix.system Why: The 'os' module provides direct access to operating system functions. ``` ## Integration in Workflows ModelAudit is particularly useful in CI/CD pipelines when incorporated with Promptfoo: ```bash # Example CI/CD script segment npm install -g promptfoo pip install modelaudit promptfoo scan-model --format json --output scan-results.json ./models/ if [ $? -ne 0 ]; then echo "Security issues found in models! Check scan-results.json" exit 1 fi ``` ### Exit Codes ModelAudit returns specific exit codes for automation: - **0**: No security issues found ✅ - **1**: Security issues detected (warnings or critical) 🟡 - **2**: Operational errors or inconclusive scans occurred (installation, file access, timeouts, no scanned files, etc.) 🔴 :::tip CI/CD Best Practice In CI/CD pipelines, exit code 1 indicates findings that should be reviewed but don't necessarily block deployment. Only exit code 2 represents actual scan failures. ::: ## Requirements ModelAudit is included with Promptfoo, but specific model formats may require additional dependencies: ```bash # For TensorFlow models pip install tensorflow # For PyTorch models pip install torch # For Keras models with HDF5 pip install h5py # For YAML configuration scanning pip install pyyaml # For SafeTensors support pip install safetensors # For HuggingFace URL scanning pip install huggingface-hub # For cloud storage scanning pip install boto3 google-cloud-storage # For MLflow registry scanning pip install mlflow ``` ### NumPy Compatibility ModelAudit supports both NumPy 1.x and 2.x. If you encounter NumPy compatibility issues: ```bash # Force NumPy 1.x if needed for full compatibility pip install modelaudit[numpy1] ``` ## See Also - [Advanced Usage](./usage.md) - [Scanner Reference](./scanners.md)