# Running PentAGI with vLLM and Qwen3.5-27B-FP8

This guide explains how to deploy PentAGI with a fully local LLM setup using vLLM and Qwen3.5-27B-FP8. This configuration enables complete independence from cloud API providers while maintaining high performance for autonomous penetration testing workflows.

## Table of Contents

- [Model Overview](#model-overview)
- [Hardware Requirements](#hardware-requirements)
- [Prerequisites](#prerequisites)
- [vLLM Installation](#vllm-installation)
- [Server Configuration](#server-configuration)
- [Testing the Deployment](#testing-the-deployment)
- [PentAGI Integration](#pentagi-integration)
- [Performance Benchmarks](#performance-benchmarks)
- [Troubleshooting](#troubleshooting)

---

## Model Overview

**Qwen3.5-27B** is a state-of-the-art dense language model from Alibaba Cloud with 27 billion parameters fully active on every token. It features a hybrid architecture combining:
- **75% Gated DeltaNet layers** (linear attention)
- **25% Gated Attention layers** (traditional attention)
- **Native context window**: 262,144 tokens
- **Extended context**: Up to 1,010,000 tokens via YaRN
- **Quantization**: FP8 W8A8 with block size 128 (performance nearly identical to BF16)

This model is particularly well-suited for PentAGI's multi-agent workflows due to its:
- Strong reasoning capabilities with native thinking mode
- Excellent function calling support
- Large context window for complex security analysis
- Fast inference speed with FP8 quantization

---

## Hardware Requirements

FP8 W8A8 hardware acceleration requires GPUs with **Compute Capability ≥ 8.9** (Ada Lovelace, Hopper, or Blackwell architectures). On older GPUs like Ampere (A100, A6000, RTX 3090), FP8 falls back to W8A16 mode via Marlin kernels with reduced performance.

### Supported GPU Configurations

| Configuration | Total VRAM | Max Context | FP8 Mode | Status |
|---|---|---|---|---|
| 2× RTX 5090 (64 GB) | 64 GB | ≤131k | W8A8 | Good |
| **4× RTX 5090 (128 GB)** | **128 GB** | **262k (native)** | **W8A8** | **Tested (~30 GB/GPU)** |
| 1× H100 SXM (80 GB) | 80 GB | 262k | W8A8 | Single GPU |
| 2× H100 SXM (160 GB) | 160 GB | 262k | W8A8 | Excellent |
| 4× A100 80GB (320 GB) | 320 GB | 262k | W8A16 | Slower fallback |

---

## Prerequisites

### System Requirements

- **OS**: Linux (Ubuntu 22.04+ recommended)
- **CUDA**: 12.1 or higher
- **Python**: 3.9 - 3.12
- **GPU Drivers**: Latest NVIDIA drivers (535+)
- **NCCL**: 2.27.3+ (for multi-GPU setups)

### Required Software

Install CUDA toolkit and verify installation:

```bash
nvidia-smi
nvcc --version
```

Install Python package manager (uv recommended for faster installation):

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

---

## vLLM Installation

### Install vLLM Nightly Build

**IMPORTANT**: The `qwen3_5` architecture is not recognized in stable vLLM releases. You **must** use the nightly build until vLLM v0.17.0 is released.

**Option 1: Using uv (recommended)**

```bash
uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
```

**Option 2: Using pip**

```bash
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```

**Option 3: Docker (alternative)**

```bash
docker pull vllm/vllm-openai:nightly
```

### Verify Installation

```bash
python -c "import vllm; print(vllm.__version__)"
```

---

## Server Configuration

### Recommended vLLM Parameters

The following configuration has been tested and optimized for 4× RTX 5090 GPUs with ~30 GB VRAM usage per GPU at `--gpu-memory-utilization 0.75`:

| Parameter | Value | Explanation |
|---|---|---|
| `--model` | `Qwen/Qwen3.5-27B-FP8` | HuggingFace model identifier |
| `--tensor-parallel-size` | `4` | Number of GPUs (1 shard per GPU) |
| `--max-model-len` | `262144` | Native context window size |
| `--max-num-batched-tokens` | `4096` | Optimal for low inter-token latency in chat |
| `--block-size` | `128` | Matches FP8 quantization block size |
| `--gpu-memory-utilization` | `0.75` | VRAM allocation ratio (adjust as needed) |
| `--language-model-only` | flag | Skip vision encoder → +2-4 GB KV-cache |
| `--enable-prefix-caching` | flag | Cache repeated system prompts |
| `--reasoning-parser` | `qwen3` | Enable Qwen3.5 reasoning/thinking mode parser |
| `--tool-call-parser` | `qwen3_xml` | Prevents infinite `!!!!` bug with long contexts |
| `--attention-backend` | `FLASHINFER` | Best for Ada/Hopper/Blackwell GPUs |
| `--speculative-config` | `'{"method":"qwen3_next_mtp","num_speculative_tokens":1}'` | Enable Medusa-based speculative decoding (MTP) |
| `-O3` | flag | Maximum optimization via torch.compile |

### Start vLLM Server

**For Single GPU (H200, B200, B300):**

```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 4096 \
  --block-size 128 \
  --gpu-memory-utilization 0.75 \
  --language-model-only \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' \
  -O3 \
  --host 127.0.0.1 \
  --port 8000
```

**For Multi-GPU (4× RTX 5090):**

```bash
NCCL_P2P_DISABLE=1 vllm serve Qwen/Qwen3.5-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-batched-tokens 4096 \
  --block-size 128 \
  --gpu-memory-utilization 0.75 \
  --language-model-only \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' \
  -O3 \
  --host 127.0.0.1 \
  --port 8000
```

**Multi-GPU Note**: The `NCCL_P2P_DISABLE=1` environment variable is **required** for Blackwell GPUs (RTX 5090) with tensor parallelism > 1 to prevent NCCL hangs. Update `nvidia-nccl-cu12` to version 2.27.3+ for additional stability.

### Optional: Disable Thinking Mode by Default

To disable the thinking mode at the server level (can still be enabled per-request):

```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  # ... other parameters
```

### Important: Multi-Turn Conversations

**Best Practice**: In multi-turn conversations, the historical model output should **only include the final output** and **not the thinking content** (`<think>...</think>` tags). This is automatically handled by vLLM's Jinja2 chat template, but if you're implementing custom conversation handling, ensure thinking tags are stripped from message history.

---

## Testing the Deployment

After starting the vLLM server, verify it's working correctly with these test requests.

### Test 1: Thinking Mode Enabled (Default)

```bash
curl "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-FP8",
    "messages": [{"role": "user", "content": "hey! what is the weather in Moscow?"}],
    "temperature": 1.0,
    "top_k": 20,
    "top_p": 0.95,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0
  }'
```

**Expected**: Response includes `<think>` tags with reasoning process.

### Test 2: Thinking Mode Disabled (Non-Thinking)

```bash
curl "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-FP8",
    "messages": [{"role": "user", "content": "hey! what is the weather in Beijing?"}],
    "temperature": 0.7,
    "top_k": 20,
    "top_p": 0.8,
    "min_p": 0.0,
    "presence_penalty": 1.5,
    "repetition_penalty": 1.0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

**Expected**: Direct response without `<think>` tags.

### Test 3: Higher Temperature Reasoning

```bash
curl "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-FP8",
    "messages": [{"role": "user", "content": "hey! what is the weather in New York?"}],
    "temperature": 1.0,
    "top_k": 40,
    "top_p": 1.0,
    "min_p": 0.0,
    "presence_penalty": 2.0,
    "repetition_penalty": 1.0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

**Expected**: Creative/diverse responses without thinking tags.

If all tests return valid JSON responses with appropriate content, your vLLM server is ready for PentAGI integration.

---

## Recommended Sampling Parameters

The Qwen team provides official recommendations for sampling parameters optimized for different use cases:

| Mode | temp | top_p | top_k | presence_penalty |
|---|---|---|---|---|
| **Thinking, general tasks** | 1.0 | 0.95 | 20 | 1.5 |
| **Thinking, coding (WebDev)** | 0.6 | 0.95 | 20 | 0.0 |
| **Non-thinking (Instruct), general** | 0.7 | 0.8 | 20 | 1.5 |
| **Non-thinking (Instruct), reasoning** | 1.0 | 1.0 | 40 | 2.0 |

**Additional parameters:**
- `repetition_penalty=1.0` for all modes
- `max_tokens=32768` for most tasks
- `max_tokens=81920` for complex math/coding tasks

These parameters are already applied in the PentAGI provider configuration files referenced below.

---

## PentAGI Integration

### Step 1: Configure Custom Provider in PentAGI

PentAGI includes pre-configured provider files for Qwen3.5-27B-FP8 with optimized sampling parameters for different agent roles.

**Two provider configurations are available:**

1. **With Thinking Mode** (default): [`examples/configs/vllm-qwen3.5-27b-fp8.provider.yml`](../configs/vllm-qwen3.5-27b-fp8.provider.yml)
   - Enables `<think>` tags for primary agents (primary_agent, assistant, adviser, refiner, generator)
   - Uses `temp=0.6` for coding agents (coder, installer, pentester)
   - Recommended for maximum reasoning quality

2. **Without Thinking Mode**: [`examples/configs/vllm-qwen3.5-27b-fp8-no-think.provider.yml`](../configs/vllm-qwen3.5-27b-fp8-no-think.provider.yml)
   - Disables thinking for all agents via `chat_template_kwargs`
   - Uses `temp=0.7` for general tasks, `temp=1.0` for reasoning
   - Recommended for faster responses

### Step 2: Add Provider via PentAGI UI

1. Start PentAGI (see [Quick Start](../../README.md#-quick-start))
2. Navigate to **Settings → Providers**
3. Click **Add Provider**
4. Fill in the form:
   - **Name**: `vLLM Qwen3.5-27B-FP8` (or any custom name)
   - **Type**: `Custom`
   - **Base URL**: `http://127.0.0.1:8000/v1` (or your vLLM server address)
   - **API Key**: `dummy` (vLLM doesn't require authentication by default)
   - **Configuration**: Copy contents from one of the YAML files above
5. Click **Save**

### Step 3: Verify Provider Configuration

Test the provider by creating a simple flow:

1. Navigate to **Flows**
2. Click **New Flow**
3. Select your newly created provider
4. Enter a test task: `"Scan localhost port 80"`
5. Monitor execution logs

---

## Performance Benchmarks

Based on internal testing with 4× RTX 5090 GPUs and 10 concurrent requests:

| Metric | Value |
|---|---|
| **Prompt Processing Speed** | ~13,000 tokens/sec |
| **Completion Generation Speed** | ~650 tokens/sec |
| **Concurrent Flows** | 12 flows simultaneously with stable performance |
| **VRAM Usage** | ~30 GB per GPU (at 0.75 utilization) |
| **Context Window** | Full 262K tokens supported |

These benchmarks demonstrate that Qwen3.5-27B-FP8 provides excellent throughput for running multiple PentAGI flows in parallel, making it suitable for production deployments.

---

## Troubleshooting

### Issue: Automation Flow Runs for Hours or Repeats Commands

**Cause**: Broad penetration testing prompts can leave smaller local or custom models exploring too much state, especially when the task has no stopping criteria. Repeated commands may indicate model weakness, target complexity, tool-call/provider issues, or a flow that needs additional supervision.

**Solution**: First check the existing PentAGI execution controls before changing runtime behavior or assuming the target is broken:

- Enable execution monitoring with `EXECUTION_MONITOR_ENABLED=true` so the Adviser can review repeated or inefficient tool-call patterns.
- Tune `EXECUTION_MONITOR_SAME_TOOL_LIMIT` and `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT` if Adviser reviews happen too late or too often for your model and target.
- Enable planning with `AGENT_PLANNING_STEP_ENABLED=true` for complex pentest flows so specialist agents receive a bounded execution plan.
- Review hard tool-call limits with `MAX_GENERAL_AGENT_TOOL_CALLS` and `MAX_LIMITED_AGENT_TOOL_CALLS`; these remain the final guardrails for runaway executions.

Also narrow the task prompt. Instead of only `Perform penetration testing on the host 192.168.136.136`, include the authorized target, scope, expected output, and stopping criteria, for example: enumerate exposed services, try likely public exploits, avoid repeating failed commands more than twice, and stop with a concise findings report if no path is found.

Use PentAGI flow logs, Docker logs, and provider or vLLM logs together when diagnosing a long run:

- Repeated identical shell/browser actions point toward tool-loop behavior.
- Long gaps between actions point toward slow model generation or provider latency.
- Frequent malformed tool calls point toward provider/tool-call parser compatibility.
- Varied but slow exploration can simply mean the target is complex or the prompt is too broad.

Qwen3.5-27B-FP8 can run useful local flows, but complex autonomous pentests may still need execution monitoring, task planning, tighter prompts, and careful log review.

### Issue: "Unknown architecture 'qwen3_5'"

**Cause**: Using stable vLLM release instead of nightly.

**Solution**: Install vLLM nightly build:

```bash
uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
```

### Issue: NCCL Hangs on Multi-GPU Setup

**Cause**: Blackwell GPUs (RTX 5090) require P2P communication to be disabled when using tensor parallelism.

**Solution**: Set environment variable before starting vLLM:

```bash
export NCCL_P2P_DISABLE=1
```

Also update NCCL library:

```bash
pip install --upgrade nvidia-nccl-cu12
```

### Issue: `enable_thinking` Parameter Ignored

**Cause**: Parameter must be passed inside `chat_template_kwargs`, not at root level.

**Solution**: Use correct JSON structure:

```json
{
  "messages": [...],
  "chat_template_kwargs": {"enable_thinking": false}
}
```

### Issue: Infinite `!!!!` Generation on Long Contexts

**Cause**: Using `qwen3_coder` parser with long contexts triggers a known bug.

**Solution**: Switch to XML parser:

```bash
--tool-call-parser qwen3_xml
```

### Issue: Out of Memory (OOM)

**Cause**: Insufficient VRAM for chosen context length.

**Solution**: Reduce `--max-model-len` or `--gpu-memory-utilization`:

```bash
# Reduce context window
--max-model-len 131072

# Or reduce VRAM allocation
--gpu-memory-utilization 0.7
```

### Issue: Speculative Decoding Errors

**Cause**: `num_speculative_tokens > 1` is unstable in current nightly builds.

**Solution**: Use only 1 speculative token:

```bash
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}'
```

---

## Advanced: Extended Context with YaRN

Qwen3.5-27B natively supports 262K tokens. For tasks requiring longer context (up to 1,010,000 tokens), you can enable YaRN (Yet another RoPE extensioN) scaling.

### Enable YaRN via Command Line

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.5-27B-FP8 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
  --max-model-len 1010000 \
  # ... other parameters
```

**Important Notes:**
- YaRN uses a **static scaling factor** regardless of input length, which may impact performance on shorter texts
- Only enable YaRN when processing long contexts is required
- Adjust `factor` based on typical context length (e.g., `factor=2.0` for 524K tokens)
- For most PentAGI workflows, the native 262K context is sufficient

---

## Additional Resources

- **Official Qwen3.5 Documentation**: [HuggingFace Model Card](https://huggingface.co/Qwen/Qwen3.5-27B-FP8)
- **vLLM Documentation**: [docs.vllm.ai](https://docs.vllm.ai/)
- **vLLM Qwen3.5 Recipe**: [Official vLLM Guide](https://docs.vllm.ai/en/latest/models/supported_models/)
- **PentAGI Main Documentation**: [README.md](../../README.md)
- **Provider Configuration Reference**: See example configs in [`examples/configs/`](../configs/)