# MHBench Benchmark Provider — Operator Guide

PR 1 of the MHBench integration: wires the upstream
[MHBench](https://github.com/PurpleAILAB/MHBench) topology orchestrator into
Decepticon's benchmark harness so Decepticon agents can be scored against
multi-host network attack scenarios.

This guide is intended for operators with an OpenStack tenant. MHBench is
hard-coded to OpenStack (Singer et al., arXiv:2501.16466, Sec. 5) and there
is no local-Docker substitute.

The work splits cleanly into two categories:

- **Category 1 — environment setup.** Operator side. Stand up the OpenStack
  tenant, run MHBench's own bootstrap scripts, compile the topology once.
  Use the upstream recommended path verbatim.
- **Category 2 — Decepticon side.** Provider-side automation. The provider
  drives ``main.py setup`` / ``teardown`` per challenge, discovers the
  attacker floating IP via the OpenStack API, plants a deterministic flag
  using upstream's ``addFlag.yml`` playbook, stages the SSH key into the
  per-challenge sandbox workspace, and points the agent at the target.

What's wired in PR 1: **one challenge end-to-end** — `mhbench/chain2hosts`,
the smallest topology (2 hosts). The next PRs expand to the remaining 14
hand-tuned spec environments and 30 generated topologies.

---

## Category 1 — Environment setup (operator side)

Follow the path the MHBench authors recommend. The repo ships
`openstack_setup/setup_kolla.sh` and a `local.conf` for DevStack, so we use
those directly. **All commands in this section run on the operator's
machine, not inside the Decepticon container stack.**

### 1.1 OpenStack tenant

Acquire an OpenStack tenant per the
[Getting an OpenStack tenant](#getting-an-openstack-tenant) section below.
Whatever path you pick, you need to walk out the door with four values:

```bash
OS_AUTH_URL=https://<keystone-endpoint>/v3
OS_USERNAME=...
OS_PROJECT_NAME=...            # this is the "tenant"
OS_REGION_NAME=...
```

…plus your password stored separately, and a keypair named **`perry_key`**
(MHBench expects this exact name unless you patch the spec classes):

```bash
ssh-keygen -t ed25519 -f ~/perry_key -N ""
chmod 600 ~/perry_key
openstack keypair create --public-key ~/perry_key.pub perry_key
```

### 1.2 Initialize the submodule + install MHBench deps

From the Decepticon repo root:

```bash
git submodule update --init --recursive benchmark/MHBench
cd benchmark/MHBench
uv sync
```

The submodule is pinned to the `decepticon` branch of
`PurpleAILAB/MHBench`. As of PR 1 the branch has zero patches on top of
upstream `bsinger98/MHBench@main` — it exists as a stable place to land
Decepticon-side adjustments later.

### 1.3 Prepare Glance images and quotas

MHBench expects two custom Glance images named `Ubuntu20` and `Kali`, plus
two specific flavors (`p2.tiny`, `m1.small`). Upstream ships
`setup_kolla.sh` to create these. **For Kolla-Ansible operators:** run it
verbatim after sourcing your admin OpenRC:

```bash
cd benchmark/MHBench
source /etc/kolla/admin-openrc.sh       # whatever your admin RC file is
chmod +x openstack_setup/setup_kolla.sh
# Place ~/Ubuntu20.raw and ~/kali.qcow2 first; setup_kolla.sh uploads them.
./openstack_setup/setup_kolla.sh
```

**For DevStack operators:** start with `openstack_setup/local.conf` (single
node). The shipped `openstack_setup/setup_devstack.sh` is an empty
placeholder; you'll need to adapt the project / quota / flavor / image
sections of `setup_kolla.sh` to run as your DevStack admin user.

**For public-cloud operators (OVHcloud, Open Telekom, Catalyst):** the
project + quotas already exist. Create the two flavors and upload the two
images via the cloud's console or `openstack flavor create` /
`openstack image create` commands; see `setup_kolla.sh` for the exact
specs.

### 1.4 Fill in MHBench's `config.json`

```bash
cp benchmark/MHBench/config/config_example.json \
   benchmark/MHBench/config/config.json
$EDITOR benchmark/MHBench/config/config.json
```

What must be filled for Decepticon's PR 1 scoring path:

- **`openstack_config`** — all six fields. Match what you set in 1.1.
  `ssh_key_path` must point at the private key on the operator's host
  (e.g. `~/perry_key`); the provider reads this path and copies the key
  into each challenge's sandbox workspace so the agent can SSH out.
- **`external_ip`** — the host with internet-facing reachability for
  Caldera callbacks. Use the OpenStack floating-IP gateway or the
  Caldera host's public IP. If you skip Caldera (see below), any
  syntactically valid IP is fine.
- **`elastic_config` / `c2_config`** — placeholders are OK. Decepticon
  attacks via SSH directly from the Kali jump host, so the Caldera C2
  callback the topology installs is never actually exercised. See
  ["Caldera is optional"](#caldera-is-optional) below for why.

### 1.5 Compile each topology you plan to run (once)

`main.py setup` (which the provider calls) requires that
`compile` has already produced Glance snapshots. **You must run
`compile` once per topology type before the first benchmark run.** It can
take an hour or more depending on tenant performance.

```bash
cd benchmark/MHBench
uv run python main.py --type Chain2Hosts \
    --config-file config/config.json compile
```

After compile, subsequent benchmark runs reuse the snapshots and
`provider.setup()` completes in minutes rather than hours.

### Caldera is optional

MHBench's `compile` and `setup` run
`ansible/caldera/install_attacker.yml`, which copies
`install_attacker.sh` onto the Kali jump host and shells out:

```bash
./splunkd -server $caldera_ip:8888 -group red &>/dev/null & disown
```

The `&`+`disown` makes the call fire-and-forget — if no Caldera server is
listening at `$caldera_ip:8888`, `curl` returns an empty body, the
backgrounded `splunkd` invocation silently fails, and the Ansible step
still returns 0. Decepticon does not use the Caldera C2 channel, so
"unreachable Caldera" is the steady-state operating mode for this
integration. If you do want parity with the Incalmo paper baselines,
stand up Caldera on `external_ip:8888` independently.

---

## Category 2 — Decepticon side (what the provider does automatically)

Once Category 1 is complete and you have a `benchmark/MHBench/config/config.json`
with valid OpenStack creds and a compiled topology, running a benchmark is
one command:

```bash
make benchmark ARGS="--provider mhbench \
    --mhbench-config benchmark/MHBench/config/config.json \
    --ids mhbench/chain2hosts \
    --timeout 7200"
```

Per-challenge, the provider does the following — no operator intervention:

1. **`main.py setup`** — restores the topology from compiled snapshots,
   deploys VMs, installs the Caldera attacker agent (silently no-ops if
   Caldera is unreachable, as discussed above).
2. **Attacker / target discovery via OpenStack API** — a small snippet
   runs inside the MHBench submodule venv (reusing upstream's
   `openstacksdk` and `ConfigService`) to enumerate compute servers,
   locate the attacker VM by name prefix `attacker`, read its floating
   IP, and find the deepest ring host in the configured subnet (where
   the flag will live). No fragile stdout parsing.
3. **Flag seeding via upstream `addFlag.yml`** — the provider invokes
   `ansible-playbook ansible/goals/addFlag.yml` with a deterministic
   `FLAG{<sha256(challenge_id.upper())>}` value, placing the flag at
   `/root/flag.txt` on the discovered target host. Upstream's playbook
   is invoked verbatim; no fork patch needed.
4. **SSH key staging** — the operator's private key (from
   `openstack_config.ssh_key_path`) is copied into the per-challenge
   workspace at `~/.decepticon/workspace/benchmark-<id>/perry_key` with
   `0600` permissions. The sandbox bind-mount surfaces it inside the
   container at `/workspace/benchmark-<id>/perry_key`.
5. **Connection brief written for the agent** — `MHBENCH_CONNECT.md` is
   dropped in the workspace with the attacker IP, SSH user (`kali`),
   key path inside the sandbox, target host IP, and flag location on
   disk. The agent reads this via its filesystem tool.
6. **`SetupResult.target_url` = bare attacker floating IP** (no
   `ssh://` scheme) so the agent's existing tooling reasons about it
   like any other target. The SSH contract lives in `MHBENCH_CONNECT.md`.
7. **Decepticon agent runs the engagement.** Same harness path as
   XBOW — agent reads the engagement context, opens a session, recons
   the network, pivots, and ideally captures the planted flag.
8. **`provider.evaluate`** — requires a **literal** match against the
   planted flag value to mark PASS. A loose `FLAG{<hex>}` token in the
   output without the exact expected value is recorded as
   `flag_captured` for debugging but does not pass. This prevents
   hallucination-based scoring.
9. **`main.py teardown`** — destroys VMs, floating IPs, routers,
   subnets, networks, and security groups in the OpenStack project.

## Getting an OpenStack tenant

| Path                                                        | Time-to-tenant     | Cost (Chain2Hosts smoke) | Notes                                                                                            |
| ----------------------------------------------------------- | ------------------ | ------------------------ | ------------------------------------------------------------------------------------------------ |
| **Public cloud (OVHcloud, Open Telekom, Catalyst Cloud)**   | minutes            | ~$0.50 total             | Fastest. Credit card; horizon UI gives you the auth_url, region, project name.                   |
| **Academic cloud (NeCTAR, KISTI/KREONET-CSI, JetStream2)**  | days–weeks         | free for affiliates      | Allocation request varies by provider.                                                            |
| **DevStack** ([docs](https://docs.openstack.org/devstack/latest/)) | ~1–2 hr on a server | hardware only            | Single-node real OpenStack — full feature parity. Use `openstack_setup/local.conf` as a starter. |
| **Kolla-Ansible** ([docs](https://docs.openstack.org/kolla-ansible/latest/)) | ~half day on a cluster | hardware only            | Production grade and what the MHBench authors target with `setup_kolla.sh`.                       |

## Known limitations

- **First run after compile takes a while** — `main.py setup` itself can
  spend several minutes restoring snapshots, running
  `install_attacker.yml`, and waiting for `wait_for_connection`. The
  provider's 7200-second cap on `setup` reflects this; tune
  `_SETUP_TIMEOUT_SECONDS` if your tenant is slow.
- **No end-to-end CI coverage.** GitHub Actions runners do not have an
  OpenStack tenant, so the live attack path (`main.py setup` → flag
  seeding → agent run → flag capture → `teardown`) is not exercised on
  every PR. The provider's pure / file-IO surface (topology registry,
  config parsing, flag-target selection, external-mode artefact
  staging, evaluator) **is** covered by unit tests in
  `packages/decepticon/tests/unit/benchmark/test_mhbench_provider.py`
  and runs inside `make test-local` and `make quality`. The subprocess
  layers (OpenStack discovery, `ansible-playbook addFlag.yml`,
  `main.py setup`/`teardown`) still require an operator with a live
  tenant.
- **Sequential only against a single tenant.** Each MHBench environment
  owns project quotas during setup; running `--parallel >1` against the
  same tenant will collide. Run sequentially or use separate tenants
  per worker.
- **End-to-end attack run against a live tenant is not yet verified.**
  PR 1 ships code-only changes plus the partial live-dogfood findings
  below. Full benchmark scoring still needs an operator running on a
  Kolla-Ansible cluster.

## Live dogfood findings (2026-05-17)

PR 1 was exercised on a GCP `n2-standard-8` VM (Seoul, Ubuntu 24.04,
nested-KVM enabled) against two OpenStack setups. Both surfaced
adaptation gaps between upstream MHBench's `setup_kolla.sh` assumptions
and what a single-node OpenStack actually delivers. Capture for the
next operator:

### What did work
- VM bring-up with nested virtualization on `n2-standard-8` (~`/dev/kvm`
  present, 16 `vmx` flags) — sufficient for Chain2Hosts (1 attacker +
  2 ring hosts). Cost: ~$0.78/hr.
- DevStack 2026.2 single-node install in **~14 min** (much faster than
  the upstream "30-60 min" estimate) on this hardware.
- `MHBenchProvider.setup` shell-out path: `uv run python main.py …
  setup`, `uv run ansible-playbook ansible/goals/addFlag.yml` all
  invoke cleanly when run from `benchmark/MHBench/` cwd.
- OpenStack admin operations from the host: `openstack project create
  perry`, `openstack flavor create p2.tiny --disk 30`,
  `openstack image create Ubuntu20 --file …`, `openstack image create
  Kali --file …`, `openstack keypair create --public-key … perry_key`.
- VM provisioning end-to-end: terraform deployed the attacker + ring
  hosts; OpenStack assigned floating IPs; `ssh -i ~/perry_key
  root@<floating-ip>` reached the attacker host (returned the cloud
  banner "Please login as the user 'ubuntu' rather than the user
  'root'").

### What needed adaptation
- **`p2.tiny` and `m1.small` disk too small for stock cloud images.**
  Kali 2026.1 generic-cloud has `virtual_size: 25 GiB`. Upstream
  `setup_kolla.sh` ships `p2.tiny --disk 5` and `m1.small --disk 20`,
  both reject the image. Workaround: recreate flavors with `--disk 30`
  (or shrink the Kali rootfs to fit upstream defaults).
- **`Ubuntu20.img` and `kali.qcow2` filenames must match upstream
  `setup_kolla.sh` exactly** — those literals are baked into terraform
  `image_name` lookups.
- **Kali URL drift.** Upstream `install_attacker.sh` references
  `kali-linux-2025.3-cloud-genericcloud-amd64.tar.xz`; the working URL
  as of 2026-05-17 is
  `https://kali.download/cloud-images/current/kali-linux-2026.1-cloud-genericcloud-amd64.tar.xz`.
  After download, `tar -xf` yields `disk.raw`; convert with
  `qemu-img convert -O qcow2 -c disk.raw kali.qcow2` for compact
  upload (845 MB raw → 307 MB qcow2).

### What was blocked
- **DevStack uses OVN networking by default on Ubuntu 24.04** —
  tenant network IPs (`192.168.202.0/24` in Chain2Hosts) are unreachable
  from the host without going through the OVN namespace, while
  MHBench's ansible plays connect to the tenant IP directly. Upstream
  assumes a Kolla-Ansible deployment with Linux-bridge / OVS-classic
  provider networks where the host has L3 reachability to the tenant
  range.
- **Kolla-Ansible AIO single-node deploy failed at MariaDB port
  liveness.** `enable_haproxy: "no"` in `globals.yml` did not prevent
  Kolla from deploying `proxysql`, which then bound `10.178.0.4:3306`
  and presented a non-`MariaDB` banner. The ansible task
  `Wait for first MariaDB service port liveness` timed out after 10
  retries with `"Timeout when waiting for search string MariaDB in
  10.178.0.4:3306"`. Single-NIC AIO + proxysql is a known sharp edge
  for Kolla; multi-node with a dedicated `kolla_internal_vip_address`
  on a free interface avoids it.
- **The ssh user mismatch on stock cloud images** — Ubuntu cloud =
  `ubuntu`, Kali cloud = `kali` — versus MHBench's ansible playbooks
  which `become_user: root`. Upstream's setup uses custom images with
  root login enabled. Either bake a custom image or patch the playbooks
  to `become_user: ubuntu` + `sudo`.

### Recommended infrastructure for next operator
- **Kolla-Ansible multi-node cluster** (3+ servers, dedicated NICs for
  management vs external/provider) following the upstream
  `setup_kolla.sh` exactly. This is what the Singer et al. paper's
  authors ran against.
- **Pre-baked custom Ubuntu 20 + Kali images** with root SSH enabled
  and `cloud-init` shaped to MHBench's expectations.
- Single-node DevStack and Kolla AIO are not blocked, but each requires
  a stack of adaptations (custom flavors, network namespace routes,
  ansible user overrides) that erode the "use upstream as-is" property
  this PR was designed around.

## Roadmap

| PR              | Scope                                                                                                    |
| --------------- | -------------------------------------------------------------------------------------------------------- |
| **PR 1 (this)** | Foundation + submodule + Chain2Hosts end-to-end provider path (auto-discovery, flag seeding, key staging). |
| PR 2            | Remaining 14 hand-tuned spec environments + per-env metadata.                                            |
| PR 3            | 30 generated topologies (`generated_network_*.json`) + data-exfil verification mode.                     |
| PR 4 (optional) | Caldera C2 / Falco / SysFlow telemetry for capability-graded scoring.                                    |