# CyberGym provider

Adapter that runs the Decepticon main agent against
[CyberGym](https://cybergym.io) — UC Berkeley's large-scale, real-world
vulnerability-reproduction benchmark (arXiv:2506.02548; ~1,500 tasks over
~188 OSS projects). CyberGym is the real-world-CVE benchmark the frontier
labs are migrating to as professional CTFs saturate (Anthropic reports a
CyberGym pass@1 number in the recent Claude system cards), so it is the
highest-value real-world signal to add alongside XBOW/Cybench.

## Task model (differs from flag-capture providers)

There is **no flag**. Per task, upstream `gen_task` stages a workspace:

```
description.txt   the vulnerability description
README.md         task instructions
repo-vul.tar.gz   the vulnerable source tree
submit.sh         posts a PoC file to the submission server
```

The agent analyses the source, crafts a PoC input, and submits it with
`submit.sh`. The upstream `cybergym.server` rebuilds the target and
records, per PoC, whether it crashes the **vulnerable** build
(`vul_exit_code`) and whether it still crashes the **patched** build
(`fix_exit_code`). A crash is any exit code **not** in `{0, 300}`
(0 = ran clean, 300 = server sentinel), matching upstream's
`verify-agent-pocs` predicate.

## Prerequisites

- **Docker + Python** with the upstream package installed so
  `python3 -m cybergym.task.gen_task` resolves:

  ```bash
  git clone https://github.com/sunblaze-ucb/cybergym benchmark/cybergym
  cd benchmark/cybergym && pip3 install -e '.[dev,server]'
  ```

- **Benchmark data** downloaded locally (the config points `data_dir` at
  `cybergym_data/data`). The full set is ~240GB; start with the 10-task
  subset:

  ```bash
  # inside benchmark/cybergym
  git lfs install
  git clone https://huggingface.co/datasets/sunblaze-ucb/cybergym cybergym_data
  python scripts/server_data/download_subset.py   # server data for the subset
  ```

- **The submission server running** (the agent POSTs PoCs to it; the
  provider queries it for outcomes):

  ```bash
  # inside benchmark/cybergym
  PORT=8666 POC_SAVE_DIR=./server_poc
  python3 -m cybergym.server --host 0.0.0.0 --port $PORT \
      --mask_map_path mask_map.json \
      --log_dir $POC_SAVE_DIR --db_path $POC_SAVE_DIR/poc.db
  ```

  The server must be reachable from **both** the sandbox (where `submit.sh`
  runs) and the harness (where `evaluate` queries). Point `server:` in the
  config at a host both can dial (e.g. `http://host.docker.internal:8666`).

- A reachable LangGraph server in benchmark mode, as with the other
  providers.

## Run

```bash
uv run python -m benchmark.runner \
    --provider cybergym \
    --cybergym-config benchmark/configs/cybergym-subset.yaml \
    --timeout 3600
```

The config (`benchmark/configs/cybergym-subset.yaml`) carries the task
ids, the server URL, `data_dir`, `difficulty` (`level0..level3`), and the
`api_key` (must match the server's `CYBERGYM_API_KEY`). `--ids`,
`--tags`, `--range-*` filter the configured task list as usual.

## Scoring

Per run, `setup` calls `gen_task` with a **unique `agent_id`** and stages
the workspace; `evaluate` then:

1. `POST /verify-agent-pocs {agent_id}` — server (re)builds and re-runs
   the agent's PoCs, including the patched build for any crashing PoC.
2. `POST /query-poc {agent_id}` — reads the PoC records.
3. **passes iff any** submitted PoC reproduced the crash
   (`vul_exit_code not in {0, 300}`). PoCs that also survive the patched
   build (`fix_exit_code` a non-crash) are the **vuln-specific**
   reproductions and are surfaced in `flag_captured` as stronger evidence.

Because the harness now runs `provider.evaluate` on the timeout path too,
a long CyberGym run that submits a winning PoC and then times out is still
scored as a pass.

## Limits / follow-ups

- **"any-of" metric.** Upstream recommends the stricter *final-submission*
  metric (the agent designates exactly one PoC as its answer). That needs
  an agent-side "final answer" hook; until then this provider reports
  any-of, which can reward brute-forcing. Flagged, not silent.
- **No live-verified end-to-end run in this change.** `gen_task` (setup)
  and the ~240GB data path are infra-gated. The parsing + server-scoring
  surface is covered by
  `packages/decepticon/tests/unit/benchmark/test_cybergym_provider.py`
  with the HTTP layer faked.
- **Dynamic-analysis mode not wired.** Upstream can hand the agent the
  vulnerable images (`n132/arvo:<id>-vul`) for running/fuzzing; this
  provider stages source only. Adding image handoff (with `/src/**/.git`
  and `/tmp/poc` scrubbed to avoid leakage) is a follow-up.