CWE-bench v2 releases 100 public tasks to test frontier cyber capabilities
Collinear AI has released CWE-bench v2, a public set of 100 tasks designed to test the limits of frontier cyber capabilities. The setup remains unchanged from the previous version: an agent receives access to a real open-source repository and a short instruction, with the goal of finding and fixing a security weakness without breaking existing functionality. Scoring is fully programmatic, ensuring results are deterministic and reproducible without extra cost. All programmatic checks have been curated with the help of LLM judge panels to ensure hardness and fairness. The team maintains a held-out set to track progress on tasks no model has seen. To evaluate your model on this held-out set, contact cy***@*******ar.ai.
The objective is exhaustiveness. A defender must anticipate the ways an attacker could gain entry and succeed every single time. These 100 tasks cover 91 different CWEs, spanning a wide range of vulnerability classes. A model that performs well here must understand the multiple weakness surfaces a potential attacker can exploit. Recognising a familiar pattern is not enough.
To maintain quality, the team checked that specific conditions were true for every task:
- The instruction reads like a realistic request and not an evaluation setting.
- Every graded behaviour can be inferred from the instruction.
- The verifier checks for correct behaviour and not for the specific structure of one fix.
- The starter code fails and the reference fix passes.
- The original functionality still works with the new fix.
- The verifiers fail memorised solutions like the known upstream patch.
Every task was also reviewed by hand. Open-sourcing the set is part of the same effort to improve quality. The team welcomes feedback on any task and contributions to the public set.
Results
Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open-weight models and updates on its own: labs and the community add a result by opening a PR on the model’s repo with a results file. For the full comparison, including API-only models such as Gemini, Claude and GPT, see the CWE-bench leaderboard Space, where new models are added through a PR on the Space.
Open-weight models are listed on the dataset page. All evaluated models, including API-only ones, are listed in the leaderboard Space.
Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). 95 tasks are solved by at least one model, an indication that the tasks are indeed solvable. However, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. The public set, the grader and the harness are open. Run your model, and tell us where it stands.
On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.
Run it from the Hub
Every task is a self-contained Harbor task (a Dockerised environment, an instruction, and a deterministic verifier), and the dataset ships a registry.json file. Harbor can therefore run the whole benchmark straight from the Hugging Face repo, without cloning anything yourself:
harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 \
--agent <agent> --model <model> --env docker \
--allow-agent-host <model API host>The -k 4 flag runs 4 attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated: accept its access conditions on the dataset page first. Harbor fetches the tasks with git, so also save your HF token to git’s credential helper once:
hf auth login --token $HF_TOKEN --add-to-git-credentialThe tasks are also tagged rl-environment: the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them.
