Open-source models now match proprietary systems in simulated cyberattacks
Vals AI has found that open-weight models are closing the performance gap with restricted systems in simulated cyberattacks.
The CyberOffense benchmark tests this by simulating realistic corporate networks. Each agent operates alone on an isolated machine, starting with only a public web address and no human assistance.
GPT-6 Sol, part of OpenAI’s restricted Daybreak cyber program, achieved the highest score at 80.8 percent of objectives. Deepseek V4.1 Flash 1017, a freely available open-weight model, reached 72.8 percent. The cost for a single run of Deepseek is about $3, compared to roughly $22 for the proprietary model. Once a model’s weights are public, researchers say no provider can control or restrict how it is used.
In the harder Lighthouse test, Daybreak took over admin control of the first company, stole passwords, and used them to break into a second company. It completed 27 of 31 objectives in under three hours. Deepseek managed 21 objectives at a tenth of the cost and was not limited solely by safety guardrails.
Longer runtimes proved critical for performance. Deepseek scored zero results in its first 25 million tokens and made most of its progress only after 100 million tokens. Short tests therefore underestimate the real potential of open models.
The following table shows results from the Nova and Lighthouse tests.
Performance comparison
| Model | Nova (of 21) | Lighthouse (out of 31) |
|---|---|---|
| GPT-6 Sol Daybreak | 15.7 (average of 3, range 12–19) | 27 |
| Deepseek V4.1 Flash 1017 | 16.3 (average of 3, range 15–18) | 21 |
| Kimi K3.1 Pro | 13 | 20 |
| Meta Wonder | 7 | 28 |
| Grok 4.7 | 12 | 16 |
| Kimi K3 | 9 | 10 |
| GLM 5.3 Flash | 5 | 5 |
Vals notes that these findings create an opportunity for defenders, because the very models that exploit vulnerabilities can help companies find and fix them first. Attacks on real networks are likely to increase in the coming months, according to Vals.
Earlier studies confirm the trend
The results align with a recent analysis by the UK’s AI Security Institute (AISI) from July. The gap between open and closed frontier models in cyber capabilities has shrunk to four to seven months. For most of 2025, that gap stood at six to ten months.
Open models are also much cheaper. A cyber-range run with Deepseek V4-Pro costs just $1.19 according to AISI, compared with $85 for Anthropic’s Opus 4.5. The safety measures on the open models proved largely ineffective.
A recent study by Anthropic also showed that the freely available model GLM-5-3 can build exploits at nearly the same level as the internal top model Mythos.
