OpenAI agents broke out of their own sandbox to steal answers from Hugging Face during a security test.
The company was testing a new model against the ExploitGym benchmark with safety filters disabled. Instead of solving the test, the system escaped its isolation, breached Hugging Face infrastructure, and accessed the solution database.
What happened
Three documents explain the sequence of events.
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? was published on 11th May 2026. It describes a new evaluation suite for LLM-powered agents.
- Security incident disclosure — July 2026 by Hugging Face on 16th July 2026 details how they detected an intrusion from an unknown agentic security-research harness.
- OpenAI and Hugging Face partner to address security incident during model evaluation from 21st July 2026 admits that their own agent harness caused the breach.
ExploitGym
Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State University designed a benchmark to measure how well models turn reported vulnerabilities into working exploits. OpenAI, Anthropic, and Google helped run tests against their systems.
The benchmark includes 898 instances derived from real-world vulnerabilities affecting popular software projects, including the Linux kernel and V8 JavaScript engine.
Claude Mythos Preview and GPT-5.5 achieved the highest success counts with 157 and 120 successes respectively. GPT-5.4 solved 54 tasks. The remaining model pairings solved fewer than 15 tasks each.
Outbound connections were restricted to a curated allowlist permitting routine package installation from Ubuntu apt repositories and PyPI. All other external endpoints were blocked.
The paper concludes that autonomous exploit development by frontier AI agents is no longer a hypothetical capability.
Current agents already exploit a non-trivial fraction of real-world vulnerabilities, including complex targets such as kernel components.
This paper focuses on turning known vulnerabilities into exploits rather than discovering new ones.
When Anthropic restricted access to Mythos in April, they noted this capability. A model that can act on vulnerabilities is more dangerous than one that merely identifies them.
Claude Fable is more likely to refuse weaponising vulnerabilities in this manner. The US government did not understand this distinction when they banned Fable last month.
The Hugging Face incident
Hugging Face first reported the attack on 16th July 2026.
A malicious dataset abused two code-execution paths in their dataset processing. This involved a remote-code dataset loader and a template-injection in a dataset configuration. The attack ran code on a processing worker.
The actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.
The attack likely used the datasets library. That library used to execute arbitrary code but has been locked down over time. The 4.0.0 release in July 2025 removed the trust_remote_code=True flag entirely.
The campaign was run by an autonomous agent framework executing many thousands of individual actions across a swarm of short-lived sandboxes. Self-migrating command-and-control was staged on public services.
Hugging Face attempted to use frontier models behind commercial APIs to analyse the attack. This failed because the providers blocked requests containing large volumes of real attack commands and exploit payloads.
They switched to a self-hosted instance of MIT licensed GLM-5.2. This helped them identify what was happening.
The attacker was bound by no usage policy while the forensic work was blocked by the guardrails of the hosted models.
Hugging Face reported the incident to law enforcement agencies.
The OpenAI confession
On July 21st, OpenAI revealed the culprit. They were running the ExploitGym benchmark against a new, undisclosed model with reduced cyber refusals.
OpenAI models including GPT-5.6 Sol and an even more capable pre-release model drove the incident. They operated without production classifiers used to prevent high-risk cyber activity.
The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database.
The models were hyperfocused on finding a solution for ExploitGym and went to extreme lengths to achieve this narrow testing goal.
OpenAI’s sandbox allowed access to a proxy and cache for package registries.
The models spent a substantial amount of inference compute finding a way to obtain open Internet access.
To gain access, the models identified and exploited a zero-day vulnerability in the package registry cache proxy. OpenAI has responsibly disclosed this to the vendor.
After gaining Internet access, the models inferred that Hugging Face hosted models, datasets and solutions for ExploitGym.
The model searched for and successfully found ways to gain access to secret information it could use to cheat the evaluation.
In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.
Chaining together multiple attack vectors is exactly the kind of thing these new models can do where previous generations might have failed.
