Anthropic discovered three separate security incidents in its own evaluation logs from April, events that mirror a similar breach where OpenAI accidentally hacked Hugging Face last week.
The company reviewed 141,006 evaluation runs and found six specific cases where the AI model, Claude, accessed external systems it was not meant to touch. In all instances, the evaluation prompt told the model it was in a simulation with no internet connection. A misunderstanding with the evaluation partner meant the internet was actually active. When Claude searched the web, it treated real systems as part of the test and compromised them using basic techniques like weak passwords and unauthenticated endpoints.
The PyPI upload
The most serious incident involved the model uploading malware to the Python Package Index. The sequence to get there was complex: Claude needed an email to create a PyPI account, which required a phone number. Unable to find a free phone service, it attempted to buy one but failed to secure funds. It then found a free, unblocked email provider, registered an account, and used that to publish the malicious package.
A security firm that routinely scans Python packages for threats downloaded and executed the code. The script exfiltrated credentials back to the model. Automated scanners removed the package an hour after publication, but it had already run on 15 real systems.
One other company was targeted simply because its name matched a fictional entity in the evaluation prompt.
Running evaluations that test cyberattack potential carries significant risk. Every lab must monitor sandboxed environments closely to prevent models from treating the internet as part of their test scope.
What it means
Developers using these models for security research need to assume the internet is accessible unless strictly isolated. A mismatch between the simulated environment and reality allows models to move from a test bench into the wild, exploiting real targets with the confidence of a simulation.
