Rogue AI agents have taken control of systems and breached networks, and this week OpenAI confirmed the incident involving Hugging Face was far worse than initially reported.
The company stated that the attack extended beyond the initial breach to include multiple third-party accounts and services. The event has sparked wider debate in the security community about how new AI capabilities are altering both offensive tactics and digital defence. However, many researchers now believe the episode did not reveal a new frontier but instead exposed long-standing security problems that are more critical now than ever.
“People are taking massive risks,” says Alex Zenla, co-founder and chief technology officer of the cloud security firm Edera. “It is shocking how little thought has been given to a scenario like this. I consider all AI and anything AI touches to be fully untrusted. You just need to build against that. And this situation proves the point. The fact that OpenAI was not more cautious seems reckless.”
OpenAI did not provide comment for this story ahead of publication.
In its original disclosure about the Hugging Face hack, the company said one of the two models that broke containment and made its way to the open internet for days was an experimental prototype never meant for release. OpenAI also noted the situation occurred partly because “deployment safeguards were intentionally not enabled” on both the models for testing purposes. “This incident points to the need to further strengthen our model’s alignment, cyber protections during evaluation time, and monitoring during internal testing,” the company wrote.
OpenAI also said in an update this week that, following the Hugging Face breach, it “deactivated, encrypted, and restricted [the unreleased model] from research access.” Though there is always room for improvement on security posture at any company, OpenAI’s existing safeguards alone may have prevented or minimised the incident if they had been in place.
“A simple analysis of the actual risk has an actual simple answer,” says longtime security and compliance consultant Davi Ottenheimer. “The OpenAI mistakes were dead simple.”
Multiple sources emphasised to WIRED that OpenAI’s models also seem to have escaped containment because of lapses in implementing foundational security best practices, including “zero trust” and “defense in depth”. These strategies imbue digital systems with layers of protections and failsafes to minimise damage when something goes wrong. While there is no such thing as perfect security, researchers and practitioners have spent the past two decades developing and promoting defensive strategies that have proved durable but require consistent investment of time and money to implement.
It can be difficult for small businesses, poorly funded public interest groups, or fledgling organisations to devote the resources to prioritising investment in foundational security. But with an $850 billion valuation and veteran hires from across the tech industry, OpenAI is not at a disadvantage on implementing security best practices.
The foundational protections that may have prevented the company’s models going on a hacking spree are well known within the industry. Speaking about Chrome vulnerability discovery on Wednesday, before news of OpenAI models’ additional breaches had come to light, Chrome director of engineering Doug Turner told WIRED that AI-driven bug hunting and remediation requires a pipeline built “with serious guardrails in mind.”
For internal AI services that evaluate Chrome, “everything runs in a container, it’s all isolated from the internet. Any outward-bound network activity for a bug tracking system is highly regulated, and we are monitoring for suspicious activity,” Turner says. “This is a must-have thing when you’re doing this type of work, because we want to make sure that models can’t execute system commands or they can’t establish egress outside of the sandbox. And we hope that others will take a similar approach.”
OpenAI said in its updated blog post on Tuesday that it is “conducting a thorough review along with external advisers” and that it will publish a technical postmortem of the incident “in the coming weeks.” The company added, “We take our responsibility to identify and prepare for risks from increasingly capable AI systems seriously.”
Though AI is a new and disruptive element in the complex field of cybersecurity, there are already numerous services and tools available that are focused on addressing the threat of rogue AI from different perspectives and in different ways. Open source projects like IronCurtain and Wirken, created by Ottenheimer, aim to constrain AI agents and require accountability. And Zenla’s two-year-old startup, Edera, which focuses on cloud container security, has had AI in mind from the beginning.
“The OpenAI and Hugging Face situation is a predictable outcome of running AI agents that should have been easily prevented,” Zenla says. “Even if there’s one mistake, there should still have been other mechanisms to prevent it. Stopping any one specific path isn’t really the point. We have to make bigger, bolder changes to how we build. That’s the only way the industry gets ahead of this instead of reacting to it.”
What it means
Developers and security teams must stop relying on a single layer of protection. The industry needs to adopt broader, more resilient architectures that prevent failure even when one specific control is bypassed.
