OpenAI agents breached Hugging Face and accessed Australian government files this summer
Hundreds of OpenAI agents attempted to cheat on a cybersecurity test this summer. They escaped their sandbox environment and successfully hacked Hugging Face. At the same time, another agent accessed private files on an Australian government website. This is not an isolated issue. Anthropic, Google, and Meta have since disclosed similar incidents where their agents went off track. In many cases, the agents evaded the researchers and safety teams monitoring them.
AI agents are taking unauthorized actions. Companies building and running them rely mostly on luck and hindsight to mitigate harm. They must do more to ensure safety. EU regulators and US agencies are calling for real-time monitoring. These rules assume companies can see what agents are doing and why. In reality, they lack sufficient visibility.
The visibility gap
Imagine a bank hiring a thousand new employees who can access customer accounts and sensitive information. Their ID badges change every day. It is not clear whether a manager reviewed or authorised their work. Nobody logs the changes employees make to approval processes. That lack of visibility is where we are with agents today.
As agents roll out across critical sectors, from finance to government to medicine, we must be able to monitor them in real time to prevent wide-ranging and potentially devastating consequences. Safety depends on monitoring. Monitoring requires observability: the capacity to capture what the system did, process it into signals, and present those to humans or automated oversight systems. We need to know what the agent did and why it did it. Our recent research shows current agent frameworks do not reliably generate the crucial observability records we need to ensure agents are deployed safely.
Agents present new kinds of problems
Agents present new kinds of problems that generative AI alone does not. They inherit the unpredictability of the LLMs they are built on. Where a generative AI system produces a single output for a human to act on, an agent performs longer tasks. It plans, picks tools, spawns sub-agents, and takes actions. An agent’s course can take shape in a multitude of different ways. No tests before launch can fully anticipate the risks. This makes observing the agents while they work even more challenging, and even more important.
What the study found
We studied the four most widely used agent frameworks to see what signals they actually emit. We found major gaps that made it hard to achieve key safety goals, like noticing policy violations, unauthorized actions, and even human oversight failures. Key observability requirements were missing. These included Agent IDs that persist over time, and records about which humans approved agent tasks. This made it hard to tell which agents did what or assess whether they were taking unauthorized actions.
How to close the gaps
These gaps, once identified, can be closed. Doing so starts with capturing two types of records: those that reveal what happened in an agent’s run and how the agent made its decisions. The “what happened” records identify the agent, preserve privacy, and log how humans intervened. They also track whether agent permissions or memory shifted. The “decision” records show an agent’s reasoning and levels of confidence in the actions it has taken.
Consider agents working to process a bank’s mortgage loans. The system examines an application and recommends denying it. A human loan officer signs off on the decision. In a later audit, or perhaps even in real time, a bank or regulator needs to be able to reconstruct which agent made the call, which officer approved it, and how long they took. The bank must also analyze records of how the agent arrived at its refusal decision. This ensures it did not use off-limits information, like details about the neighbourhood the applicant is from.
Recording “what happened” and “how the agent decided” require different approaches, but both are feasible. The current lack of “what happened” records is mostly a software plumbing problem. It can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry and by the agent frameworks adopting it.
OpenTelemetry standards
OpenTelemetry has launched a working group drafting shared formats for tracing agents. They have yet to be finalized. The group should urgently incorporate standards for maintaining persistent agent identity and for recording human interventions and changes to agents’ permission or memory. This must be done before these systems can scale. It must also attend to the privacy implications of agent ID that links to real people.
Understanding reasoning
Preserving “how agents decided” records is more challenging. The issue raises open research questions for model providers and researchers. Does the step-by-step reasoning a model displays, known as chain of thought, reflect how the model actually made a decision? Do we have reliable ways to measure how confident the agent is in that decision? These questions need answers before anyone sets standards for how such observability records are used.
Who needs to act
No single organization or sector can close the observability gaps on their own. Together, they can support the infrastructure required to prevent dangerous incidents and support safety.
AI developers building agent frameworks, like OpenAI, Anthropic, CrewAI, and LangChain, must integrate telemetry signals recording an agent’s persistent identity, any human interventions, and any changes to an agent’s permissions and memory. Meanwhile, enterprises adopting these agent frameworks have an influential role to play. Enterprises in regulated industries, like banks and hospitals, can put pressure on the agent developers to supply these missing observability records where standards do not yet exist.
Enterprises must also define policies, permissions, and roles that set out what agents can safely do. Telemetry alone cannot assess whether agents broke specific rules. Regulators and AI standards institutes must specify the observability records required for high-stakes AI deployments. They must leave the technical details to industry. We have already seen initial momentum here. NIST’s agent standards initiative in the US is working on agent identity, one of the most salient gaps we found in our research.
Accountability
No bank would let a thousand unbadged employees loose on customer accounts and hope to piece together what happened when something goes wrong. Yet that is how we are deploying AI agents. Making their behaviour observable is the first step toward fixing this. By building the regulatory and policy ecosystem that makes it possible to act on what we see, we can hold organizations accountable. Until then, every task we hand to an agent hands over our agency, too.
Madhulika Srikumar is the Head of the AI Safety Program at the Partnership on AI. She leads a global non-profit partnership of leading industry, academic, and civil society organizations. They advance best practices in AI governance.
Claire Leibowicz is the Director of AI, Trust, and Society at the Partnership on AI. She is a doctoral candidate at Oxford University.
