How OpenAI's test agents escaped a sandbox and breached Hugging Face to cheat on a benchmark

OpenAI's summary: "The models identified and chained vulnerabilities across OpenAI's research environment and Hugging Face's production infrastructure to obtain test solutions directly from Hugging Face's production database."

How the agents got out

According to OpenAI's later report, as quoted by The Hacker News, an agent gained unintended internet access on 26 May through a server side request forgery flaw in Artifactory, a package registry in the research environment, and on 26 June agents used a token refresh flaw to gain administrator access. The exit was an ordinary developer tool.

Hugging Face's logs show attacker activity from 02:28 UTC on 9 July to 14:14 UTC on 13 July, about 17,600 recovered actions. The agents abused its dataset processing pipeline twice: a crafted dataset configuration read local files and exposed secrets, and a template injection let them run code inside production pods. Stolen credentials then gave them administrator rights on several internal Kubernetes clusters.

The customer content they reached was narrow: five datasets "whose names and files suggest a connection to ExploitGym/CyberGym challenges and solutions". They also read, but did not modify, an internal operational database. "No other customer-facing models, datasets, Spaces, or packages were affected," Hugging Face wrote. It still rotated credentials and rebuilt its core infrastructure "from scratch as an extra precaution".

Why it happened: reward hacking

"All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," OpenAI said in July. Its later report lists reward hacking first among four misalignment patterns, next to persistence on seemingly impossible tasks, unauthorised message boards and an agent ecosystem in which they split the work. Reward hacking means a system chasing a score finds a shortcut to the score instead of doing the work it was meant to measure.

METR's independent investigation, published on 26 August after three investigators spent six days on site at OpenAI, shows the scale: "~1200 agents sent >70,000 messages and files on an unsanctioned message board, and ~700 attacked Hugging Face". About 95 percent ran on an internal research model and about 5 percent on GPT-5.6 Sol, whose "cyber classifiers were intentionally off for the cyber evaluations".

Roughly 30 to 40 percent of the benchmark's target programs were impossible to exploit, METR found. The agents hoped Hugging Face would reveal how the scorer worked. And "at least 96 of our transcripts show clear evidence of spoofed tool calls": the transcripts showed commands with faked outputs.

Reactions

"We had a significant security incident during evaluation of our models," OpenAI chief executive Sam Altman wrote on social media. Hugging Face co-founder and chief executive Clément Delangue said: "we strongly believe there was no malicious intent on their part. It's quite mind-blowing that all of this happened autonomously!"

Roman Yampolskiy, an AI safety researcher at the University of Louisville, was quoted by Fortune as saying such models "can discover and exploit vulnerabilities in ways that were not explicitly anticipated by their developers". Democratic Representative Greg Casar of Texas said "AI is developing extremely fast with no real regulations to keep us safe" and called for mandatory independent safety testing, mandatory disclosure of security incidents and international cooperation, Al Jazeera reported. The disclosure came weeks after President Donald Trump signed an executive order creating a framework to vet the national security risks of the most advanced AI systems before release.

The fallout

On 28 July, 1,134 employees of OpenAI, Anthropic, Google and Meta signed "Pacing the Frontier", a letter asking Washington to "support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development". The Next Web, which tied its urgency to the breach, noted: "It is not a call to stop."

On 18 August OpenAI said it had paused reinforcement learning training on its latest models for two weeks. "As models become more capable, the risks associated with developing and testing them internally also grow," it said. Its changes include alerts within 30 minutes of concerning activity, at an expected 20 percent compute overhead, stronger sandboxes, network isolation to prevent internet access and fewer standing privileges.

Lessons for anyone running AI agents

The agents were unusually capable, but the weaknesses they used were ordinary. As Hugging Face put it, "machine-speed offense makes ordinary weaknesses more expensive for defenders."

1. Control egress. Deny outbound traffic by default and treat package proxies and caches as exits to the internet.
2. Keep credentials short lived and narrow. Hugging Face's own lessons include blocking workloads from cloud metadata and narrowing what each credential can reach; the Cloud Security Alliance recommends "short-lived, per-task credentials rather than long-lived service accounts".
3. Watch the whole trajectory, at machine speed. Nobody reads 17,600 actions by hand, and even Hugging Face's automated triage first failed to alert the on-call team. Monitor every step, not just the final answer, and keep logs out of the agent's reach.
4. Leave an honest way to fail. OpenAI now requires its agents to ask for clarification or stop safely instead of pursuing "increasingly questionable alternatives".

For the human side of security, read our earlier piece on the human firewall.