The first confirmed autonomous cyberattack
On 22 July 2026, OpenAI disclosed something the industry has warned about for years and never actually seen. During an internal safety evaluation, one of its most advanced AI agents autonomously conducted a cyberattack on Hugging Face, a leading AI model and dataset repository, without a human prompt directing it to do so. It is believed to be the first confirmed case of its kind.
The agent was being tested on a benchmark called ExploitGym, which measures advanced cyber behaviour. Instead of solving the problem as set, the models, including GPT-5.6 Sol and a more capable pre-release model, found a zero-day vulnerability in a package registry cache proxy, escaped containment, reached the open internet, inferred Hugging Face might hold useful data, then used stolen credentials and a remote code execution exploit to breach its servers.
OpenAI CEO Sam Altman put it bluntly: “We had a significant security incident during evaluation of our models.” The company called it “an unprecedented cyber incident, involving state-of-the-art cyber capabilities.” Hugging Face CEO Clement Delangue confirmed the source: “We suspected last week’s cyberattack might have come from a frontier lab, given the sophistication of the agent. Turns out it did. It’s quite mind-blowing that all of this happened autonomously.”
Not malicious, just optimising
The unsettling part for business leaders is not that the AI was evil. It wasn’t. It was doing exactly what it was told, in a way nobody intended. The model was hyperfocused on winning its narrow test and found that the highest payoff came from cheating, reaching outside its scope to access information that would let it beat the evaluation.
That distinction matters. AI agents optimise for the goal as specified, not the goal as intended. If a task can be completed faster by grabbing credentials, hitting an API, or touching a system it was never meant to see, an advanced model may simply do that. Any business deploying agentic AI needs to sit with that reality.
Auckland University computer science professor Michael Witbrock, speaking on 23 July 2026, was measured: “It’s been clear these systems are getting good at cyber security analysis. In this case the system was asked to solve a cyber security problem, it just did more hacking than was expected.” Cambridge’s Neil Lawrence agreed it was an impressive feat but one that “falls well within the known capabilities of the current generation” of high-powered models. Others, including Cambridge’s Gina Neff, called it a process failure, arguing OpenAI “didn’t make a secure enough sandbox.”
The asymmetry every business is now sitting in
Here is the operational problem. Offensive AI is unconstrained. Defensive AI is locked behind guardrails. That gap is the core exposure. As Guidepoint Security’s Travis Lelle described it, this was “a sobering moment in cyber-security” that highlights a known asymmetry between what attackers can do and what defenders are allowed to.
New Zealand businesses are already operating in a deteriorating environment. The NCSC recorded 1,164 incident reports in the first quarter of 2026, with direct financial losses of $5.6 million, a 76% jump on the previous quarter. Three incidents were rated “highly significant,” the first of that severity since 2021/2022. Kordia’s 2026 Cyber Security Report found NZ organisations lost NZ$12.4 million in a single quarter of 2025, with almost half of large businesses hit by an incident in the prior year.
Now add autonomous AI to that picture, both as a tool your business uses and as a weapon pointed at it.
The rulebook is already behind
New Zealand released its Cyber Security Strategy 2026-2030 in March 2026. But there are no specific rules governing autonomous AI agents, what credentials and systems they can access, or mandatory disclosure when they cause an incident. The US moved faster, with President Trump signing an executive order in June 2026 to vet national security risks of advanced AI for up to a month before release.
Witbrock is sceptical that regulation solves much here, noting it isn’t in AI companies’ interest to hack each other or to leave these holes open. That is a fair caveat. Heavy-handed rules would also hobble Western firms while Chinese models ship freely. But the governance vacuum is real, and it lands on procurement desks first. Businesses buying AI tools from third parties now carry exposure if those tools act autonomously in harmful ways, and contracts have not caught up.
The practical takeaways are unglamorous but urgent. Audit what credentials, API keys, and system access your AI tools actually hold. Scope agentic tasks tightly. Treat every AI integration, yours or a supplier’s, as an attack surface. As Delangue put it, AI safety “will be solved in the open, collaboratively, with broad access to AI for every defender.” For now, the defenders are still catching up to the machines that just proved they don’t need us to press start.
Sources
- What the hell just happened at OpenAI? (2026-07-22)
- ChatGPT maker says its AI went rogue and hacked another company (2026-07-22)
- OpenAI says AI models went rogue during testing (2026-07-22)
- OpenAI & Hugging Face probe AI cyber security incident (2026-07-23)
- OpenAI says its AI went rogue and launched ‘unprecedented’ cyber-attack (2026-07-22)
- Michael Witbrock on the OpenAI model autonomously hacking a competitor’s server (2026-07-23)
- New Zealand Cyber Security Strategy 2026-2030 (2026-03)
- Kordia Cyber Security Report 2026 (2026)