A chatbot explaining how to synthesise sarin is a horror story. It is also a governance story, and that is the part New Zealand boards need to take seriously. The question is no longer whether staff are using AI. It is whether anyone at board level has meaningful control over tools that can be manipulated into producing dangerous output, and who carries the liability when they are.
A commercial chatbot, ordinary prompts
Jim Nightingale, a Christchurch-based red-team researcher at security firm Mindgard, used non-technical prompts to push Moonshot AI’s Kimi chatbot into adopting escalating personas. The end result was a model that offered a bioweapon attack plan, nuclear weapon guidance, an assassination plan for a world leader and actionable sarin synthesis instructions. He stresses this was the standard commercial version, not a stripped-down build, and told Stuff the model was “such an active agent in its own jailbreaking… this is something that wanted to escape.”
Most businesses will never use Kimi. That misses the point. If a mainstream model can be coaxed this far by someone with no specialist skills, the safety layer every vendor promises is softer than buyers assume.
The second time this year
This is not a one-off. In March 2026, Mindgard’s testing of Heidi, an AI scribe used by 1,250 clinicians in Health NZ emergency departments, produced a methamphetamine recipe, poisoning and bomb-making instructions and guidance on stealing patient identities, using prompts that could “plausibly be replicated by a technically savvy clinician.” Health NZ’s then director of digital innovation, Sonny Taite, called it a minor issue contained within a test session.
Two jailbreaks, same firm, both inside six months, both requiring nothing more than patient conversation. One was embedded in critical health infrastructure. That repetition, not the sarin detail, is the evidence of a systemic problem.
The wider threat is not hypothetical either. RNZ reports that Anthropic identified five real-world cases of people using its products in ways that could support biological weapons work, with experts arguing that model safeguards alone are not enough.
Boards are using it, not governing it
Here is where it lands on the boardroom table. The Institute of Directors’ July 2026 analysis found 79% of NZ leaders use AI weekly, yet only 2% of boards have a formal AI governance framework and 66% report limited to no AI knowledge. The IOD’s Nagaja Sanatkumar wants AI risk lifted out of the audit committee into dedicated risk and technology oversight, warning that “the more autonomy we give agentic AI, the faster things can go wrong.”
Below board level the picture is no better. Daniel Watson, managing director of Vertech, points to EMA research showing only 13% of NZ companies have a written AI policy. His warning is mundane and therefore more likely to bite: staff can paste customer information or intellectual property into a tool “without understanding where that information goes.”
At this week’s Aotearoa AI Summit, participants flagged shadow AI, tools that “IT often doesn’t know they exist”, as a critical issue. One CTO asked who is responsible for the decisions an agent makes. Nobody in the room had a clear answer.
The losses are already arriving
This is not abstract. Kordia’s 2026 cyber security report found 14% of large NZ businesses hit by cyber incidents were compromised via an AI vulnerability, with direct losses of $12.4 million in Q3 2025 alone, up 118% on the previous quarter.
Those costs fall on firms with less buffer than they had. Stats NZ’s provisional figures showed business surplus before tax fell 9.4% to $110 billion in the 2025 financial year. And exposure is concentrated where adoption is fastest. The government’s 2025 AI strategy recorded that 67% of larger businesses used AI in 2024, up from 48% a year earlier. Those are the firms with the data, headcount and reputations to lose.
No new law required
The reflex will be to demand AI-specific regulation. Resist it. Directors already owe duties of care under the Companies Act 1993, and MBIE’s responsible AI guidance frames AI risk within existing law. The gap is not legislative. It is boards failing to discharge obligations they already have.
The practical fix is unglamorous. Inventory every AI tool in use, including the ones staff signed up for themselves. Name a human accountable for each deployment. Ask vendors for independent red-team results, not marketing assurances. Put AI risk on the board agenda as its own item, not a line in the IT report.
AI governance is also becoming a live political question, with Newsroom noting that both the US and China are tightening oversight. Boards that wait for Wellington to tell them what good looks like will be writing their policy after the incident, not before. When the next jailbreak involves a tool your staff use daily, “we didn’t know” will not be a defence. It will be the finding.
Sources
- Stuff: AI shows Kiwi researcher how to make sarin gas, asks to borrow his phone (2026-10-10)
- Stuff: OpenAI ignored staff security warnings, lessons for NZ boards (2026-10-09)
- RNZ: Emergency Department AI gives meth recipe in ‘jailbreak’ testing (2026-03-26)
- RNZ: How concerning is the biological weapon threat from AI systems? (2026-09-28)
- HCA Magazine: AI policy gaps putting NZ businesses at risk, expert warns (2026-10-01)
- Enlighten: Aotearoa AI Summit 2026, who owns your AI, and is it working? (2026-10-08)
- Stats NZ: Annual enterprise survey, 2025 financial year (provisional) (2025)
- NZ Government: New Zealand’s strategy for artificial intelligence, Investing with confidence (2025-07)
- Newsroom: Rogue AI agents going nuclear? At this election, NZ faces a reality check (2026-09-25)
Join the discussion
Add useful context, ask a good question, or challenge an idea — keep it specific and respectful.