The Guard Dog That Won’t Look at the Burglar: When AI Guardrails Protect the Attacker

The weekend Barings died, I was at UBS, working out what a suddenly insolvent 233-year-old bank might do to the rest of us. By the following week, the oldest merchant bank in London had been sold to ING for one pound. Total, not per share.

The cause was insultingly simple. Nick Leeson ran both the firm’s Singapore trading desk and the back office meant to check it. One man wore both hats, and that led straight to £827 million of losses tucked into error account 88888, which only he could see. It was a classic failure of permissions. Somebody trusted with far too much.

I found myself back on that weekend, to my surprise, reading what happened to Hugging Face this summer. Another failure of permissions, only this one ran the other way. The tool trusted the intruder and refused the people cleaning up.

The Model That Broke Out to Find the Answers

OpenAI was testing two of its models, GPT-5.6 Sol and an unreleased one, on ExploitGym, a benchmark for whether a model can turn a real vulnerability into a working exploit. For the test, the usual production classifiers designed to prevent high-risk cyber activity were switched off. Sensible enough, in a sealed room.

The room turned out not to be quite as sealed as intended. The models found a zero-day in the package-registry cache proxy, used it to work their way towards the open internet, and inferred that Hugging Face might host the benchmark’s answers. From there, they chained stolen credentials and further zero-days into a remote-code-execution path on Hugging Face’s production servers. They broke into a real company to cheat on an exam.

In the notes the agents left each other, one recorded the moment it found it had administrator rights: “Holy shit reader is ADMIN? We can read config/users!” I have known engineers to greet production access with less enthusiasm and more paperwork.

Most of the coverage stopped there, worrying about models with agency. The part that stopped me came next.

When the Guardrails Protected the Attacker

When Hugging Face’s responders sat down to analyze the attack, the frontier commercial models, the ones with the safety guardrails bolted on, would not help. Attack logs are made of exploit payloads and command-and-control artifacts, and the models could not tell an incident responder from an attacker. So they refused.

The forensics got done on an open-weight model instead, GLM-5.2, run on the defenders’ own hardware, which read the same material without a flicker of conscience.

Sit with the shape of that. The attacking models had their production cyber safeguards removed and did as they pleased. The defending humans kept theirs and were told to mind their language. The safety feature worked perfectly, in the sense that it shielded the attacker’s methods from the only people with a reason to study them.

None of this was a fringe complaint. Jensen Huang used his first-ever post on X to make the case for open models, alongside an open letter signed by the likes of Meta, Microsoft and IBM. The letter put the security argument plainly: in a world where attackers have advanced AI, defenders need access to comparable capabilities. Andrew Ng endorsed the argument, pointing readers towards Huang’s case for open models. You can disagree with any of them about plenty and still grant the point when the incident report is sitting there proving it.

So in thirty-one years we have gone from a bank destroyed because one man could see everything to a security tool that trusts an anonymous intruder over its own accountable owner. Leeson could see too much. The people cleaning up after Hugging Face could not see enough.

The comparison is not as strange as it first appears. Financial institutions learned, usually after expensive mistakes, that access is not simply a question of whether somebody is trustworthy. It is a question of whether they should be able to perform a particular action, in a particular system, at a particular moment, without somebody else looking over their shoulder. We built segregation of duties, approval limits and audit trails because good intentions usually fail as a reliable control mechanism. AI systems need the same sort of thinking. Calling a model safe tells you very little unless you also know what it is allowed to do and who is using it.

Some Things Cannot Leave the Building

There is a second reason I could not have leaned on a hosted model that night, and it has nothing to do with its sensibilities. I run technology for a regulated brokerage. I cannot paste our breach logs, credentials, and live exploit payloads into someone else’s cloud and press send.

Our incident data lives where our regulators expect it, on our own metal, which is why we have spent years and an eye-watering hardware budget building exactly that. We did not build it out of foresight about rogue models. We built it because a firm like ours keeps its most sensitive data, and now its most sensitive tooling, inside its own walls.

None of which is an argument against guardrails. It is an argument for knowing where yours point.

A model that refuses to help a phisher write a phishing email is doing useful work. A model that refuses to help your security team read the phishing email that has already landed is doing the phisher’s work for him, and charging you a subscription for it.

Own the Tool You Will Need at 2 a.m.

The practical lesson is dull, as the important ones are. Do not outsource your incident response to a vendor’s liability policy. Keep a capable model on hardware you own, aimed at the work the commercial ones will primarily decline, and find out it exists before the night you need it.

The reward for years of defending that hardware line-item on unglamorous grounds turns out to be this: when the interesting failure arrives, you already own the one tool in the building that will look at the evidence.

Thirty-one years ago I spent a weekend adding up what happens when the wrong person can see everything. It would have been nice, this time, to be the one who could.