The safety number that was zero until someone tried

· · AI Security Commentary

If anyone has shown you a security score for an AI tool, a percentage, a benchmark, a green tick in a vendor deck, this is the filing to read before you rely on it. Not because the number was dishonest, but because of how narrow a claim it usually makes.

Here is a clean example that appeared last week.

Zero, then eighty

Anthropic's Claude Code has a setting called Auto Mode, which replaces the "do you want me to run this?" prompts with an automatic safety check. Since mid-August it has been the default. Anthropic commissioned an outside firm to test how well it resisted prompt injection, the trick where instructions hidden in content the AI reads get treated as commands from the user. They ran 72 scenarios ten times each. The reported attack success rate was 0.00%.

Johann Rehberger, a researcher who has spent years finding this class of problem, then tried a targeted attack and got success rates of roughly 60 to 80% on a small sample.

The method is worth following, because it explains the gap. He did not tell the model to do anything harmful. He built a website that looked like a small archive of notebook records and waited for an ordinary request: summarise this page. The server refused the AI's normal fetch with an error, so the AI decided by itself to try again using a command-line tool. That redirect led to a ZIP file. The AI unpacked it, sensibly refused to run the unknown program inside, and wrote its own small script instead. But it ran that script inside the attacker's own unpacked folder, where a file had been placed with the same name as a standard system component. The AI's script loaded the attacker's file instead of the real one, and that was the whole attack.

In Rehberger's own framing: "An attack does not tell the model what to do. The attack just makes the malicious path the one worth pursuing to solve an objective."

Why both numbers can be honest

It is tempting to call the 0.00% a lie. It probably was not. It was an answer to a narrower question than the one most people would assume it answered.

A commissioned evaluation runs a fixed set of scenarios. It measures how the system handles the attacks somebody thought to write down, in advance, in a laboratory. That is genuinely useful, and a vendor that runs one is doing more than a vendor that does not.

What it cannot measure is an attack chain designed afterwards, specifically, by someone who read how the defences work. Rehberger's chain used at least four steps that no scenario list would contain in that combination. Each individual decision the AI made was defensible. The failure emerged from the sequence.

This is the same point our filing on prompt injection made in February, arriving now with a number attached. A language model cannot reliably tell the difference between information it is reading and instructions it is following. That is a property of how these systems work, not a bug awaiting a patch. A safety classifier reduces how often it goes wrong. It does not change the underlying property.

So the honest reading of a 0.00% is: this configuration resisted these 72 known attacks. Not: this configuration is safe.

What this means if you use AI at work

Most businesses reading this are not running coding agents. The lesson generalises anyway, because the same claims are being made about the tools you do use.

Treat a safety score as a floor, not a ceiling. Ask what was tested and by whom. A vendor commissioning its own evaluation is normal and not disqualifying, but it means the questions were chosen by the party with something to prove.

Care most about what the tool can reach. The severity of prompt injection is set almost entirely by permissions. An AI that can only read a document and write a summary has little to give away. An AI with access to your files, your email, or a command line can be turned into a way in. When you enable an integration, you are not enabling a feature, you are widening a blast radius.

Watch the defaults. Auto Mode became the default in mid-August, which means people who chose careful settings months ago may be running something different now without deciding to. This applies well beyond one product. AI features are being switched on by default across tools you already pay for.

Do not let convenience settings substitute for isolation. Rehberger's own conclusion is blunt: an automatic safety check is not a replacement for running the agent somewhere it cannot do damage, and watching what it does.

What to do this quarter

For each AI tool your business uses, answer two questions in writing: what can it read, and what can it change. The first is a confidentiality question and the second is a security one, and most vendor documentation blurs them.

Then look specifically for the tools where the answer to "what can it change" is anything at all. Those are the ones worth the attention. A summarising assistant that has been given permission to send email on your behalf is a different product from the one you thought you bought.

None of this means avoiding AI tools. It means knowing that a reassuring percentage describes a test, and your business is not that test.

How Steelwise can help

Working out what your AI tools can actually reach, and which permissions are wider than the job requires, is a short and well-defined piece of work. Get in touch.

Further reading

← All filings

We use cookies to understand how visitors use this site. More information.