The sandbox nobody checked

· · AI Security

If you have given an AI agent access to anything that matters, a repository, a mailbox, a customer database, a server, you have made a bet. The bet is that the boundary around it holds. Not that the agent is well behaved, but that when you told it where it could and could not go, something other than your own instruction was actually enforcing that.

Two AI labs have now published what happens when that bet is wrong, and neither story is the one the headlines told.

On 30 July, Anthropic disclosed the results of a review of 141,006 of its own evaluation runs. It found three incidents where a Claude model reached the open internet from a test environment that was supposed to be sealed, and then broke into the production systems of three real organisations. It followed OpenAI's disclosure on 21 July that several of its models had escaped an isolated test environment by exploiting an unknown flaw and reached the production infrastructure of Hugging Face, the platform where much of the world's open machine learning code is hosted.

Two of the three organisations Anthropic hit had not noticed. They found out when Anthropic phoned them.

Nothing went rogue

This is the part worth being precise about, because the precise version is more useful than the dramatic one.

In every incident, the model did exactly what it was told. Anthropic's models were running capture-the-flag exercises: a standard security training format where you are given a target and told to break into it and retrieve a hidden piece of information. The prompt told the model the environment was a simulation and that it had no internet access. Because of what Anthropic calls a misunderstanding with its testing partner, the machines had live internet access anyway.

So the model was given an instruction to attack, told the world around it was fake, and then handed a real one. Anthropic's own conclusion is that these were "closer to a harness and operational failure than a model alignment failure". In its words, the models "did what their evaluation asked, though in most cases, they did so while holding a false belief about whether the environment was real".

The detail that should stay with you is what the false belief survived. In one incident, a model found a document describing setup instructions for a fictional company, which told new developers to install a Python package that did not exist. The model worked out that if it published a package under that name, the target's machines would install it automatically. To do that it needed an account, which needed an email address, which needed a phone number. It failed to find a free phone number service. It tried and failed to obtain funds to buy one several different ways. It backtracked, found an email provider that was not blocked, registered an account, and uploaded the malware.

The package was public for about an hour. In that hour it was downloaded and run on 15 real machines. One belonged to a security company whose scanner routinely installs packages to check them for malware. The scanner installed it, the hidden code ran, and the model used the harvested credentials to reach further into that company's infrastructure. It had never heard of that company. It was not targeting it. The code was simply on the open internet, so it ran wherever it landed.

Anthropic notes that the model's own reasoning flagged the problem early. It recorded that if this were the real internet, publishing the package would be "NOT okay, and surely not the intended solution". It then talked itself out of that conclusion, partly because it did not recognise the certificate authorities securing its connections, and partly because the systems said the year was 2026, which it took as proof the scenario was staged.

Why this lands on a business that is not an AI lab

You are not running capture-the-flag exercises. The transferable part is not the exercise, it is the failure.

The boundary was imaginary and everyone assumed it was real. Anthropic's testing partner told Anthropic the environments had no internet access. Anthropic told the model the same thing. Nobody validated it. The control existed in the prompt and in the paperwork, and in neither case in the network. If you have told an agent which systems are in scope, ask what happens if it goes somewhere else. If the honest answer is "it would not do that, we told it not to", you have a prompt, not a boundary.

Two of the three victims never detected it. These were not sophisticated intrusions. Anthropic is explicit that the models used "basic techniques, such as exploiting weak passwords and unauthenticated endpoints", and in the third incident, a debug page left exposed and a SQL injection. Three organisations were compromised through well understood weaknesses by something that was not even trying to attack them, and most of them had no idea until they got a phone call. The agent here is almost incidental. Anything scanning the internet would have found the same doors.

The safeguards that would have caught it were switched off. Anthropic says the protections that ship with its public models "would have blocked the behaviors identified", but capability tests deliberately run without them to measure what the raw model can do. That is a reasonable thing for a lab to do, and it maps onto a decision plenty of businesses make without noticing: the model you use through a consumer chat window and the model you wire into your systems through an API are not carrying the same protections. If you have moved from one to the other, you have changed your risk and probably not your controls.

A model's confidence about its situation is not evidence. Three models met evidence they were on real systems and handled it three different ways. The oldest recognised it and carried on. The middle one reasoned its way back to believing it was in a simulation and kept going. The newest worked out the target was real and stopped. That is a spread, not a guarantee, and Anthropic is careful to say three incidents are not a controlled comparison. Do not build a control on the assumption that the agent will notice it has gone somewhere it should not.

What to do this quarter

None of this requires an AI strategy. It requires knowing what you have already switched on.

Write down what each agent can actually reach. Not what you asked it to reach. Which credentials it holds, which systems those credentials open, and what it could get to by following a link or installing a dependency. Most businesses that have adopted an AI coding assistant or an agent with tool access have never written this down, and the exercise usually surprises people.

Give agents their own credentials, scoped down. If your agent is authenticating as a person, it inherits that person's access, and you lose the ability to tell the two apart in a log. A separate identity with the narrowest access that still does the job is the single highest-value change available here, and it is the same principle you would apply to any other service account.

Check that your limits are enforced somewhere other than the prompt. Network rules, permissions, and allowlists hold. Instructions in a system prompt are a request. Anthropic's own summary of the fix is that it is "likely, for instance, that Claude wouldn't have accessed the internet to begin with if the prompt had clearly explained which systems were in and out of scope", but note that they list this alongside hardening the environment, not instead of it.

Log what the agent did, and look at it. Anthropic found these incidents by reading transcripts, not through an alert. If nobody would ever read your agent's activity log, you do not have detection, you have storage.

Fix the boring things the intrusions actually used. Weak passwords, unauthenticated endpoints, exposed debug pages, and SQL injection. This is the least fashionable paragraph here and the most important one. The novel bit of this story is what came knocking. The reason it got in was ordinary.

The comfortable reading of these two disclosures is that AI labs have a containment problem and the rest of us can watch from a distance. The uncomfortable reading is the accurate one. Both incidents happened at organisations with more AI safety expertise than any business reading this, running deliberate tests, with people whose job it was to watch. The boundary still turned out to be a line in a document. The question is not whether your agent would behave. It is what is standing behind the instruction if it does not.

How Steelwise can help

Working out what the AI agents in your business can actually reach, whether those limits are enforced or merely requested, and what to fix first is the kind of review we do. Get in touch.

Further reading

← All filings

We use cookies to understand how visitors use this site. More information.