The website your AI invents, and the attacker waiting to register it

· Carl Heaton · AI Security

Someone on your team asks an AI assistant for a supplier's login page, a courier's tracking site, or the portal for a bank you use. The assistant answers with a tidy, plausible web address. They click it, or paste it into their code, without a second thought. That is the moment this attack is built around, and the address the AI gave them may never have existed until a criminal bought it last week.

Large language models, the technology behind AI assistants like ChatGPT and Claude, do not look web addresses up. They generate them from patterns in language, the same way they generate sentences. So they routinely produce domains that are not real. Attackers have worked out that they can register those invented addresses first, stand up a fake login page, and wait for the AI to send people to them. Palo Alto Networks' threat intelligence team, Unit 42, calls this phantom squatting, and its new research shows it is already happening.

Why a made-up address is so dangerous

The danger is trust. When a model hands back a link, the person or the tool receiving it increasingly treats it as fact. There is no phishing email to spot, no dodgy advert to hover over. The suggestion comes from a tool the reader chose to use and expects to be helpful.

A brand-new domain is also invisible to the usual defences. Blocklists, threat feeds, and reputation scores all need a website to misbehave for a while before they flag it. A domain registered yesterday has no track record, so those filters have nothing to catch. By the time they notice, the visitor has already been sent there by a tool they trusted.

To measure the scale, Unit 42 asked two AI models 685,339 questions about 913 well-known brands, across technology, finance, healthcare, and government. The models produced 2.1 million links. Around 250,000 of the invented domains had no owner at all, each one available for whoever registers it first. Worse, the models often invent the same fake address for the same question, which means an attacker can guess the next target rather than wait for it.

It is already stealing real money

This is not a laboratory finding. In one case Unit 42 predicted that AI models would invent a domain resembling a national postal service's marketplace. Twenty-three days later, an attacker registered that exact address and put up a phishing kit that copied the real storefront in real time. It stole card numbers, bank-transfer details, and identity documents, with a Telegram bot letting the operator wave through victims' one-time passcodes by hand. In a second case, Unit 42 flagged a fake postal domain 51 days before anyone registered it. The attacker then dressed it up with a fake 4.8-star rating and a claim of two million users, and used it to push a malicious Android app.

It has a sibling worth knowing about if your team writes software. Slopsquatting is the same trick aimed at code: AI coding tools invent names for software packages that do not exist, and attackers register those names and fill them with malware. One campaign, PhantomRaven, hid malware in 126 packages that were installed more than 86,000 times. In both cases the pattern is the same, and it is the real shift to understand: AI output is quietly becoming input. People and automated agents act on invented links and names before anyone checks whether they are real.

What to tell your team this quarter

The fix does not need new software. It needs a habit, and it is worth saying out loud to the people who use these tools every day.

  • Do not trust a link because an AI gave it. Before anyone types a password or paying details into a site an assistant suggested, confirm it is the real, official address. Get there the way you always did: a bookmark, a search you recognise, or the address on a physical letter or card.
  • Keep AI agents on a lead. If you use tools that can open links or download files on their own, do not let them act on a model-generated address without a person checking it first. An agent has no instinct to hesitate the way a person does.
  • Treat model output as a draft, not an authority. A link, a domain, or a package name from an AI is an unverified suggestion. Verify it before it becomes an action.

None of this asks your team to distrust the tools. It asks them to keep one old instinct switched on: check the address before you trust it. That instinct is exactly what these attacks are counting on people to drop.

How Steelwise can help

Setting a sensible policy for how your colleagues use AI tools, and where a human check has to sit before an AI-suggested link or file is acted on, is the kind of practical review we do with clients. Get in touch.

Further reading

← All filings

We use cookies to understand how visitors use this site. More information.