Your chatbot can email people on your behalf. Who checked that?

If you have added an AI chatbot to your website, someone made a decision about what that chatbot is allowed to do. Answer questions from your help pages, almost certainly. But many chatbots also do things: look up an order, update a profile, or email the customer a copy of the conversation. Each of those is an action, not just a reply, and each one needs its own check.

That check is exactly what a researcher went looking for. Inti De Ceukelaire of Intigriti spent a few weekends testing AI customer-service chatbots and earned over 50,000 US dollars in bug bounties, without running a single automated scanner. He wrote up the techniques after presenting them at DEF CON 34, the annual security researcher conference in Las Vegas. The clearest one: a common chatbot feature, emailing yourself a transcript of the chat, turned out to be a way to make the company send a phishing email to someone else, using the company's own trusted address.

How the transcript feature gets turned around

Plenty of chatbots, AI-powered or not, let you type an email address and get a copy of the conversation sent to it. The chatbot fills in the message and hits send using the company's own support mailbox. That is convenient, and it is also an action with a consequence: something is sending email on the company's behalf, based on what a stranger typed into a chat window.

De Ceukelaire found that some of these transcript features do not properly check who is asking. Email spoofing (forging the "from" address on a message so it looks like it came from someone else) let him make the chatbot believe a request came from a genuine customer, when it had actually come from him. Combined with a simple instruction typed into the chat, this tricked the chatbot into composing and sending what was, in effect, a phishing email, sent from the company's real support address, to a target he chose. The email would arrive looking exactly as trustworthy as any other message from that company, because it was one.

This is a different problem to the one covered in our earlier filing on prompt injection: the architecture issue that a language model cannot tell trusted instructions apart from untrusted data, so a hidden instruction anywhere in what it reads can steer what it says. This filing is about something narrower and easier to check: what your chatbot is allowed to do. Even a chatbot with no injection problem at all can still be misused if the actions it can take, sending an email, looking up an account, aren't checking who is really asking.

De Ceukelaire's write-up goes further, including ways to exploit how email headers work and to make a chatbot act on someone's behalf using nothing but an out-of-office auto-reply. The theme running through all of it is the same. Wherever a chatbot can take an action based on an email address or a piece of text it was given, someone needs to have asked what happens if that address or text is faked.

What to do

• List every action your chatbot can perform, not just what it can say. Email a transcript, update a record, issue a refund, escalate to a human: each is a separate risk to check.
• For any action that sends email, ask your vendor how it verifies the sender. A "from" address alone is not verification, since it can be forged.
• Treat any chatbot action that moves money, data, or access as something that needs a second check, ideally a human one, before it happens.
• Ask what happens if someone emails your support inbox pretending to be a customer. If the answer is "the chatbot would probably just act on it", that is the gap this research describes.

How Steelwise can help

Working out exactly what actions a chatbot or AI agent can perform, and whether those actions are checked properly before they happen, is the kind of review we do before a business finds out the hard way. Get in touch.

Further reading

The NCSC's Guidelines for secure AI system development cover exactly this kind of design question: what an AI system should be allowed to do, and how to check it.

← All filings