AI system·Intermediate·Mixed audience

Your customer-support chatbot is tricked into revealing other customers' data

The prompt — read this to the room

A user posts screenshots on social media of your customer-support chatbot. By pasting a crafted instruction into the chat, they got it to look up and display order histories, addresses and partial card numbers for three other customers. The bot uses a large language model with a tool that queries the order database using the customer ID the model supplies. The post has 40,000 views and replies show other people trying the same trick. The bot handles 9,000 conversations a day and is the only support channel open overnight. It is 22:30.

Total duration

1 h 45 min

People to involve

Best fit

Retail, Saas, Finance

Who to involve

Most important first.

CISO / Head of security
Developer / DevOps
Privacy officer / DPO
Customer support
Communications / PR
Legal counsel
Incident response lead

What it tests

Decide quickly whether to take a customer-facing AI system offline, and what replaces it.
Scope the leak from conversation and tool-call logs, not from the social media post.
Fix the design flaw, the model choosing whose data to fetch, rather than adding prompt filters.
Handle notification and public response for an AI-driven data exposure.

Outline of activities

1The post: confirm the behaviour and its reach
2Kill switch: disable the bot or its tools
3Scoping through logs: who tried it and what came back
4The real fix: authorisation outside the model
5Notification, public response and AI governance

Run sheet (Pro)

The facilitator's timeline of execution, the detailed steps for running the session and the questions to ask each person.

Loading…