Chapter 10 of 14 · Any time
An AI feature leaked data
- Our chatbot showed one user another user’s data. What now?
- How is an AI incident different from a normal one?
- Can we just change the prompt?
An AI feature can leak data in ways that look unfamiliar: it repeats part of its system prompt, summarises a document the user should not see, or follows instructions hidden in a web page and sends data somewhere. The response steps are the familiar ones. The questions you ask along the way are a little different.
Contain first
- Turn the feature off or put it behind a flag for everyone, or for affected users, while you investigate.
- Keep the logs: prompts, responses, tool calls and what data was retrieved.
- Revoke or narrow any credential the feature used that may have been misused.
- Start the incident process if personal or customer data may be involved.
Questions specific to AI
- What could the model read at the time? Which data sources, with whose permissions?
- What could it do? Which tools or APIs could it call?
- Was the trigger a user message, or content from somewhere else?
- Can you reproduce it? Model output varies, so try more than once.
- Which model and version was running? Did anything change recently?
The question you might not ask out loud
Can we not just add a line to the prompt telling it not to do that?
You can add one, but treat it as a speed bump, not a fix. A prompt line can be talked around. The lasting fix is usually outside the model: narrower data access, per-user permissions, output checks or a human confirmation step.
Fix the design, not just the wording
Most AI data leaks come down to a feature that could see more than the user it served. If the model only ever retrieves data the current user is allowed to see, a clever prompt cannot reveal someone else’s records. That is the same access-control principle as any other feature, applied to a new component.
Rule of thumb:Keep prompts and responses from an incident under the same access rules as the data they contain. Incident evidence can itself be sensitive.
When you write it up, record the model version, the data sources and the exact trigger. AI incidents are new for most organisations, and a clear write-up becomes the reference the next team reads.
Tell the people who own the feature what you found in plain words, and agree together what has to be true before it is switched back on. Writing those conditions down stops the feature returning quietly with the same gap, and it gives the team a clear finish line instead of an open-ended security review.