All tabletop exercises
Insider·Starter·Technical audience
A cleanup script deletes a third of the customer records
The prompt — read this to the room
At 16:30 on a Thursday a developer at a 200-person subscription software company runs a data-retention cleanup script against what he believes is the staging database. It is production. Before he kills it, the script has deleted 38% of customer account records and their linked invoices. The app is still up, but thousands of customers now see empty accounts or ‘user not found’. Support tickets are arriving at 40 a minute. The last full backup finished at 02:00. Nobody acted maliciously. How do you get the data back, and what do you tell customers?
Total duration
1 h 15 min
People to involve
6
Best fit
Saas
Who to involve
Most important first.
- Developer / DevOps
- Cloud engineer
- Incident response lead
- Customer support
- CIO / CTO
- Communications / PR
What it tests
- Choose a recovery method that limits data loss between the backup and the deletion.
- Stop further damage, including writes that make the deleted data harder to recover.
- Keep customers informed accurately while recovery is under way.
- Run a blameless review that fixes the access path, not the person.
Outline of activities
- 1The mistake and the first five minutes
- 2Freeze or keep serving? Limiting further damage
- 3Recovery options: point-in-time restore, replay, partial merge
- 4Customer communication while the fix runs
- 5Blameless review and guardrails
Run sheet (Pro)
The facilitator's timeline of execution, the detailed steps for running the session and the questions to ask each person.
Loading…