A cleanup script deletes a third of the customer records
The prompt — read this to the room
At 16:30 on a Thursday a developer at a 200-person subscription software company runs a data-retention cleanup script against what he believes is the staging database. It is production. Before he kills it, the script has deleted 38% of customer account records and their linked invoices. The app is still up, but thousands of customers now see empty accounts or ‘user not found’. Support tickets are arriving at 40 a minute. The last full backup finished at 02:00. Nobody acted maliciously. How do you get the data back, and what do you tell customers?
Who to involve
Most important first.
- Developer / DevOps
- Cloud engineer
- Incident response lead
- Customer support
- CIO / CTO
- Communications / PR
What it tests
- Choose a recovery method that limits data loss between the backup and the deletion.
- Stop further damage, including writes that make the deleted data harder to recover.
- Keep customers informed accurately while recovery is under way.
- Run a blameless review that fixes the access path, not the person.
Outline of activities
- 1The mistake and the first five minutes
- 2Freeze or keep serving? Limiting further damage
- 3Recovery options: point-in-time restore, replay, partial merge
- 4Customer communication while the fix runs
- 5Blameless review and guardrails
Free preview from the run sheet
First inject · minute 6
Script output
cleanup_retention.py: connected to db-prod-01 (env var DB_URL). Deleted 1,420,388 rows from accounts, 3,904,112 from invoices. Process killed at 16:34.
Questions for the Developer / DevOps
- What do you stop or pause right now so the damage does not spread?
- How do you merge records created after the deletion back into a restored database?
- What guardrail would have stopped this script from connecting to production?
Built on: NIST SP 800-34r1 Contingency Planning Guide for Federal Information Systems; NIST CSF 2.0 (RC.RP incident recovery plan execution); GDPR Article 4(12) definition of personal data breach (includes accidental loss) and Article 33 (72 hours); Google SRE book: Postmortem Culture (blameless postmortems).
The full run sheet below has every timed inject, the facilitator guide and questions for each role.
Run sheet
The facilitator's timeline of execution, the detailed steps for running the session and the questions to ask each person.
Loading…