All tabletop exercises
Outage / resilience·Starter·Technical audience
An expired certificate takes payments offline overnight
The prompt — read this to the room
At 00:01 on the first of the month, card payments at a 300-store pharmacy chain start failing. Tills show ‘connection error’ and the mobile app cannot place orders. Within twenty minutes the on-call engineer finds the cause: the TLS certificate on the internal payment gateway expired at midnight. Nobody knew it existed. The engineer who installed it left two years ago, and the renewal reminders went to his old mailbox. Issuing a new certificate needs approval from someone who is asleep. Stores open at 07:00. How do you fix it tonight, and how do you find the next one?
Total duration
1 h 15 min
People to involve
5
Best fit
Retail, Healthcare
Who to involve
Most important first.
- IT operations
- Developer / DevOps
- Cloud engineer
- CIO / CTO
- Customer support
What it tests
- Restore service quickly through an agreed emergency change path.
- Avoid unsafe shortcuts such as disabling certificate validation on clients.
- Build an inventory of certificates and owners so the next expiry is not a surprise.
- Prepare for shorter certificate lifetimes with automation.
Outline of activities
- 1Payments fail at midnight: finding the cause
- 2The emergency fix and who can approve it
- 3Stores, app users and the 07:00 deadline
- 4Finding every other certificate that could do this
- 5Automation and ownership
Run sheet (Pro)
The facilitator's timeline of execution, the detailed steps for running the session and the questions to ask each person.
Loading…