All tabletop exercises
Outage / resilience·Intermediate·Technical audience
The retail trading platform fails at market open
The prompt — read this to the room
At 09:31 on the morning of a major interest-rate announcement, your online brokerage's order-entry service starts returning errors. Customers can log in and see positions but cannot place or cancel orders. Traffic is four times a normal day. A configuration change to the order-routing gateway went out at 08:50, and the network team is also seeing a surge from a few thousand IP addresses that could be a DDoS or could be your own customers' apps retrying. Customers with open positions are posting angrily on social media. Roll back, scale out or block traffic? You have one move to make first.
Total duration
1 h 45 min
People to involve
6
Best fit
Finance
Who to involve
Most important first.
- IT operations
- Developer / DevOps
- CIO / CTO
- Incident response lead
- Cloud engineer
- Customer support
What it tests
- Separate a self-inflicted change failure from an attack using evidence, not instinct
- Choose and sequence recovery actions (rollback, scale, rate-limit) with one decision owner
- Run a single incident channel with timed status updates during a fast-moving outage
- Capture what the firm owes customers and its regulator for missed or failed orders
Outline of activities
- 1Errors at market open: the first five minutes
- 2Change or attack: evidence and hypotheses
- 3Recovery action and verification
- 4Orders in flight, customer impact and status updates
- 5Regulatory notice and post-incident review
Run sheet (Pro)
The facilitator's timeline of execution, the detailed steps for running the session and the questions to ask each person.
Loading…