Incidents and Retrospectives
Stabilize first, then learn how the system allowed the failure. · Execution & Communication · Lesson 68 · 3 min
Incidents and Retrospectives · 3 min
Situation
Customers cannot complete checkout.
Support reports failures after a release. Engineers are investigating. Stakeholders want an explanation and an estimated recovery time.
During an incident, speculative certainty can do harm. The immediate priorities are restoring service, understanding impact, and communicating what is known.
During
Give the response clear ownership.
Identify an incident lead and the people needed for diagnosis and mitigation. Reduce unrelated interruptions so responders can work. Product can help clarify customer impact, priorities, and communication.
Share confirmed symptoms, affected scope, actions underway, and the next update time. Avoid promising a recovery time before there is evidence for one.
Example
Mitigate without waiting for the perfect explanation.
If disabling a newly released payment path safely restores checkout, it may be reasonable while the team investigates. Record what changed and the observed result.
Check for partial transactions and duplicate attempts. Restored page availability does not prove every affected customer record is correct.
After
Reconstruct the chain without blame.
Build a timeline from evidence. Identify the technical trigger, contributing conditions, detection gaps, and why existing controls did not catch the issue.
“The engineer made a mistake” is rarely a useful prevention plan. Ask how review, tests, configuration, rollout, and monitoring made the failure possible or harder to contain.
PM decision
Choose prevention that changes the system.
Assign specific actions with owners and review dates: a missing validation check, a safer rollout boundary, a clearer alert, or an operational reconciliation process.
Avoid an unprioritized list of every imaginable improvement. Focus on actions that reduce recurrence or shorten detection and recovery for material failures.
Remember this
Incident response and learning are different phases.
Stabilize, communicate, and assess impact during the incident. Afterward, improve the system using evidence, clear ownership, and a blameless explanation of contributing causes.