How to write a blameless postmortem
An outage is only wasted if you learn nothing. Here is a template for blameless postmortems that lead to real fixes.
A postmortem (incident review) is a written analysis of an outage or failure, written to learn and prevent a repeat. A good one changes the system. A bad one finds someone to blame.
Why blameless
If people fear punishment, they hide mistakes and details, and you lose the information you need. In most incidents, a person made a reasonable choice that the system allowed to become a disaster: a missing check, a confusing tool, an unsafe default. Ask "how did this become possible?", not "who did it?".
A simple template
1. Summary. Two or three sentences: what broke, for whom, for how long.
2. Impact. Be specific: which users or features, duration, error rate, orders or money affected, any data loss.
3. Timeline (in UTC). From first symptom to full recovery:
- Time the problem started (often found in the logs, not the alert).
- Time it was detected, and how (alert or a customer).
- Time of each key action and decision.
- Time of mitigation and resolution.
The gaps are insights: long time to detect means monitoring is weak; long time to mitigate means runbooks or rollback are weak.
4. Root cause and contributing factors. Use "five whys": ask why repeatedly until you reach a fixable system cause. There is rarely one cause, and usually several things lined up.
5. What went well. Fast rollback, good alerts, clear communication. Keep what works.
6. What went wrong, and where we got lucky. Luck means next time may be worse.
7. Action items. The most important part. Each item has:
- A concrete task, not "be more careful".
- An owner and a due date.
- A priority, and a link to the ticket.
Prefer actions that remove the whole class of problem (automation, guardrails, tests, safer defaults) over actions that depend on people remembering.
After writing
- Review it with the team and anyone affected.
- Share it widely. Others learn from it.
- Track the action items to completion. A postmortem with unfinished actions is a record of the next outage.
Tips
- Write soon, while memory is fresh, but after the incident is calm.
- Collect facts first: graphs, logs, chat history.
- Avoid words like "human error" as a root cause. Dig deeper.
Takeaway: describe the incident factually, find system causes, and finish the action items.
System Design series, Part 5 (failure and security): 3 of 6. Previous: Circuit breaker and bulkhead: stop one failure from spreading.
