Most post-mortems are elaborate rituals for burying the truth. A team gathers, a timeline gets drawn, and a list of action items emerges—most of which will never be completed. The real failure mechanisms stay hidden, ready to trigger the next incident. A post-mortem that actually produces lessons learned is not a meeting. It’s an investigation, a negotiation, and a test of whether the organization can stomach the truth. Here’s a method for running one that treats failure as a system problem, not a people problem.
Start with the System, Not the Human
When a production outage or a missed deadline occurs, the reflex is to find the person who pulled the wrong lever. That’s a dead end. Human error is a symptom—the starting point, not the conclusion. The real question is why the system allowed that error to happen and why it wasn’t caught earlier. In failure forensics, we distinguish between the proximate cause (the engineer who ran the wrong script) and the systemic cause (no peer-review gate for database changes). A post-mortem that stops at the first human touchpoint has failed before it began.
This thinking draws from safety science, particularly James Reason’s work on latent conditions and active failures. Active failures are the unsafe acts—the typo, the misconfiguration. Latent conditions are the organizational pathogens lying dormant: poor design, conflicting incentives, training gaps. A post-mortem that only documents the trigger has learned nothing. The real work is excavating and neutralizing those latent conditions.
Mapping the Contributing Factor Graph
Once you have a solid timeline, map the contributing factors. Avoid the term “root cause.” In complex systems, failures rarely have a single origin. Build a graph where each node is a contributing condition or event, and edges represent causal influence. Include technical factors (a misconfigured load balancer), process factors (no code review for hotfixes), and organizational factors (pressure to deploy on Friday afternoons). The graph exposes interactions that a linear “5 Whys” analysis would miss.
For each node, ask: Was this condition known before the incident? If yes, why was it accepted? If no, why was it invisible? This line of questioning often reveals that the organization was operating with known risks that had been normalized. The post-mortem’s job is to de-normalize those risks and make their cost explicit.
Preconditions for Honesty
You can’t run a useful post-mortem in a culture of blame. If the outcome of the process is used to punish individuals, the information you gather will be filtered, softened, and incomplete. People will describe what they think is safe to say, not what actually happened. This isn’t a call for a “blameless culture” as a slogan—it’s a practical requirement. If your organization lacks the psychological safety to discuss failures openly, fix that first. Otherwise, don’t bother with post-mortems. They’ll be theater.
One structural safeguard: separate the post-mortem process from performance evaluation. The facilitator should not be the team’s direct manager. The report should not name individuals except to credit them for surfacing information. When describing actions, use roles: “the on-call engineer” rather than “Jane.” This isn’t about being polite. It’s about getting accurate data. People will tell you what really happened if they trust you not to weaponize it.
Timeline Construction: The Backbone
Build a detailed, minute-by-minute timeline of the incident. This is not a log dump. Integrate multiple data sources: monitoring alerts, deployment records, chat logs, human recollections, system metrics. Each entry must answer three questions: What was observed? Who observed it? How did they respond? The timeline exposes gaps between detection and response, reveals where information was lost or delayed, and often shows that the “root cause” was actually a cascade of smaller failures.
A common mistake is to stop the timeline at the point of remediation. Continue it through the post-incident period. How long did it take to confirm the fix? Were there secondary effects? Did the remediation itself introduce new problems? Many organizations declare victory too early and miss the second-order failures that are often more instructive than the initial trigger.
Writing Action Items That Don’t Rot
Most post-mortem action items are vague aspirations disguised as tasks. “Improve monitoring” is not an action item. “Add an alert for connection pool exhaustion with a p99 latency threshold of 500ms, assigned to the database reliability team, due by end of sprint 23” is an action item. Each one must have a clear owner, a concrete deliverable, and a deadline. If you can’t define what “done” looks like, the item isn’t ready to be written.
Classify action items by type. Is this a fix for a specific bug? A process change? A training need? A design review trigger? Classification prevents teams from defaulting to “add more alerts” for every problem. Sometimes the right action is to remove an alert that generates noise. Sometimes it’s to change an on-call rotation. Sometimes it’s to accept the risk and document it explicitly. Not every contributing factor needs a fix, but every one needs a decision.
The Review: Pressure-Testing the Findings
Before the post-mortem is considered complete, the draft must be reviewed by people who were not directly involved in the incident. This isn’t bureaucracy. Insiders develop blind spots. They accept constraints that an outsider would question. The reviewer’s job is to ask: Does this timeline make sense? Are there missing perspectives? Do the action items actually address the contributing factors, or are they just busywork? A good reviewer will push back on weak causal links and demand evidence for claims.
Schedule the review meeting with a strict timebox. The goal is not consensus—it’s accuracy and completeness. If the group cannot agree on a finding, document the disagreement in the report. A post-mortem that buries controversy is a post-mortem that will be ignored when the same incident happens again.
Publishing and the Living Document
A post-mortem that sits in a private wiki is a post-mortem that doesn’t exist. Publish it broadly. Make it discoverable. Tag it with the affected services, the failure modes, the detection methods. Over time, this builds an organizational memory that new engineers can search when they encounter similar symptoms. The post-mortem becomes a diagnostic tool, not just a historical record.
Revisit post-mortems periodically. Did the action items get completed? Did they have the intended effect? Did the incident recur despite the fixes? This follow-through is where most organizations fail. They treat the post-mortem meeting as the end of the process, when it should be the beginning of a remediation cycle. A “post-mortem review” six months later, checking on action item completion and incident recurrence, closes the loop and holds the organization accountable.
Common Failure Patterns in Post-Mortems
Recognize these anti-patterns and eliminate them from your process:
- The Blame-Shift Post-Mortem: The root cause is always “human error,” and the action item is always “retrain the human.” This is a sign that the organization doesn’t understand systems thinking.
- The Vague Post-Mortem: Filled with phrases like “improve monitoring” and “enhance communication.” If you can’t specify what will change, you haven’t learned anything.
- The Overstuffed Post-Mortem: Twenty action items, no owners, no deadlines. This is a wish list, not a plan. Prioritize the top three items that would have prevented or mitigated the incident, and assign them to specific people with specific dates.
- The Hero Narrative: The post-mortem focuses on how the team rallied to fix the problem, not on why the problem happened. Celebrating heroics incentivizes creating situations that require heroics.
Integrating Post-Mortems into Broader Risk Management
A single post-mortem is useful. A collection of post-mortems, analyzed for patterns, is transformative. Look across incidents for recurring contributing factors. Are the same services failing? The same teams? The same time of day? These patterns reveal systemic weaknesses that no individual post-mortem would surface. This is the connection between incident analysis and proactive risk reduction—a topic explored further in our article on failure pattern taxonomies.
Post-mortems also feed into pre-mortems and design reviews. When proposing a new system or a major change, review past post-mortems for similar architectures or failure modes. Ask: “How would this design fail in the ways we’ve seen before?” This turns post-mortem data into a forward-looking risk assessment tool.
FAQ: Post-Mortems That Actually Work
What’s the difference between a post-mortem and an incident review?
A post-mortem is a specific type of incident review focused on systemic learning. An incident review might be a quick operational debrief to restore service. A post-mortem is a deeper forensic analysis that asks not just “what happened” but “why did our controls fail to prevent or detect this, and what will we change to reduce the likelihood of recurrence?”
How long should a post-mortem take to complete?
For a significant incident—one with customer impact or data loss—allocate two to three hours for the initial timeline construction and contributing factor mapping, then another hour for the review meeting. The write-up should take no more than two hours. If it takes longer, you are likely over-analyzing or the incident scope is too large. For smaller incidents, a lightweight “post-mortem lite” process of 60-90 minutes total is appropriate.
Should we include vendors or external partners in our post-mortems?
Yes, if their systems or services were part of the failure chain. However, be prepared for them to be defensive. Share your timeline and contributing factor graph with them before the meeting. Frame the discussion as a joint problem-solving exercise, not a blame session. If they refuse to participate or share their own analysis, document that refusal—it is itself a contributing factor to your risk posture.
What if we can’t determine the cause?
Publish what you know, and explicitly state what you don’t know. A post-mortem that says “we were unable to determine the trigger due to insufficient logging” is valuable—it makes the logging gap visible and actionable. Do not invent a cause to satisfy the need for closure. Uncertainty is data. Treat it as such.
Closing the Loop
A post-mortem that produces lessons learned is a post-mortem that changes behavior. The measure of success is not the quality of the document but the reduction in incident recurrence and the speed of future detection. Track these metrics. If your post-mortems are well-written but your incident patterns don’t change, your process is performative. Stop writing and fix the process.