Organizational post-mortems are a ritual of failure. A project collapses, a release goes sideways, or a critical system blacks out under load, and someone schedules a meeting with a name that already implies the patient is dead. The term itself—post-mortem—sets a strange tone. It suggests finality, a forensic examination of a corpse. But in software and IT, the corpse is often still walking around, patched together with hotfixes and held upright by exhausted engineers. The real objective is not to bury the dead but to prevent the same organism from dying in exactly the same way six months later. This article is about how to run that examination so it produces lessons learned, not just a document that sits in a wiki until the next incident overwrites it.
Most post-mortems fail because they confuse documentation with understanding. They produce a timeline, a list of who did what, and a set of action items that sound responsible but change nothing. A real post-mortem is an act of organizational forensics. It treats the incident as a symptom of deeper structural conditions—brittle architecture, perverse incentives, information hiding, normalization of deviance. The goal is not to assign blame or even, primarily, to fix the immediate bug. The goal is to expose the systemic vulnerabilities that made the failure possible and, in many cases, inevitable.
Why Most Post-Mortems Produce Placebo Lessons
The standard post-mortem template is a trap. It asks for a timeline, a root cause, and a list of action items. This structure encourages linear thinking about complex failures. It produces a single “root cause”—often something like “human error” or “a misconfigured load balancer”—and a corresponding fix that addresses the symptom while leaving the disease untouched. The action items are typically assigned to the same people who were involved in the incident, who are already overworked, and who have no authority to change the organizational conditions that set them up to fail.
This approach is not just ineffective; it is actively harmful. It creates an illusion of control. Management sees a list of completed action items and believes the problem is solved. Engineers learn that the post-mortem process is a bureaucratic exercise to be endured, not a mechanism for real improvement. The next incident will have a different proximate cause but the same underlying pattern, and the cycle repeats.
A useful post-mortem starts from a different premise: complex systems fail in complex ways. The proximate cause is almost never the whole story. Behind every “operator error” is a system that made the error easy to commit and hard to detect. Behind every “unexpected edge case” is a design that didn’t account for the messy reality of production. The post-mortem’s job is to trace the failure back through the layers of the socio-technical system until it finds the structural conditions that can actually be changed.
Preconditions for an Honest Post-Mortem
You cannot run a useful post-mortem in an organization that punishes failure. If engineers believe that admitting a mistake will damage their career, they will not admit mistakes. They will frame the incident in terms of external factors, ambiguous requirements, or unforeseeable circumstances. The post-mortem will produce a list of action items that are safe, vague, and useless.
This is not a cultural nicety. It is a practical requirement. The information you need to understand a failure lives in the heads of the people who were involved. If they do not trust the process, they will not share that information. You will get a sanitized version of events that protects individuals but leaves the system vulnerable. The precondition for a real post-mortem is a blameless culture—not a culture without accountability, but one where accountability means honestly examining decisions in their full context, not scapegoating the person who happened to touch the keyboard last.
Blamelessness is often misunderstood. It does not mean ignoring negligence or malice. It means recognizing that in complex systems, individual actions are heavily constrained by the tools, processes, incentives, and information available at the time. The question is not “Who screwed up?” but “What conditions made this action seem reasonable to a competent person at that moment?”
Structuring the Post-Mortem for Actual Insight
The standard post-mortem template is a timeline followed by a root cause. This is backwards. The timeline is an input to the analysis, not the analysis itself. A better structure separates the descriptive work (what happened) from the analytical work (why it happened) and the prescriptive work (what we will do about it).
1. The Incident Narrative
Start with a plain-language description of what happened, written for someone who knows nothing about the system. Avoid jargon. Avoid assigning responsibility. Describe the impact: what users experienced, what services were affected, for how long. This section should be short—a paragraph or two—and should answer the question: “What was the business impact of this incident?”
2. The Timeline
The timeline is a factual record, not an analysis. It should include: when the incident was detected, by whom, and how; when key decisions were made; when mitigations were applied; when the incident was resolved. Each entry should be a dry statement of fact. No interpretation. No assignment of blame. The timeline is evidence, not verdict.
3. The How: Technical Root Cause
This is the narrowest part of the analysis. What specific technical condition triggered the incident? A null pointer exception? A database deadlock? A misconfigured firewall rule? This is the “how”—the mechanism of failure. It is necessary but not sufficient. Most post-mortems stop here, which is why they fail.
4. The Why: Systemic Contributors
This is where the real work begins. For each systemic contributor, ask: what made this failure possible? What made it hard to detect? What made it hard to respond to? Common categories include:
- Design and architecture: Were there single points of failure? Was the system’s behavior under stress well-understood? Were there assumptions about dependencies that turned out to be false?
- Testing and validation: Why didn’t existing tests catch this? Was the failure mode tested? Was the testing environment representative of production?
- Observability and alerting: Did we detect the problem quickly? Were the right metrics being monitored? Were alerts actionable, or were they noise that trained people to ignore them?
- Process and coordination: How was the incident response handled? Were roles clear? Was communication effective? Did the incident management process help or hinder?
- Human factors: What was the cognitive load on the operators? Were they fatigued? Were they working with incomplete information? Did the tooling support or obstruct their decision-making?
5. The Action Items
Action items should be specific, owned, and time-bound. But more importantly, they should address systemic contributors, not just the technical root cause. A good action item changes the system so that the entire class of failure becomes less likely, not just the exact scenario that occurred. For example, instead of “Fix the null pointer exception in module X,” consider “Add static analysis to the CI pipeline to catch null pointer dereferences across all modules.”
Action items should also be prioritized ruthlessly. Not every lesson needs to become an action item. Some lessons are worth recording but not worth the opportunity cost of implementing a fix. The post-mortem should distinguish between “things we will do” and “things we have learned and will apply to future design decisions.”
Running the Post-Mortem Meeting
The post-mortem meeting is not where the analysis happens. The analysis should be done beforehand, by a small group of people who were close to the incident, working from the timeline, chat logs, monitoring data, and their own recollections. The meeting is for review, challenge, and alignment. It brings in a broader group—including people who were not involved in the incident—to pressure-test the analysis and ensure that the lessons are valid.
The facilitator’s job is to keep the conversation focused on the system, not the individuals. When someone says “I should have checked the logs before restarting the service,” the facilitator redirects: “What made checking the logs difficult? Was the information you needed easy to find?” The goal is to move from self-blame to system critique.
Timebox the meeting. Two hours is usually enough. Longer than that and you are either re-litigating decisions that cannot be changed or trying to solve architectural problems that belong in a design review. The post-mortem identifies the problems; it does not need to solve them all in the room.
Writing the Post-Mortem Document
The document should be written for an audience that includes people who will join the organization a year from now. It should be self-contained, clear, and honest. Avoid euphemisms. If a decision was bad, say it was bad, and explain why it seemed reasonable at the time. If a system is fragile, say it is fragile. The document’s credibility depends on its willingness to name uncomfortable truths.
Store the document somewhere searchable. Link it from the incident tracking system, the runbooks, the architecture decision records. A post-mortem that nobody reads is a post-mortem that never happened. The organization should treat these documents as a growing body of knowledge about how the system actually behaves under stress—a counterweight to the optimistic assumptions baked into design docs.
From Lessons Documented to Lessons Learned
Documenting a lesson is not the same as learning it. Learning requires that the lesson change behavior. This means the post-mortem process must include a mechanism for verifying that action items are completed and, more importantly, that they had the intended effect. A “done” action item that didn’t reduce risk is a failure of the post-mortem process.
One effective practice is the “follow-up review”: three to six months after the post-mortem, revisit the incident and the action items. Did the changes actually make the system more resilient? Did a similar incident occur? If not, was it because of the changes, or because the triggering conditions simply haven’t recurred? This follow-up closes the loop and creates accountability for outcomes, not just outputs.
Another practice is the “incident pattern review”: periodically look across multiple post-mortems to identify recurring systemic contributors. If five different incidents all point to inadequate testing of edge cases, that is a signal that the testing strategy needs fundamental change, not just another action item on the next post-mortem.
Common Failure Modes of Post-Mortems
Even with good intentions, post-mortems can fail in predictable ways. Recognizing these patterns helps you avoid them.
The Blame-Shift Post-Mortem: The root cause is identified as a third-party service, a vendor bug, or an “act of God.” The action items are all external: “file a ticket with the vendor,” “request a feature from the cloud provider.” This is a way of abdicating responsibility. Even when an external dependency fails, the question should be: why were we vulnerable to that failure? What could we have done to degrade gracefully or fail over?
The Overcorrection Post-Mortem: The incident triggers a massive, expensive effort to prevent that exact scenario from ever happening again. This is often disproportionate to the actual risk. The post-mortem should assess the probability and impact of recurrence and recommend mitigations that are commensurate. Not every incident requires a multi-sprint remediation project.
The Vanity Post-Mortem: The document is written to make the team look good. It emphasizes how quickly the incident was resolved, how well everyone worked together, how the response was a “success.” It downplays the fact that the incident happened at all. This is a cultural failure. An organization that cannot admit its failures is an organization that will keep having them.
When Not to Run a Post-Mortem
Not every incident deserves a full post-mortem. The process has a cost: the time of the people involved, the cognitive load of reliving a stressful event, the risk of generating busywork action items. Reserve the full process for incidents that meet a threshold of severity—significant user impact, data loss, security implications, or near-misses that could have been catastrophic. For minor incidents, a lightweight “incident review” that captures a few bullet points in a shared log may be sufficient.
Also, do not run a post-mortem when the organization is still in crisis mode. If the incident is part of an ongoing pattern of instability, the priority is to stabilize the system, not to analyze it. A post-mortem conducted under pressure will be rushed and superficial. Wait until the system is stable and the team has had time to recover.
Building an Organizational Memory
The ultimate purpose of post-mortems is to build an organizational memory that outlasts individual engineers. People leave. Teams reorganize. Code is rewritten. But the patterns of failure in complex systems are remarkably persistent. They recur because they are shaped by the same structural forces: tight coupling, insufficient observability, misaligned incentives, production environments that differ from test environments in ways nobody documented.
A well-maintained body of post-mortems is a strategic asset. It tells you where your system is brittle. It tells you what your engineers are struggling with. It tells you where to invest your reliability budget. It is also a training resource for new engineers, who can read through past incidents and learn how the system actually behaves, as opposed to how it was designed to behave.
This is the difference between a post-mortem that produces lessons documented and one that produces lessons learned. The former fills a wiki page. The latter changes how the organization thinks about failure. It makes the system not just more reliable but more understandable—and an understandable system is one that people can operate with confidence, even when things go wrong.
Frequently Asked Questions
What is the difference between a post-mortem and an incident review?
A post-mortem is a structured, in-depth analysis of a significant incident, focusing on systemic contributors and producing documented lessons and action items. An incident review is a lighter-weight process for minor incidents, often just a brief discussion and a few notes in a shared log. The distinction matters because applying the full post-mortem process to every small outage creates process fatigue and dilutes the value of the practice.
How do you run a post-mortem when the root cause is unclear?
Unclear root causes are common in complex systems where failures emerge from interactions between components, not from a single broken part. In these cases, resist the urge to force a single root cause. Instead, document the contributing factors and their interactions. Use techniques like the “Five Whys” cautiously—they can be useful for drilling into a specific causal chain but can also oversimplify. A better approach is to map the multiple contributing factors and look for patterns across them.
Should the same people who were involved in the incident write the post-mortem?
Yes, but not alone. The people who were directly involved have the most detailed knowledge of what happened and why decisions were made. However, they also have blind spots and may be too close to the event to see systemic patterns. The post-mortem should be drafted by the involved parties and then reviewed by people who were not involved—ideally someone with a broader view of the system architecture and someone with a focus on organizational processes. This combination produces an analysis that is both accurate and contextual.
How do you measure whether a post-mortem process is working?
Do not measure the number of post-mortems completed or the percentage of action items closed. Those are vanity metrics. Measure whether similar incidents recur. Measure the time to detect and resolve incidents over time. Measure whether the action items from past post-mortems actually changed the system’s behavior under stress. A post-mortem process is working when the organization’s failure patterns shift—when the same class of incident stops happening, or when it happens but the impact is smaller and the recovery is faster.