TOPIC 06 — CULTURE
Blameless Postmortems
An outage you paid for and didn't learn from is a bill without the goods. The postmortem is how SRE converts pain into permanent system improvement.
Why blameless — and why it's hard
When something breaks, every organisation instinctively asks “who?” Blameless culture replaces it with “how?” — not out of niceness, but because blame is operationally expensive. The moment engineers fear consequences, they stop reporting near-misses, shade the timeline, and hide the exact details you need to prevent the repeat. You trade the illusion of accountability for a team that lies to you politely.
If your analysis concludes “human error,” you stopped too early. Ask why the system allowed it: Why did one typo reach every region at once? Why was there no dry-run mode? Why did rollback take 40 minutes? Humans err at a fairly constant rate — systems decide the blast radius.
Blameless does not mean consequence-free. Recklessness and repeated negligence are management problems handled separately. The postmortem's scope is the system, and inside that scope honesty must be safe.
When to write one
Agree the triggers in advance so it's never a judgment call under pressure:
- Any user-visible downtime or degradation beyond the SLO threshold
- Any data loss, however small
- On-call intervention beyond a set effort (e.g. an incident that needed a second responder)
- A resolution that took longer than your target
- Anyone — anyone — asks for one. Cheap to grant, expensive to refuse.
The anatomy of a good postmortem
| Section | What it contains |
|---|---|
| Summary | Three sentences: what broke, for how long, who felt it. Readable by a VP in 20 seconds. |
| Impact | Numbers: users affected, requests failed, error budget spent, revenue if known. |
| Timeline | Timestamped: first cause, detection, declaration, each mitigation attempt, resolution. Detection→mitigation gap is your key metric. |
| Root cause(s) | The full causal chain, not the last domino. Most incidents have several contributing causes. |
| What went well | Genuinely important — you want to keep doing these things under pressure. |
| What went poorly | Including luck. “It was Tuesday 2pm” is not a detection strategy. |
| Action items | Each with an owner, a deadline, and a tracking ticket. The only section that changes the future. |
A miniature example
Summary: Checkout API returned 5xx for 23 min (14:02–14:25 UTC). ~4% of purchase attempts failed. 31% of monthly error budget consumed.
Trigger: Deploy 2025-341 changed the DB connection string; the pool pointed at a read-only replica.
Root causes: (1) config change not covered by staging smoke tests; (2) canary skipped — flagged “config-only”; (3) rollback required manual approval, adding 9 min.
Action items: AI-1: canary ALL deploys incl. config-only (owner: platform, ticket OPS-2211) · AI-2: staging smoke test exercises a write path (owner: checkout, OPS-2212) · AI-3: auto-approve rollbacks during active SEV (owner: release eng, OPS-2213)
Notice what's absent: no names, no “the engineer should have been more careful.” Every action item hardens the system.
Common pitfalls
- Blame with synonyms. “The change wasn't sufficiently reviewed” is “Dave screwed up” in a suit. Fix the review system.
- Action items that die in a doc. Untracked items are wishes; put them in the same backlog as features.
- Twenty action items. Nobody does twenty. Pick the three that matter most.
- Postmortem theatre — a beautiful document, an unchanged system, the same outage in six months.
- Hoarding them. Google publishes internally company-wide; the repeat you prevent is usually on another team.