TOPIC 05 — OPERATIONS
Incident Response
Incidents are inevitable; chaos is optional. SRE borrows from emergency services: clear roles, practiced procedures, and one rule above all — mitigate first, explain later.
What counts as an incident?
An incident is any unplanned event that degrades (or threatens to degrade) service for users and needs a coordinated response. The keyword is coordinated: a bug one engineer quietly fixes is a bad day; the moment two people must work together under time pressure, you want incident structure. Most teams grade by severity:
| Level | Meaning | Example | Response |
|---|---|---|---|
| SEV1 | Critical — major user-facing outage or data loss | Checkout down globally | Page immediately, all hands, exec comms, status page |
| SEV2 | Major — significant degradation, workaround exists | Search slow for 20% of users | Page on-call, incident channel, hourly updates |
| SEV3 | Minor — limited impact, not time-critical | One region's batch jobs delayed | Ticket, handle in business hours |
Declaring an incident is cheap; a late response is not. If you're debating whether it's an incident, it's an incident. You can always close it in ten minutes with “false alarm” — that's a win, not an embarrassment.
Roles: the Incident Command System
Borrowed from firefighting, ICS solves the two classic failure modes: everyone debugging and nobody deciding, or five people making conflicting changes at once.
| Role | Owns | Does NOT do |
|---|---|---|
| Incident Commander (IC) | Decisions, priorities, delegation, keeping the big picture | Touch keyboards. The IC who starts debugging has abandoned command. |
| Operations Lead | Hands-on mitigation — the only one changing production | Field questions from stakeholders |
| Communications Lead | Status page, exec updates, customer comms on a fixed cadence | Guess. Comms states only what's confirmed. |
| Scribe | Timestamped log of actions and decisions — gold for the postmortem | Editorialise |
In a three-person team the same human wears several hats — the point isn't headcount, it's that each hat's job is explicit and someone owns it.
The lifecycle: mitigate before you understand
- Detect — ideally your monitoring, not a customer tweet. (Track which one it was; it's a revealing metric.)
- Triage — how bad, how wide, getting worse? Set severity, declare, open a channel.
- Mitigate — the counterintuitive heart of SRE response: make it stop hurting before you know why. Roll back the deploy, shift traffic, flip the flag, add capacity. Understanding can wait; users can't.
- Resolve — with pressure off, find and fix the actual cause. Verify with the metrics that caught it.
- Learn — every SEV1/SEV2 gets a blameless postmortem. The incident you don't learn from is rescheduled, not resolved.
“What changed?” is the highest-value question in any incident — most outages follow a change. If a deploy is in the blast window, roll it back first and check your theory afterwards. A wrong rollback costs minutes; a wrong theory costs the whole night.
Sustainable on-call
On-call is where burnout starts, so SRE treats its health as an engineering requirement, not an HR nicety:
- Rotations big enough to breathe — 6–8 people per rotation; being on-call one week in six is sustainable, one in two is not.
- A load ceiling — Google's guideline: at most ~2 incidents per 12-hour shift. More means the service needs engineering work, not tougher humans.
- Compensation — paid or time-in-lieu. Unpaid on-call is an outage-shaped resignation letter.
- Runbooks at the point of need — every alert links to the doc that explains it. 3am is not the time for archaeology.
- Practice — game days and “Wheel of Misfortune” drills, so the first SEV1 isn't the first rehearsal.
Common pitfalls
- Root-causing while users bleed. Mitigation and diagnosis are separate phases; do them in the right order.
- The hero IC who also debugs, communicates, and types — and drops all four balls.
- Silent incident channels. No update for an hour reads as “nobody's on it,” even when everyone is.
- Punishing the person who declares. If declaring feels career-risky, incidents get reported late — permanently.