TOPIC 02 — RISK
Error Budgets
The single best idea SRE gave the industry: turn “how reliable should we be?” from an endless argument into a number both dev and ops agree to spend.
Why budgets, not promises
Traditionally, dev teams are rewarded for shipping and ops teams for stability — so they fight. The error budget dissolves the fight with arithmetic. Once the business has chosen an SLO of 99.9%, it has also chosen to tolerate 0.1% unreliability. That 0.1% is not shameful failure; it's a resource. Spend it deliberately:
- Risky launches and big migrations
- Chaos experiments and failover drills
- Skipping a canary for an urgent fix
- Planned maintenance that briefly degrades service
An outage stops being “a thing that must never happen” and becomes “an expected expense against a budget everyone agreed to.” That flip is what lets teams ship quickly without gambling the service.
The arithmetic
For a 30-day window, the budget in minutes is (1 − SLO) × 43,200:
| SLO | Budget | Downtime / 30 days | Feels like |
|---|---|---|---|
| 99% | 1% | 7.2 hours | An afternoon outage every month is fine |
| 99.9% | 0.1% | 43.2 minutes | One bad deploy a month, if you catch it fast |
| 99.95% | 0.05% | 21.6 minutes | Every incident needs a practiced response |
| 99.99% | 0.01% | 4.3 minutes | Humans are too slow — recovery must be automatic |
| 99.999% | 0.001% | 26 seconds | Telecom territory. Eye-wateringly expensive |
Partial outages spend the budget proportionally: if 10% of requests fail for 100 minutes, you've burned the equivalent of 10 minutes of full downtime.
The error budget policy
A budget only works if running out has consequences agreed in advance, in writing, signed by engineering leadership. A typical policy:
- Budget > 50%: normal operations. Ship freely.
- Budget < 50% and burning fast: risky launches need SRE sign-off; postmortem action items get priority.
- Budget exhausted: feature freeze. All engineering effort goes to reliability work until the rolling window recovers.
A policy nobody enforces is worse than no policy — it teaches the org that reliability targets are decorative. If leadership won't honour the freeze, renegotiate the SLO downward instead. An honest 99.5% beats a fictional 99.95%.
Burn rate: the early-warning system
Burn rate is how fast you're consuming budget relative to the sustainable pace. Burn rate 1 means you'll spend exactly your budget by the end of the window. Burn rate 14.4 means you'll torch a 30-day budget in 50 hours.
The SRE Workbook's recommended alerting strategy pairs a fast and a slow signal, each checked over two windows to avoid flapping:
| Alert | Burn rate | Windows | Budget consumed | Action |
|---|---|---|---|---|
| Page | 14.4× | 1 h + 5 min | 2% in 1 hour | Wake someone up now |
| Page | 6× | 6 h + 30 min | 5% in 6 hours | Respond within the hour |
| Ticket | 1× | 3 days + 6 h | 10% in 3 days | Investigate this week |
This is why error budgets change alerting culture: you page on “the user experience is degrading fast enough to matter,” never on “a CPU is busy.”
Common pitfalls
- Having a budget but no policy — a bank account where overdrafts are ignored.
- Resetting the budget on the 1st of the month. Rolling windows prevent “it's the 28th, ship everything” behaviour.
- Letting one dependency eat the whole budget. If a vendor burns 80% of your budget, that's an engineering (or contract) problem to fix.
- Punishing the team that spends budget. Spending it is the point; hoarding it means you're shipping too slowly.