TOPIC 03 — AUTOMATION
Toil Reduction
Toil is the tax manual work levies on every growing system. SRE's rule: measure it, cap it, and engineer it out of existence.
The six signatures of toil
Google's definition gives toil six identifying traits. The more boxes a task ticks, the more toil-like it is:
| Trait | The test |
|---|---|
| Manual | A human has to run the script, click the button, type the command. |
| Repetitive | You've done it before. You'll do it again. |
| Automatable | A machine could do it as well as you. No human judgment genuinely required. |
| Tactical | Reactive and interrupt-driven, not strategy-driven. |
| No enduring value | The service is in the same state after you finish. Nothing improved. |
| Scales with growth | 2× users → 2× the work. The killer trait. |
Manually restarting a stuck worker every morning: toil. Writing the health-check that restarts it automatically: engineering. Attending the weekly planning meeting: neither — that's overhead.
Why the 50% cap exists
Without a cap, toil obeys a grim ratchet: growth creates toil, toil consumes engineering time, less engineering means more toil per unit of growth. Follow that loop far enough and your “engineering” team is an ops team with a fancy title — and your best people quit, because career growth built on repetitive tickets is no growth at all.
The 50% number is a ceiling, not a target. It guarantees each SRE spends at least half their time on work that permanently reduces future toil: automation, better tooling, and pushing fixes upstream into the service itself.
The toil audit: an exercise you can run this week
- Log it. For two weeks, everyone tags time spent on operational work: what, how long, how often.
- Score it. For each recurring task: minutes per occurrence × occurrences per month = monthly cost.
- Sort by cost. The top three items usually account for most of the pain.
- Compare cost vs automation effort. A task costing 10 hrs/month that takes 20 hrs to automate pays for itself in two months.
- Schedule the fix like a feature. Toil reduction that isn't on the roadmap doesn't happen.
The automation ladder
Not all automation is equal. Climb this ladder rung by rung:
- Document it — a runbook turns tribal knowledge into a repeatable procedure.
- Script it — the runbook becomes one command a human runs.
- Gate it — the system runs the script; a human approves.
- Automate it — the system detects the condition and fixes itself; humans read a log entry.
- Eliminate it — the best rung: change the design so the failure mode no longer exists.
Automation has failure modes of its own: a bad script does damage at machine speed, and skills atrophy when humans never practice the manual path. Automate the decision and the audit trail, and rehearse the manual fallback occasionally.
Common pitfalls
- Heroism. The engineer who “just handles it” every week is hiding a systemic problem and burning themselves out.
- Measuring nothing. If you don't track toil, the 50% cap is a vibe, not a policy.
- Automating the wrong layer — a beautiful script for a task that shouldn't exist at all.
- Counting on-call incident response as toil. It's operational load, tracked separately; conflating them muddies both numbers.