TOPIC 07 — DELIVERY
Release Engineering
Most outages walk in through the front door: a change you shipped on purpose. Release engineering makes deploys so boring they stop being news.
The paradox: ship more often, break less
Intuition says fewer releases mean fewer outages. Data (see the DORA research) says the opposite. A quarterly release contains three months of entangled changes — when it breaks, which of 400 commits did it? A daily release contains a day's worth: small blast radius, obvious suspect, trivial rollback. Frequency is a forcing function for safety: you can't ship daily without automation, and automation is where safety actually lives.
The four principles (Google's version)
Self-service
Teams release their own software. No release-engineer bottleneck, no ticket queue — the platform makes the safe path the easy path.
High velocity
Release often, in small increments. Some Google teams build hourly and deploy from whichever build passes.
Hermetic builds
Same source → same artifact, on any machine, any day. Pinned compilers and dependencies; no “works on the build box.”
Enforced policy
Code review, tests, and signed artifacts are gates in the pipeline, not suggestions in a wiki.
Progressive delivery: the risk dial
Never show a change to 100% of users in one step. The standard ladder:
| Stage | Exposure | You're watching for |
|---|---|---|
| CI + staging | 0 users | Tests, integration, smoke checks on a production-like environment |
| Canary | ~1% of traffic | Error rate, latency, and saturation vs the old version, side by side |
| Percentage rollout | 5% → 25% → 50% | SLI regressions at each step; automatic halt on threshold breach |
| Full rollout | 100% | Business metrics — and the burn rate of your error budget |
A bug that would burn your whole monthly error budget in an hour at 100% exposure burns ~1% of it during a one-hour, 1% canary. The canary isn't bureaucracy — it's a 100× discount on the cost of being wrong.
Feature flags: deploy ≠ launch
Flags separate shipping code from exposing behaviour. Code rides to production dark, then lights up for 1% of users, then 10%, with an off-switch that works in seconds — faster than any rollback. The discipline they demand: delete stale flags, or accumulate a config file where 2ⁿ combinations have never been tested together.
Rollback first, diagnose second
When a release goes bad, the winning move is almost always revert now, understand tomorrow. That requires engineering done in advance:
- One-command rollback, tested regularly — an untested rollback path is a rumour, not a capability.
- Backwards/forwards-compatible schema changes — the classic rollback-killer is a DB migration the old code can't read. Expand → migrate → contract, over several releases.
- Version-N/N−1 compatibility — old and new must coexist during every rollout anyway; keep that property and rollback is free.
“Fix forward” under incident pressure means writing new code, at speed, angry, with no canary. That's how one outage becomes two. Fix forward only when rollback is genuinely impossible — then go fix that.
Common pitfalls
- The Friday-freeze mindset. If deploys scare you on Friday, they're unsafe on Tuesday too — fear is a symptom, cadence isn't the disease.
- Exempting “config-only” changes from the pipeline. Config changes cause a huge share of major outages; they deserve canaries too.
- Staging that resembles production the way a postcard resembles a city.
- Canaries judged by eyeball. Define the comparison metrics and abort thresholds before the rollout, and let automation halt it.