The assignment
Drop into an unfamiliar Kubernetes codebase and make releases safe: a way to ship a new version to a slice of real traffic, watch it, and either promote it to everyone or pull it fast — no 20-step checklist that only one person knows.
The key decision
There are two common ways to split traffic for a canary, and they are not equivalent:
- Replica-ratio splitting: run N stable pods and 1 canary pod and let the load balancer spread requests across them. Simple, but the traffic split is now coupled to pod count: to send 5% to canary you need the replica math to work out, and autoscaling silently changes your split.
- Per-request splitting: the ingress routes each request to stable or canary, independent of how many pods exist.
I chose per-request splitting using nginx-ingress canary annotations, with a Kustomize stable/canary overlay. The split is a number you set, not an emergent property of the cluster, so it stays correct when pods scale up or down.
What I built
- A Kustomize base plus
stableandcanaryoverlays, so both tracks share one source of truth and diverge only where they must. - A Makefile wrapping the whole flow: one command to deploy a canary, one to promote it to stable, one to roll back. No memorized kubectl incantations.
- An incident runbook for the failure I most expected: a 503 storm right after promotion. It encodes one rule: capture logs before you roll back, then
rollout undo. Roll back first and you destroy the evidence that tells you why it broke.
Promote
One command. Canary becomes stable; no hand-edited manifests.
Roll back
One command. After logs are captured, not before.
Traffic split
Decoupled from pod count. Survives autoscaling.
Operator load
A runbook, not tribal knowledge.
The hardening plan I left behind
Shipping the mechanism is half the job; the other half is making it boring and safe to run repeatedly:
- A smoke-test gate that must pass before a promote is allowed.
- A gradual ramp (10% → 25% → 50% → 100%) instead of a single cutover.
- Auto-rollback on a failed rollout, so a bad deploy doesn't depend on a human noticing.
Forward-deployed work is judged on one thing: can the team operate what you built after you leave? That's why the runbook mattered as much as the rollout.
Kubernetesnginx-ingressKustomize overlaysMakefile release flowIncident runbook