My first serious production incident, I made it worse. Checkout was failing, I was fairly sure it was a bad deploy, so I rolled back. It kept failing. Then I restarted the app servers. Still failing. Then I noticed the database CPU at 100% — a migration I had run an hour earlier was still rebuilding an index, and my restarts had thrown away every connection and made the queue worse.
Twenty minutes of confident action, all of it wrong, because I had diagnosed in the first ninety seconds and spent the rest defending that guess.
Being good at incidents is mostly about not doing that. It is a procedural skill more than a technical one, and it can be learned deliberately rather than by accumulating scars.
Stabilise Before You Understand
The instinct is to find the cause. The correct first move is to stop the bleeding, even if you do not yet know why it is bleeding.
If a deploy went out in the last hour, roll it back. Not because you have proven it is the cause — because it is the cheapest reversible action available and it removes a variable. If a feature is failing, flag it off. If one customer's traffic is drowning everyone, rate limit them.
Understanding can happen afterwards, with the system healthy and nobody watching the clock. Root cause analysis during an outage is how twenty minutes becomes ninety.
The one exception: never take an action you cannot undo. Restarting is usually safe. Deleting data, force-pushing, or truncating a queue during a panic is how a bad hour becomes a bad quarter.
The First Five Minutes
A fixed sequence, because judgement is worse under adrenaline than a checklist is.
Say something in the channel. "Looking at checkout errors, investigating." Ten seconds, and it stops five people starting parallel uncoordinated investigations.
Establish blast radius. All users or some? One endpoint or everything? One region? This is the question that most changes what you do next and it is regularly skipped.
Check what changed. Deploys, feature flags, config changes, migrations, certificate expiry, and third-party status pages. Most incidents are caused by a change, and the change is usually recent.
Look at the four signals. Error rate, latency, traffic, saturation. Where the graph turned is often more informative than what it says.
Name a coordinator if more than two people are involved. Someone who is explicitly not debugging — they track what has been tried, keep the timeline, and handle updates. Without this, three people restart the same service and nobody knows what state anything is in.
Debugging Under Pressure
Two habits that would have saved my twenty minutes.
Write down what you believe and what would disprove it. "I think it is the deploy. If so, the rollback fixes it." When the rollback did not fix it, that should have killed the hypothesis immediately. Instead I kept acting as though it were still true.
Narrate every action in the channel with a timestamp. "14:32 rolled back to build 4471." This does three things: it keeps others informed, it stops duplicate actions, and it produces the timeline for the post-mortem for free. It also slows you down slightly, which under adrenaline is a benefit.
And prefer bisection over inspiration. Is it the app or the database? The database. Is it reads or writes? Writes. Is it all writes or one table? Halving the search space repeatedly gets there faster than a clever theory, and it works even when you are tired.
What Makes On-Call Sustainable
Most on-call rotations are bad not because incidents are frequent but because the setup around them is careless.
Every page must be actionable. If the responder can only acknowledge and go back to sleep, that alert should not page. Move it to a dashboard. Non-actionable pages are how people learn to ignore their phone.
Alert on symptoms, not causes. "Checkout error rate above 2%" pages. "CPU above 80%" does not — high CPU with everything working fine is not an incident. Symptom alerts catch problems you did not predict; cause alerts only catch the ones you did.
A runbook per alert. Not documentation — a link in the alert itself with what it means, what to check first, and how to mitigate. Written by whoever handled it last time.
Whoever is paged can fix it. Production access, deploy rights, the ability to flip a flag. On-call without authority is just being awake for someone else's incident.
Compensate it and cap it. One week in four minimum, paid or given back as time. Treating on-call as free is how you lose senior people quietly.
Time to fix alert noise during the rotation. The person on-call is best placed to fix what woke them, and they should be expected to, rather than pushing tickets nobody picks up.
Post-Mortems That Are Worth Writing
Most are filed and never read. The ones that produce change share a few properties.
Blameless, and mean it. If the write-up names a person, the next person hides their mistake and you lose the information. "A developer ran a migration during peak hours" is not the cause; "our process allowed an unreviewed migration to run during peak hours with no way to see its progress" is.
A timeline with real timestamps. When it started, when it was detected, when it was mitigated, when it was resolved. The gap between started and detected is usually the most interesting number in the whole document and the one nobody looks at.
Ask why detection was slow, not just why it broke. Half the value of an incident is discovering that you found out from a customer.
At most three actions, each with an owner and a date. A list of fourteen improvements is a list nobody does. Three that ship are worth more.
Include what went well. Not for morale — because the things that worked are the ones you want to protect when someone later proposes simplifying them away.
The Preparation That Actually Pays
Three things, in order of value, and all of them have to exist before the incident:
Fast, boring rollback. If reverting takes forty minutes, every incident is at least forty minutes. Immutable image tags, backwards-compatible migrations, and a deploy tool that can go backwards as easily as forwards.
Distributed tracing. Nearly impossible to add during an incident, because you need the data from before it started. This is the single highest-value thing to set up while nothing is wrong.
Practise. Break something in staging deliberately and have someone who did not break it find it. Half an hour a month, and it surfaces the runbook that is wrong and the dashboard nobody can find far more cheaply than production does.
My migration incident produced exactly one action item: show long-running database operations on the main dashboard. It took an afternoon, and the next time something similar happened we saw it in the first minute instead of the twentieth.
