Deploy at Your Own Risk

Shipping fast feels like winning — until the system disagrees. The teams that ship fastest are rarely the ones taking the most risk; they are the ones who made the consequences of being wrong small enough to survive.

That distinction gets lost in the argument about deployment frequency, which is usually conducted as though speed and safety were opposite ends of one dial. They are not on the same dial. Deploy rate is one number. The time it takes to notice and undo a bad deployment is another. Raising the first without raising the second is the actual risk, and it is entirely possible to do both at once — which is why the same teams tend to score well on both.

Two different questions

"Is this change safe?" and "can we survive this change being wrong?" feel like the same question and are answered by completely different work.

The first is answered by review, tests, types, staging. All of it is worth doing and all of it has a ceiling, because every one of those mechanisms tests the change against conditions somebody thought of. The failures that matter in production are disproportionately the ones nobody thought of: a load pattern nobody modelled, a data shape nobody has in the fixtures, an interaction with a dependency that was fine in isolation.

The second question is answered by reversibility. How long between the change landing and someone knowing it is wrong? How long between knowing and it being undone? What does the system do to users in the interval? Those are engineering properties, they can be measured, and improving them protects you against failures nobody predicted — which is the category that actually causes incidents.

A team that has invested only in the first question gets steadily better at catching the failures it already understands.

Deploying is not releasing

The single change that does most for reversibility is separating the act of putting code on a machine from the act of exposing behaviour to users.

When those two things are the same event, undoing a bad release means a rebuild, a redeploy, and however long that takes — during which the bad behaviour is still live and someone is making a judgement call under pressure about whether it is bad enough to justify the disruption. That judgement is where most of the damage happens. Nobody wants to trigger a full rollback for something that might be recoverable, so they wait, and the waiting is the outage.

When they are separate, the new code ships dark and the behaviour is turned on afterwards, for a subset, by changing a value. Reversal is that same value going back. It takes seconds, it requires no build, and — critically — it is cheap enough that nobody has to be sure before doing it. Being able to reverse on suspicion rather than on proof is the property you are buying.

This is not free. Flags accumulate, and a codebase carrying two hundred of them is carrying two hundred untested combinations. They need an expiry discipline: a flag is a temporary state with an owner and a removal date, or it is technical debt with a configuration interface.

Comparing rather than hoping

The other half is exposing the new path to a slice of traffic and comparing it against the old one while both are running.

What makes this valuable is not the small blast radius, though that helps. It is that you get a control group. Absolute numbers during a deployment are hard to read — error rates move for reasons unrelated to your change, traffic shifts, a dependency has a bad minute. A comparison between two versions serving the same population at the same moment removes almost all of that noise, and it turns "does this look normal?" into a question with an answer.

It also has to be automatic to be worth much. A canary that requires a human to stare at a graph for twenty minutes will be watched carefully for the first month and then approved reflexively, because that is what humans do with a task that is almost always fine. The comparison and the abort should be the machine's job.

The part that will not roll back

All of this works for code. None of it works for state, and state is where the genuinely bad days come from.

Once a migration has run and the new version has written data in a shape the old version cannot read, "roll back" is not available at any speed. The traffic switch is instant and useless; the data has already moved. This is the failure mode that turns a ten-minute incident into an all-day one, and it is not solved by better deployment tooling, because the deployment tooling is not what broke.

The discipline that addresses it is unglamorous: split every schema change so that at every intermediate point, both the old and the new version of the application can run against the database as it currently is. Add the new column and start writing to it, while still reading the old one. Backfill. Move reads. Only then, after a full cycle where nothing has gone wrong, remove the old column.

It is more steps and more calendar time than doing it in one migration, and it means the schema is temporarily uglier than the design you had in mind. What you get for that is that every one of those steps is individually reversible, which is the property the single elegant migration does not have.

Practise on a Tuesday

A rollback path that has only ever been described is a hypothesis. The first time it runs should not be the first time it matters.

Rolling something back deliberately, in working hours, with the people who own it watching, is a cheap way to find out that the previous image was garbage-collected, or that the flag has a dependency on a config change that did not revert, or that the procedure names a system that was decommissioned. All of those are much better discovered on an ordinary afternoon than at the point where they are standing between you and the end of an incident.