Blue-Green Deployments: Your Safety Net

If you've ever deployed new code only to watch your app crash and burn live, this is for you. Blue-green gives you somewhere to switch back to — as long as the data underneath agrees to come with you.

The mechanism is simple enough to describe in a sentence. Run two complete production environments. One serves traffic; the other is idle. Deploy to the idle one, verify it, move traffic across. If the new one misbehaves, move traffic back. The switch is a routing change, so it happens in seconds rather than in however long a redeploy takes.

That last property is the entire value, and it is worth being precise about why.

Cheap reversal changes the decision

When rolling back is expensive, the decision to roll back gets deferred. Someone has to be reasonably confident the new version is genuinely bad before triggering fifteen minutes of disruption to undo it — and while they are becoming confident, the bad version is serving users. In most incidents of this shape, the time spent deciding is longer than the time spent recovering.

When rolling back is a routing change, that calculation disappears. You can reverse on suspicion. If you were wrong and the new version was fine, you have lost very little and can switch forward again once you understand the graph. The ability to act on incomplete information without penalty is the thing blue-green is actually buying, and it is a bigger deal than the reduced downtime.

Where the illusion lives

Now the part that gets skipped. Two application environments, one database.

Almost nobody runs two production databases, because keeping them consistent is a harder problem than the one blue-green was brought in to solve. So the shared datastore sits underneath both environments, and the "complete isolation" the diagram shows is isolation of the stateless half only.

This means the safety net catches exactly one class of fault: bugs in code that do not durably change state. A null dereference on a new endpoint, a broken template, a misconfigured client — for those, switching back genuinely restores the previous behaviour.

For anything that has written to the shared database, it does not. Green ran for eleven minutes, wrote several thousand rows in a new shape, and you have now switched back to blue, which does not understand that shape. The traffic is back on the old code. The data is not.

The worst version is a migration that ran as part of the deploy. Blue is now pointed at a schema it was never written against. The rollback completed successfully and the system is still broken, which is a particularly demoralising place to be at two in the morning, because the one lever you were relying on has already been pulled.

Expand, migrate, contract

The discipline that makes the net actually hold is to never have a moment where old and new code cannot both run against the current schema.

Adding a column is safe if the old code ignores it. So: add the column, ship code that writes both the old field and the new one while still reading the old one. That is one deployment, and it is reversible, because the old version is entirely happy with a table that has an extra column it does not use.

Backfill the existing rows separately, as a job rather than as part of a deploy. Then ship a version that reads the new field. Still reversible: the previous version reads the old field, which is still being written.

Only once that has been running long enough for you to be confident — a full cycle, including whatever weekly batch work touches this table — do you stop writing the old field and, later still, drop it.

Three or four deployments instead of one, spread over days, and an intermediate state where the schema carries redundant fields. In exchange, every single step can be undone by moving traffic, which is the property the elegant one-shot migration does not have at any point.

Decide the abort condition first

The other common failure is procedural. The switch happens, someone watches a dashboard, and the question "is this bad enough to go back?" gets answered live, by whoever is watching, under time pressure.

That question should be answered before the switch. Write down what would make you abort — which metric, past which value, sustained for how long — while nobody is under pressure and the discussion can be calm. Then the person watching is checking a condition rather than exercising judgement, and they can act immediately without needing to build a case.

Two practical details go with this. Connection draining needs to be measured rather than assumed: long-lived connections, websockets and in-flight background work do not respect the routing change, and "we switched at 14:02" is not the same as "the old version stopped doing things at 14:02". And the idle environment has to be held long enough to be useful — tearing blue down ten minutes after the switch means the safety net exists only for the failures that show up immediately, which are not the interesting ones.

What it is worth

Blue-green is a good technique. It is also frequently sold as a general answer to deployment risk, and it is not one: it is a fast, reversible traffic switch, which solves the stateless half of the problem completely and the stateful half not at all.

Knowing which half you are in before you deploy is most of the skill. A change that touches no persisted state can be shipped with real confidence. A change that alters the shape of stored data needs the slower, uglier, multi-step path — and the moment you find yourself reasoning about whether the rollback would leave the data readable, you are in the second category, whatever the deployment diagram says.