The Night the Pipeline Stood Still

It doesn't happen loudly. No dramatic outage. No flashing red dashboards. Just a quiet moment when deployments stop, and everything downstream of them quietly starts to queue.

Delivery pipelines have an unusual property among production systems: when they stop working, nothing reports an outage. A service that stops serving requests generates errors, and errors generate pages. A pipeline that stops running generates nothing at all, because not running is indistinguishable, from the outside, from having nothing to do.

The metric nobody graphs

Most teams can tell you their deployment success rate. Very few can tell you, without going and looking, how long it has been since a change reached production.

Those are different questions, and the second is the one that matters. A pipeline with a hundred per cent success rate over the last fortnight sounds excellent until you learn it has run twice. Success rate measures the runs that happened; it says nothing about the runs that did not.

Time since last successful production deploy is the graph that catches this, and it is almost never on a wall. It is also unusually easy to reason about: everyone on the team knows roughly what normal looks like, so a number three times larger than normal needs no threshold tuning to be obviously wrong.

Why it stops without failing

The interesting stoppages are rarely a broken build. A broken build is loud, someone owns it, and it gets fixed before lunch.

What actually stops delivery for weeks tends to be softer. A flaky test that fails about a third of the time, so people re-run rather than investigate, and the pipeline becomes a slot machine. A credential that expired on a stage nobody watches. A queue for a shared environment. An approval step whose approver is on leave. A dependency on a system that is itself mid-incident. None of these is a failure in the sense the pipeline understands. Each of them is a reason today's change did not ship, and none of them will produce an alert.

The flaky test deserves particular attention, because it does more damage than its failure rate suggests. A test that fails intermittently trains everyone to treat a red pipeline as noise. That training then applies to the genuine regression, which is caught by the suite exactly as designed and re-run into a pass by someone who has stopped reading. The suite did its job. The habit undid it.

The cost arrives later

Nothing bad happens on the day delivery stops. That is precisely the problem: the bill is deferred, and it is paid by someone else.

While the pipeline is quiet, merged changes accumulate. Each is small and was reviewed on its own terms. Together they become a release whose combined behaviour nobody has reasoned about, because nobody was ever asked to. When that release finally ships, it ships as one event.

This is where the arithmetic turns against you. If something breaks, the set of candidate causes is now everything merged since the last deploy. Bisecting is slower, reverting is coarser — you can no longer undo the change that caused the problem without also undoing thirty that did not — and the fix carries its own risk, because it is also large. The size of the release and the difficulty of recovering from it move together, and both are consequences of the quiet fortnight nobody noticed.

There is a second cost, harder to see. Long gaps between deployments make each deployment feel significant, and significance makes people careful in the wrong way: more approvals, more coordination, larger batches to justify the ceremony. That makes the gaps longer. The loop is stable and it runs in the wrong direction.

Treating it as a production system

The change in stance that helps most is deciding the pipeline is a production system with users, rather than a tool the team happens to use. Its users are the engineers whose work it delivers, and its outage is their inability to ship.

That reframing carries three practical consequences.

It gets alerting — on delivery stopping, not only on delivery failing. Something should notice when the last successful production deploy passes whatever counts as unusual for your team, and it should notice without anyone remembering to check.

It gets ownership. Not a rota of whoever last touched it, but the same named on-call that owns the service it deploys. A pipeline everybody uses and nobody owns is the default state, and it is how a two-day stoppage becomes a two-week one.

It gets its dependencies treated as real. The registry, the artefact store, the credential that signs the release: these are dependencies of production, because production cannot change without them. They deserve the monitoring and expiry tracking you would give anything else in that category.

Keeping the change small

All of the above is easier when a single change is small enough that shipping it late is inconvenient rather than dangerous.

Small changes are not a virtue in themselves. They are a way of keeping the blast radius of any one deployment bounded and the cause of any one regression obvious. They also make a stopped pipeline visible faster, because the queue behind it grows in units people can count.

And they make the rollback path something you exercise rather than something you document. A rollback of one small change is routine. A rollback of a fortnight's accumulated work is a project — which is why, at that size, teams almost always choose to fix forward instead. That choice is usually made at three in the morning, by someone who did not create the conditions that made it the only option.