Circuit Breakers: The Panic Button Your Microservices Need

If your system was a spaceship, a circuit breaker would be the big red lever that saves you from total meltdown. What it saves is not the dependency — it is everything still queued behind it.

That distinction is the whole point and it is routinely lost. A circuit breaker does nothing whatsoever for the failing service. It does not repair it, retry it into health, or route around it. What it does is stop your service from being destroyed by the act of politely waiting.

How waiting kills the caller

The mechanism deserves spelling out, because the failure is counter-intuitive: the service that goes down is frequently not the one that broke.

Your service has a finite amount of concurrency — threads, connections, event-loop slots, whatever the runtime calls it. Each in-flight request consumes one unit for as long as it lasts.

Under normal conditions the downstream call returns in a few milliseconds, so each unit is occupied briefly and the pool comfortably serves a high rate of requests. Now the dependency gets slow — not down, which would be easy, but slow. Calls that took five milliseconds take five seconds. Each in-flight request now holds its unit a thousand times longer.

The arrival rate has not changed. The service rate has collapsed. The pool fills, and every subsequent request queues for a resource that is not being released. At that point your service stops responding to everything — including requests that have nothing to do with the slow dependency, including health checks, including the endpoints that were working perfectly.

And now your callers are in exactly the same position, for exactly the same reason. The failure walks up the call graph, and by the time anyone is looking, the user-visible symptom is several hops away from the cause.

Fail fast, then probe

The breaker interrupts that chain by watching the failure rate of one dependency and, past a threshold, refusing to make the call at all. Requests to that dependency fail immediately instead of occupying a slot for five seconds.

This is worth being clear-eyed about: those requests still fail. Nobody is being served better. What has changed is that they fail in microseconds rather than seconds, so they stop consuming the resource that everything else in your service needs. The parts of your service that do not touch the failing dependency continue to work, which is the difference between degraded and down.

After an interval the breaker lets a small number of requests through to find out whether the dependency has recovered. This half-open probing is not a detail — it is what stops the breaker becoming a permanent outage of its own. It is also where implementations most often disappoint: a breaker that reopens by dumping the full retained load onto a service that has just come back will close it again immediately, and the system will oscillate.

Timeouts are the prerequisite

Circuit breakers get attention because they have a memorable name. Timeouts do more work and get almost none.

A breaker needs to be able to tell that calls are failing. If a call has no timeout, or has one longer than your caller's patience, it never reports a failure — it just sits there. The breaker never trips, because from its perspective nothing has gone wrong yet. Meanwhile the pool has filled.

So the ordering is: every remote call gets an explicit timeout, and that timeout is shorter than the deadline of whatever is calling you. This has to hold transitively across the chain, which is why deadline propagation — passing the remaining budget along with the request — is worth more than it looks. Without it, each service picks a timeout that seems reasonable in isolation, and the chain adds up to something far longer than the user's browser will wait for.

The related trap is retries. A retry policy configured per hop compounds multiplicatively down the chain: three services each retrying three times is up to twenty-seven attempts for one user request. Every one of those lands on a dependency that is already struggling, which is precisely the condition under which retrying is most harmful. If you retry, do it at one layer, with jitter, and with a budget that caps retries as a fraction of total traffic rather than per request.

Isolation, so one cannot take the rest

A single shared pool means any one dependency can exhaust the capacity all of them need. Giving each downstream its own bounded allocation contains that: the slow one saturates its own limit and the rest of the service carries on.

The cost is a little unused capacity and more configuration to get wrong. The benefit is that the blast radius of one bad dependency is one dependency. For a service that talks to a handful of downstreams of clearly different importance — a payment processor and a recommendations engine should not be able to hurt each other — that trade is usually worth taking.

A breaker with no fallback just fails sooner

The last question is the one most often left until the breaker is already in production: what does the caller do when the breaker is open?

If the answer is "return an error", you have improved things — the error is fast, and your service is still alive for everything else — but the user experience is unchanged. That is sometimes correct. You cannot invent a payment authorisation.

Often something better exists and needs deciding in advance. Serve the cached version and say it may be stale. Omit the recommendations panel and render the rest of the page. Accept the write into a queue and reconcile later. Fall back to a simpler implementation that does not need the dependency at all.

Each of those is a product decision as much as a technical one, and it is much easier to make calmly than during an incident. The graceful degradation is the actual deliverable; the breaker is only the thing that makes it possible to reach.