The Health Check That Kills the Fleet

The Liveness Trap

A liveness probe is a binary question: is this process alive and responsive? It is not a question about correctness, nor about the state of the database, nor about the availability of the message queue. The Kubernetes documentation defines the liveness probe as a mechanism to detect when a container is in a state where it cannot perform its intended function, prompting the runtime to restart it. The critical implication is the action taken: a failure triggers a restart. If the probe is too deep, the restart is not a cure; it is a symptom of a misunderstanding.

Consider a service that depends on a shared database. If the liveness probe executes a simple query to verify the connection, the probe is coupled to the database. When the database experiences a transient network partition or a brief latency spike, the probe times out. The orchestrator marks the instance as not alive and kills it. The new instance starts, attempts to connect to the same struggling database, fails the probe, and is killed again. This is a restart loop. The service is not dead; it is merely degraded. By treating a dependency failure as a process failure, the system converts a partial outage into a total loss of capacity for that instance.

The mechanism is positive feedback. The restart increases the load on the orchestrator and the remaining healthy instances. If the database is struggling due to load, the restarts add connection churn, worsening the database’s state. The probe, intended to heal the system, has become the engine of its collapse. The SRE book describes cascading failures as growing over time due to positive feedback, where a portion of the system failing increases the probability that other portions fail. A deep liveness probe is a textbook example of this dynamic.

The Readiness Paradox

Readiness probes serve a different purpose: they determine whether a service is ready to accept traffic. The Kubernetes documentation states that a readiness probe failure means the pod should be removed from the service’s load balancer endpoints, but the pod itself remains running. This distinction is vital. If a readiness probe is too shallow, the service accepts traffic it cannot handle, leading to error spikes. But if the readiness probe is too deep, it creates a different, often more subtle, failure mode.

Imagine a microservice that depends on three other services. If the readiness probe checks the health of all three, and one of them goes down, the instance reports itself as not ready. The load balancer removes it from rotation. This is correct behavior for that specific dependency. However, if the dependency is shared across many instances, and the failure is transient or localized, every instance may report not ready simultaneously. The load balancer now has no healthy endpoints. The service is effectively down, even though the primary process is running fine.

The problem is not the depth of the check, but the coupling. A readiness probe should reflect the service’s ability to serve its primary contract. If the service can serve a degraded response when a secondary dependency is down, the readiness probe should not fail. If it fails, the system is designed to be all-or-nothing. In a distributed system, all-or-nothing readiness is a fragility. The probe is asking, "Are all my dependencies perfect?" instead of "Can I handle a request?" The result is that a single downstream blip removes all upstream capacity, amplifying the failure rather than containing it.

What a Check May Safely Test

So what is the safe boundary? A liveness probe must test only the process itself. It should verify that the event loop is running, that the HTTP server is listening, and that the process is not deadlocked. It should not touch external state. If the process is alive but the database is down, the liveness probe should pass. The process is alive; it is just waiting. Let it wait. Let it queue requests. Let it return 503 errors. But do not kill it. Killing it does not fix the database; it only removes a resource that might be useful once the database recovers.

A readiness probe should test the service’s ability to handle traffic, but it must be designed with degradation in mind. If a dependency is critical to the service’s core function, the probe should fail. But if the dependency is optional, the probe should pass. The key is to define the service contract precisely. What does "ready" mean? Does it mean "can handle all features" or "can handle the primary feature"? The answer determines the probe’s depth.

The general principle is that a health check is a statement of intent. It tells the orchestrator what the system considers a failure. If the statement is too broad, the orchestrator takes drastic action based on incomplete information. The check must be narrow enough to reflect the specific failure mode it is designed to catch, and no broader. A liveness check that touches the database is not a liveness check; it is a dependency check masquerading as a liveness check. A readiness check that fails on any dependency is not a readiness check; it is a perfect-state check. In a distributed system, perfect state is a myth. Design for the imperfect, and your health checks will reflect that reality.

The Cost of Ambiguity

The danger of conflating these probes is not just theoretical; it is a common operational failure. Teams often write a single health endpoint that checks everything, then use it for both liveness and readiness. This is a design error. The liveness probe should be the most shallow check possible. The readiness probe should be the most specific check possible for the service’s contract. When these are mixed, the system becomes unpredictable. A database hiccup can trigger a restart loop (liveness failure) or a traffic blackout (readiness failure), depending on which probe is configured to check the database.

The solution is separation. Keep the liveness probe simple: ping the process. Keep the readiness probe focused: check the critical dependencies for the primary function. If a dependency is optional, do not check it in the readiness probe. If a dependency is critical, check it, but be aware that a failure will remove the instance from rotation. This is a trade-off. It is better to remove an instance from rotation than to send it traffic it cannot handle, but it is worse to remove all instances from rotation because of a transient blip.

The health check is not a magic bullet. It is a tool. Like any tool, it can be used to build or to break. If you use it to ask the wrong question, you will get the wrong answer. And in a distributed system, the wrong answer can be catastrophic. Ask the right question, and the system will be more resilient. Ask the wrong one, and you will have built a mechanism for your own failure.