The scheduler is not a spare node

The node disappears, the pod does not

The common assumption is that when a node fails, the Kubernetes control plane automatically moves the affected workload to a healthy node, rendering the incident self-healing. This view conflates the creation of a replacement object with the successful execution of that object. The Deployment controller detects the missing pod and creates a new one in the desired state, but this is merely the first step in a chain of physical prerequisites that must be satisfied before the container starts. The scheduler attempts to find a node with sufficient allocatable resources, but it cannot conjure capacity that does not exist. If the cluster was running at full utilisation before the failure, the sudden loss of a node removes both the workload and its associated compute resources. The new pod enters a pending state, waiting for a node that has enough free CPU and memory to meet its requests. The condition that stops this process is simple: the absence of unallocated capacity on any remaining node.

This distinction is critical because it shifts the burden of reliability from the control plane to the cluster’s physical configuration. The scheduler is a matching engine, not a resource generator. It matches pod requests to node allocatable resources. If the total sum of running pod requests equals the total allocatable resources of the cluster, the loss of a node creates a deficit that no amount of controller logic can resolve. The pod remains pending indefinitely, or until a human intervenes to add nodes or evict other workloads. The system is not "handling" the failure in any meaningful operational sense; it is merely recording the intent to handle it. The reality is that the system is now operating with fewer replicas than intended, and the new pod is a promise that may never be fulfilled.

The pending pod is a ghost

A pending pod is not a running pod, and treating it as such is a dangerous operational error. The Service object continues to point to the endpoints defined by the pod’s IP address, but since the pod is not running, it has no IP. The Service’s endpoint slice becomes empty for that replica, effectively reducing the capacity of the service. Traffic that would have gone to the missing node is now distributed among the remaining healthy nodes, potentially overloading them. The condition that stops the recovery here is the lack of immediate traffic shedding or load balancing adjustment. The Service does not know that the pod is pending; it only knows that the endpoint is gone. The remaining pods must absorb the load, which may cause them to hit their resource limits and trigger their own evictions, creating a cascading failure.

This is where the concept of "handled" becomes misleading. The control plane has performed its job of reconciling the desired state with the observed state, but the observed state is one of reduced capacity. The user sees a pending pod and assumes the system is working, while the service is actually degraded. The mechanism of failure is the gap between the logical representation of the workload and the physical reality of the compute resources. The pending pod is a ghost in the machine, a symbol of a resource that is owed but not paid for. It does not serve traffic, it does not process requests, and it does not contribute to the system’s resilience. It is a liability, not an asset, until the physical conditions for its execution are met.

Eviction thresholds and the pressure valve

When a node is under pressure, the kubelet initiates eviction to protect the node itself. This process is governed by thresholds that determine when resources are considered insufficient. The kubelet monitors memory, disk, and PID pressure, and when these metrics exceed the defined thresholds, it begins to evict pods based on their priority and quality of service class. The condition that stops the recovery here is the timing of the eviction. If the node is already under pressure before the failure, the kubelet may evict pods that are still running, further reducing the cluster’s capacity. This creates a scenario where the cluster is losing capacity on two fronts: the failed node and the evicted pods on the remaining nodes.

The eviction process is not instantaneous and is subject to a grace period. During this period, the pods are marked for eviction but are still running, consuming resources. This delay can be significant, especially if the node is under severe memory pressure. The mechanism of failure is the lag between the resource exhaustion and the actual removal of the pods. During this lag, the cluster is in a state of reduced capacity, and the scheduler is trying to place new pods on nodes that are themselves under pressure. This can lead to a situation where the scheduler cannot find a suitable node because all available nodes are either full or in the process of evicting their own pods. The result is a prolonged period of pending pods and degraded service.

Disruption budgets and the illusion of safety

Pod Disruption Budgets (PDBs) are designed to prevent voluntary disruptions from taking down a service. They specify the minimum number of replicas that must be available at any given time. However, PDBs do not protect against involuntary disruptions such as node failures. The condition that stops the recovery here is the interaction between PDBs and the scheduler. When a node fails, the pods on that node are considered involuntarily disrupted, and the PDB does not block their removal. The scheduler is free to reschedule them, but only if there is capacity. The PDB does not create capacity; it only limits the number of pods that can be down due to voluntary actions.

This distinction is often misunderstood, leading teams to believe that PDBs provide a guarantee of availability. In reality, PDBs are a safety net for maintenance windows, not a shield against hardware failure. The mechanism of failure is the false sense of security provided by PDBs. Teams may configure PDBs with tight limits, assuming that this will ensure high availability. However, if the cluster lacks the headroom to reschedule pods after a node failure, the PDB is irrelevant. The pods will be down, and the PDB will not prevent this. The only way to ensure that rescheduling is possible is to maintain sufficient headroom in the cluster, a practice that is often neglected in favour of cost optimisation. The scheduler is not a substitute for capacity planning; it is a tool that operates within the constraints of the physical infrastructure.