The Cache Is Not a Load Shedder

The Illusion of Reduced Load

The standard justification for placing a cache in front of a database is straightforward: it absorbs repeated reads, reducing the number of queries that reach the primary storage layer. On a normal day, this works as advertised. The application layer checks the cache first, and only on a miss does it query the database. This reduces the average load on the origin, allowing teams to provision fewer database instances or smaller compute nodes. The system appears efficient, and the capacity model reflects this efficiency. The database is sized to handle the residual load, not the total request rate. This is where the first problem begins. The capacity model is no longer based on the true demand of the application but on a derivative of that demand, filtered through the hit rate of the cache. If the hit rate is high, the origin sees very little traffic. If the hit rate drops, the origin sees more. But the infrastructure has not changed. It is still sized for the low-traffic scenario. The cache is not merely a performance optimization; it is a load-shaping mechanism that fundamentally alters the relationship between user demand and backend capacity.

This shaping creates a false sense of security. The system is stable because the cache is doing its job. But that stability is conditional. It depends on the cache being warm, populated, and functioning correctly. The moment that condition fails, the assumption that the database can handle the full request rate is exposed. The database was never designed to handle that load in the first place. It was designed to handle the load that the cache did not filter out. This is a critical distinction that is often overlooked in capacity planning. The cache is not a buffer against load spikes; it is a filter that reduces the steady-state load. When the filter fails, the system does not just slow down; it faces a step change in demand that it is not equipped to absorb.

The Sudden Shift to Origin Traffic

When a cache fails, the failure is rarely gradual. It is often a sudden event: a node crash, a network partition, a memory exhaustion, or a configuration error that causes the cache to return empty results. In any of these cases, the application layer receives a miss for every request. The result is a sudden, full transfer of the request rate to the database. This is not a spike in the traditional sense, where demand increases beyond the norm. This is a shift in the baseline demand. The database, which was previously handling a fraction of the total requests, is now handling the entire volume. If the database was provisioned to handle only ten percent of the total traffic, it is now facing a tenfold increase in load.

The physical mechanism of this overload is straightforward. The database connection pool fills up. Queries queue up. Latency increases. As latency increases, client-side timeouts trigger. Clients, expecting a response within a certain window, abandon their requests and retry. This retry storm adds even more load to the already saturated database. The system is now in a state of overload, but it is not simply slow; it is failing. The database is spending its resources processing queries that will time out before they complete. Goodput, the amount of useful work done, drops to zero. The system is up, but it is not working. This is the metastable state described by Bronson et al. in their paper on metastable failures in distributed systems. The trigger was the cache failure, but the system remains in the bad state even after the cache is restored. Why? Because the retry storm and the queued requests create a feedback loop that sustains the overload.

Why the System Gets Stuck

The reason the system does not recover immediately after the cache is restored is that the overload has created a self-sustaining cycle. The database is still processing the backlog of queries that accumulated during the cache failure. These queries are still occupying connections and CPU cycles. New requests, now hitting the warm cache, are being served quickly, but they are competing for the same resources as the stale, queued requests. The database cannot shed the old load quickly enough. It is stuck in a state where it is processing work that is no longer relevant, while new, relevant work is waiting. This is the definition of a metastable failure: a bad state that persists even when the original trigger is removed.

The sustaining feedback loop is the key to understanding this failure mode. The loop is not just the retry storm; it is the combination of the retry storm, the queued requests, and the lack of a mechanism to shed the old load. The system is not broken in the sense that it has a bug or a hardware failure. It is broken in the sense that its control logic is not able to break the cycle. The cache is warm, but the database is still overloaded. The system will remain in this state until an external intervention occurs, such as a manual restart of the database, a reduction in traffic, or a change in the application logic to prioritize new requests over old ones. Without such intervention, the system will continue to process the backlog indefinitely, or until it crashes under the weight of its own inefficiency.

Breaking the Feedback Loop

The solution to this problem is not to remove the cache, but to design the system to handle the transition between cached and uncached states. This requires a change in the control logic of the system. The system must be able to detect the shift in load and respond by shedding the old load. This can be done through several mechanisms. One is to implement a timeout on the database queries that is shorter than the client-side timeout. This ensures that the database does not spend resources on queries that will be abandoned by the client. Another is to implement a priority queue, where new requests are processed before old ones. This ensures that the system is always working on the most relevant data. A third is to implement a circuit breaker that stops sending requests to the database if the error rate exceeds a certain threshold. This gives the database time to recover from the overload.

The key insight is that the cache is not a solution to the problem of overload; it is a contributor to it. The cache creates a state where the system is vulnerable to a sudden shift in load. The solution is to design the system to be robust to that shift. This means treating the cache failure as a normal operating condition, not an exception. It means building in the mechanisms to shed load, prioritize requests, and recover from overload. It means understanding that the cache is not a magic bullet that makes the system faster; it is a component that changes the system's behavior in ways that must be accounted for in the design. The goal is not to eliminate the metastable state, but to make it a transient one, a state that the system can recover from without external intervention. This is the difference between a system that is fast and a system that is resilient.