Short analytical pieces on the mechanics of failure in distributed systems: what the received wisdom gets wrong, and the conditions under which it stops being true.
Failure Modes are written from published postmortems, incident write-ups and engineering documentation, which are listed at the foot of every piece. Nothing in this section has been run, measured or operated here — where a number, a threshold or a result appears, it belongs to the source it is credited to.
The argument for production-only chaos engineering is seductive because it feels rigorous. If you want to know if your system survives a database failover, the only way to be sure, the logic goes, is to actually fail over the database while real users are…
The standard pitch for distributed tracing is that it reveals where time goes. Engineers instrument every service, watch the waterfall diagram, and assume they hold a complete map of the request lifecycle. This assumption holds only while the trace context…
The standard operating procedure for a failed release is to redeploy the previous container image. This practice rests on a fundamental assumption: that the application state is isolated from the infrastructure state. For code, this is largely true. A binary…
The standard definition of a canary release, as outlined in the Google SRE Workbook, describes it as a partial and time-limited deployment of a change. The logic is sound: expose a small subset of production to the change, observe the metrics, and decide…
The standard architectural argument for multi-region deployment rests on a simple premise: if one location fails, the other remains functional. This logic treats geographic distance as a sufficient condition for fault isolation. However, redundancy only…
The blameless postmortem is widely celebrated as a cultural achievement. It removes the fear of punishment, allowing engineers to describe exactly what they did without self-censorship. This openness is genuinely valuable; it surfaces the messy reality of…
The standard advice for building resilient clients is straightforward: always set a timeout on every remote call. The gRPC documentation explicitly states that without an explicit deadline, a client may wait for a response effectively forever. This guidance…
The standard justification for placing a cache in front of a database is straightforward: it absorbs repeated reads, reducing the number of queries that reach the primary storage layer. On a normal day, this works as advertised. The application layer checks…
The common assumption is that when a node fails, the Kubernetes control plane automatically moves the affected workload to a healthy node, rendering the incident self-healing. This view conflates the creation of a replacement object with the successful…
A liveness probe is a binary question: is this process alive and responsive? It is not a question about correctness, nor about the state of the database, nor about the availability of the message queue. The Kubernetes documentation defines the liveness probe…
The common belief is that a retry policy acts as a safety net, smoothing over transient glitches in a distributed system. This view treats each service call as an isolated event, where a failed attempt is simply a momentary hiccup that a second try can…