“Eat the chaos” – Automate the Breakdowns

Let's get real: the internet doesn't care if your app goes down. The failure modes you have not exercised are the ones that will be discovered by your users, at the least convenient hour available.

Chaos engineering has an image problem, and the name is most of it. It sounds like breaking production for sport, which makes it an easy thing to refuse and a hard thing to get budget for. The actual practice is close to the opposite: it is a refusal to let the first execution of a recovery path be a real outage.

Designed, documented, never run

Every serious system contains recovery mechanisms that have never been used in anger. The failover that would promote the replica. The circuit breaker that would shed load. The cache that would serve stale data if the origin went away. The runbook step that says "drain the node".

These are not hypothetical in the sense of being unimplemented — they exist, someone wrote them, they were reviewed. They are hypothetical in the sense that nobody knows whether they work, because the conditions that would invoke them have never occurred. The belief that they work is inherited from the design, not from evidence.

And that belief is load-bearing. Capacity plans, availability targets and on-call expectations all rest on the assumption that when the primary fails, the secondary takes over. If that assumption is wrong, it is wrong at the worst possible moment and in a way that surprises everybody, because the plan said otherwise.

Chaos engineering is the practice of converting those beliefs into observations, on purpose, at a time you chose.

Start from a belief you actually hold

The version of this that fails is the one that starts with the tool. Someone installs a fault injector, picks the most dramatic thing it can do, kills a lot of pods, and either nothing happens — in which case the exercise proved nothing anyone doubted — or something breaks badly enough that the practice acquires a reputation and never gets a second run.

The version that works starts with a sentence somebody would defend. If the recommendations service stops responding, the product page still renders. If one availability zone goes away, we lose no writes. If the cache is empty, the database survives the resulting load.

Each of those is falsifiable, which is the only property that matters. You now have an experiment with a defined outcome, a reason to run it, and a result that is worth something either way. If the belief holds, you have converted an assumption into a fact. If it does not, you have found a genuine defect in conditions where finding it is cheap.

The best candidates are usually the beliefs held most confidently by people who have not tested them recently, and the ones inherited from a system that has since been rearchitected.

Have the abort ready before you start

The safety practices here are not complicated and they are not optional.

Know how to stop before you start. Not the theory of stopping — the actual command, tested, in a terminal that is already open. An experiment you cannot halt promptly is not an experiment; it is an outage you scheduled.

Decide the abort condition in advance, in the same way you would for a canary. Which metric, which value, how long. Deciding live, while watching something you deliberately broke, is how a controlled exercise turns into an incident.

Run it in working hours, with the people who own the system present and aware. Overnight chaos experiments combine the two things you least want combined: a deliberately induced fault and the smallest, most tired response capacity you have. The point is not to test the on-call rota's endurance. It is to learn something about the system with the maximum number of competent people watching.

Start small enough that the worst case is boring. One instance, one dependency, a fraction of traffic. Scope grows once the small version has stopped finding things.

Fix what you find before scaling up

The most common way this practice quietly dies is that it becomes a source of findings that nobody has capacity to act on.

The first few experiments surface real problems — a timeout inherited from a library default that is far longer than anything upstream will tolerate, a retry policy that multiplies load precisely when the callee is struggling, a fallback path that has never been exercised and turns out to reference a decommissioned service. Those get written down. Then the next experiment runs, because running experiments is the visible activity, and it produces more findings.

Within a few months there is a backlog of known weaknesses and a well-established practice of discovering more. The organisation is now paying for the discovery and receiving none of the benefit, and someone reasonable will eventually ask why.

The discipline is to treat each finding as blocking the next escalation. Fix it, then re-run the same experiment to confirm the fix, then increase the scope. This is slower, produces fewer impressive-sounding results, and is the only version that leaves the system more robust than it found it.

The habits it builds

The secondary benefit is often larger than the primary one, and it is about people rather than systems.

Running these exercises regularly means the team practises responding to failure in conditions where nothing is actually at stake. Somebody has to find the right dashboard, work out what is happening, decide, and communicate — with the enormous advantage of knowing that the worst case is calling it off. That is rehearsal, and the difference between a team that has rehearsed and one that has only read the runbook is visible in the first ten minutes of a real incident.

It also has a way of surfacing the gap between the architecture people describe and the one that exists. Someone will say the dependency is optional; the experiment will show that removing it takes down checkout. Nobody was lying. It was optional when it was introduced, and something changed, and the description did not.