The System Will Fail at Dawn

There is a moment before every incident. A quiet second when dashboards still glow green, the queue is still draining, and the thing that will take the system down has already happened.

The word "sudden" does a lot of unearned work in outage write-ups. Very few production failures are sudden in any useful sense. What is sudden is the arrival of load against a condition that has been building for weeks — which is why so many incidents start in the morning, and why the postmortem so often finds that the graph had been bending for a month.

Trends and thresholds

Alerting is overwhelmingly built around thresholds. Something crosses a line, a page fires, someone responds. This works beautifully for step changes: a dependency goes down, error rate jumps, everyone knows within a minute.

It works badly for the failures that are shaped like a ramp. A file handle leak, a slowly growing table with an unindexed query against it, a cache whose hit rate is degrading a fraction of a per cent a day, a disk filling at a few gigabytes a week. Every one of these is perfectly visible in the data the whole time. None of them trips anything, because at no point during the accumulation is the current value abnormal. The alert fires when the resource is exhausted, which is to say at the end, when the useful response window has closed.

The fix is to alert on where the line is going rather than where it is. Projected time to exhaustion is a more useful number than current utilisation, and it has the pleasant property of being actionable: a disk at seventy per cent tells you very little, while a disk that will be full on Thursday tells you exactly what to do and when. It also fires while the response can still be leisurely, which is the entire point.

Why dawn

The timing is not superstition. It follows from the shape of the two curves involved.

Overnight, load is low. Whatever is accumulating — memory, connections, queue depth, lock contention — accumulates against a system with plenty of headroom, so it stays invisible. Meanwhile the quiet hours are when batch work runs: backups, reports, index rebuilds, retention jobs, all the things scheduled for when nobody is looking. Those consume some of the same headroom.

Then the morning arrives, and demand goes from its overnight floor to something several times higher in the space of an hour. The accumulated condition and the daily peak meet. The system had been operating well within its limits all night, and it is now operating outside them, and nothing in between generated a signal.

This has a direct consequence for load testing. A test at average throughput tells you about a condition your system is rarely in. The interesting question is what happens at the morning ramp, from a system that has been up for eleven days rather than freshly restarted — because a freshly restarted system does not have the condition you are trying to reproduce.

The restart that hides the evidence

Which brings us to the most effective way to make this class of failure invisible: restart things regularly and treat the resulting stability as health.

A restart resets accumulated state. If your deployment cadence is such that every process is replaced every few days, no leak ever gets long enough to matter, and you will never see it. That is fine, right up until a code freeze, a holiday period, or a long-running instance in a corner of the estate that does not get replaced on the same schedule. Then the failure appears, with no recent change to blame it on, in the worst possible conditions for diagnosis.

The habit worth building is to treat "restarting fixes it" as a description of a defect rather than as a resolution. It is genuinely useful information — it narrows the cause to per-process accumulated state — and it should open an investigation rather than close a ticket. An incident that ends with a restart and no explanation has not ended. It has been deferred to a date nobody chose.

Things with expiry dates

The other reliable source of morning failures is anything with an expiry date and no owner.

Certificates are the obvious case, and mostly handled these days by automation. The ones that get you are the ones outside it: a certificate pinned in a mobile client, an internal CA nobody remembers issuing, a partner's endpoint, a signing key for artefacts, a cloud credential with a rotation policy that was set up once by someone who has left. Each is a scheduled outage that has already been scheduled, by someone, some time ago, for a date they did not think about.

The remedy is boring and works: everything with an expiry has an inventory entry, an owner who is a current employee, and an alert that fires with enough runway to renew it during working hours. The inventory is the hard part, because the items that will actually bite are precisely the ones nobody thought to write down.

Reading the quiet second

What all of this comes to is a habit rather than a tool: looking at the slow-moving numbers on purpose, on a schedule, when nothing is wrong.

Nobody does this during an incident, because during an incident you are looking at what changed in the last ten minutes. Nobody does it during feature work either, because nothing is prompting it. So it has to be deliberate — a recurring half hour with the graphs that move over months rather than minutes, asking one question: which of these is heading somewhere it should not be allowed to arrive?

Half of the answers will be nothing. Some of the rest will be the incident you would otherwise have had at six on a Tuesday morning, found at a time and in a state of mind where fixing it is cheap.