Observability, See the Chaos Before It Sees You

If your system's in chaos and you're flying blind, you're not just unlucky — you're unprotected. The dashboards are usually there. The question you actually need to ask usually isn't.

"Flying blind" gets used as though it means having no instrumentation. In practice, teams in that position almost always have plenty: several dashboards per service, alerts in the hundreds, a bill to match. What they do not have is the ability to answer the question in front of them, right now, in less time than it would take to guess.

Coverage is not volume

Telemetry volume and telemetry usefulness come apart because they are produced by different activities.

Volume comes from instrumenting what is easy to instrument, which tends to be whatever the framework exposes for free: request counts, durations, status codes, resource utilisation, per service, forever. It is real data and it is genuinely useful for the failures it describes.

Usefulness comes from instrumenting what the business logic is doing, which nothing exposes for free because only your code knows it exists. Whether the reconciliation completed. Which of the three fallback paths was taken. Why this particular request took the slow branch. That instrumentation only appears if somebody decides to add it, and the decision is easy to defer indefinitely because nothing is visibly missing until an incident makes it so.

The result is estates with excellent coverage of the transport layer and almost none of the domain. Every graph agrees the service is healthy — requests are being served, latency is fine, the pods are up — while the thing the service exists to do has silently stopped happening.

Work backwards from real incidents

The most reliable way to fix this costs nothing and is almost never done: take the last five incidents and write down the questions people actually asked, in the order they asked them.

The list will not look like a monitoring strategy. It will be things like: did this start before or after the config change? Is it all customers or one? Is the retry storm a cause or a symptom? What was different about the requests that failed? Which version were those pods running?

Then mark each one — answerable from what you had, or not. The unanswerable ones are your gap, stated concretely, prioritised by the fact that somebody genuinely needed each of them during a real outage. That list is worth more than any maturity assessment, because it is derived from your system and your failures rather than from a general model of what a mature organisation looks like.

It also tends to be shorter and cheaper than expected. Usually a handful of attributes that were not attached to events, one boundary where request context is dropped, and one dataset whose retention expires before the pattern becomes visible.

Alerts nobody acts on

The other half of flying blind is not missing data but drowned attention.

An alert that fires regularly and is routinely acknowledged without action is not neutral. It is actively harmful, because it trains the response. People learn, correctly, that this page does not require them to do anything — and that learning is a general habit, not a specific one. It transfers to the page that did require something.

The test worth applying to every alert is simple: when this fires, what does the person receiving it do? If the honest answer is "acknowledge it and carry on", it is not an alert. It might be a useful dashboard panel, or a ticket, or nothing. What it should not be is something that wakes a human, because you are spending a scarce resource — attention at three in the morning — on information that does not change any decision.

Deleting alerts feels like reducing safety. It is the opposite: an alert set small enough to be read in full is worth more than one large enough that nobody reads any of it carefully.

Fast enough to explore

There is a threshold effect in query speed that gets underestimated.

If asking a question of your telemetry takes thirty seconds, investigation is a conversation with the data: you follow a hunch, it is wrong, you follow another, and you have tried a dozen framings in ten minutes. If it takes four minutes, that stops. People form a hypothesis, ask one carefully chosen question, and reason from the answer — which means the investigation is now limited by whoever is best at guessing, and the guesses go unchallenged because checking them is expensive.

This is why the incident-time query path deserves to be treated as a latency-sensitive service in its own right. Not the retention, not the completeness — the speed of an interactive question. It is what decides whether the tooling supports exploration or merely confirmation.

The three people problem

The last symptom is organisational: every team has two or three people who can actually find the answer, and everyone else waits for them.

This is usually read as a knowledge-sharing problem and addressed with documentation, which does not work, because what those people have is not a set of facts. It is a working model of where the data lives and which queries are worth trying, built by doing it repeatedly.

What transfers that is exposure: having someone else drive during a low-severity incident while the expert watches, rather than the other way round. It is slower on the day. It is the only thing that reliably widens the set of people who can see what is happening — which is what observability is actually for, given that the alternative is a system only legible to whoever happens to be awake.