When the Signals Speak: Beyond Monitoring in DevOps

Monitoring tells you something is wrong. Observability tells you why. The difference only matters on the day the failure is one nobody wrote a check for — which, unhelpfully, is most of the days that turn into incidents worth remembering.

The distinction has been repeated often enough to become a slogan, and slogans get bought rather than understood. It is worth being precise about what the actual difference is, because it is not the volume of data and it is not the number of pillars.

Questions decided in advance

A monitoring system is a set of questions somebody chose to ask, evaluated continuously. Is the error rate above two per cent? Is the queue longer than a thousand? Is the disk over eighty per cent full? Each check exists because a person anticipated a failure and encoded it.

This makes monitoring a record of the failures you already know about. That is not a criticism — most incidents are repeats, and catching them automatically is enormously valuable. But it has a structural consequence: the checks are derived from history, so the coverage is best for failures you have already had and worst for the ones you have not.

Observability is about the other category. It is the property that you can ask a question you did not think of in advance, about a running system, and get an answer without deploying code to find out. The test is not how much telemetry you emit. It is whether a novel question can be answered from what you already have.

That framing is useful because it is falsifiable. After any interesting incident, look at the questions asked during it and check which could be answered from existing data. The ones that could not are your actual coverage gaps, and they are far more specific than any maturity model will give you.

Cardinality is the whole argument

The technical reason the two approaches diverge comes down to cardinality — how many distinct values a field can take.

Classic metrics are cheap because they are aggregates. A counter of requests broken down by endpoint and status is a small, fixed number of series, and it will happily answer "are errors up?" for approximately nothing. What it cannot answer is "up for whom?" — because the individual customer identifiers, request identifiers and build versions were summed away at collection time. That information is not hidden or expensive to retrieve. It is gone.

Almost every incident that takes hours rather than minutes involves a failure that is not uniform. It affects one tenant, one region, one client version, one shard, one code path taken by a minority of requests. In aggregate that looks like a small elevation in the overall error rate, which is exactly the shape of noise. The signal only appears when you can group by the dimension that happens to matter, and you do not know which dimension that is until you are looking.

So the practical version of "invest in observability" is: keep enough per-event context to slice by dimensions you have not chosen yet, at least where the answers live. That costs money, and pretending otherwise is how these projects lose their budget halfway through. The decision worth making explicitly is which parts of the system are worth paying full context for and which are fine with aggregates — rather than discovering the answer during an outage.

Following a request across a boundary

The second structural requirement is being able to follow one unit of work across every hop it makes.

In a system of any size, the service that is slow is usually not the service that is broken. Latency propagates: the API is slow because the orders service is slow because it is waiting on a lock held by a batch job that nothing in the request path knows about. Each team, looking at its own dashboards, sees a service that is behaving badly for reasons outside its control, and the investigation becomes a sequence of conversations.

A propagated request identity collapses that sequence. It is also the thing most likely to be almost-working: instrumented in the services one team owns, dropped at the boundary of the ones it does not, absent entirely from the queue in the middle. A trace that stops halfway is only slightly better than no trace, because the gap is exactly where the handoffs are — and handoffs are where the time goes.

Getting this right is mostly an organisational problem wearing a technical costume. The instrumentation is not difficult. Agreeing a single convention across teams that do not otherwise coordinate, and then holding it for the asynchronous hops as well as the synchronous ones, is where it stalls.

Judge it by the incidents

The failure mode of this whole area is buying capability and never converting it into practice: the data is being collected, the bill is arriving, and during an actual incident three people still open the same four dashboards they have always used.

The way to avoid that is to make the review concrete. After each significant incident, write down the questions that were actually asked, in order, and mark each one: answered from existing telemetry, answered by adding instrumentation and waiting, or answered by guessing and testing. The second and third categories are your work queue, prioritised by something real rather than by a vendor's checklist.

Do that a few times and the pattern that emerges is rarely "we need another product". It is usually a handful of specific missing attributes, one boundary where context is dropped, and a set of alerts that fire without telling anyone anything. All three are fixable, and none of them appears on a maturity chart.