The Missing Link: Why Your Traces Lie About Latency

The Illusion of Complete Visibility

The standard pitch for distributed tracing is that it reveals where time goes. Engineers instrument every service, watch the waterfall diagram, and assume they hold a complete map of the request lifecycle. This assumption holds only while the trace context survives every hop. The OpenTelemetry documentation describes context propagation as the mechanism that carries trace identifiers across process boundaries. When that mechanism works, spans nest correctly, and the total duration equals the sum of its parts. When it breaks, the trace does not show a gap; it simply ends. The system appears to have finished its work, but the user is still waiting.

This is not a minor cosmetic issue. It is a fundamental epistemic failure. If the context is dropped, the downstream work is invisible to the tracer. You are not looking at a partial view; you are looking at a truncated truth. The SRE book emphasizes that monitoring must answer specific questions about system behavior. If your tracing infrastructure cannot answer "where did the time go" because the question was cut off at a boundary, the tool has failed its primary purpose. The problem is not that the trace is noisy or too large. The problem is that the trace is silent exactly where the handoff occurs.

Teams often mistake the presence of spans for the presence of truth. A trace with five hundred spans looks impressive. But if one of those spans is the last one before a queue, and the next span belongs to a completely different trace ID, the connection is severed. The visual continuity is an illusion. The data is there, but it is not linked. You have two separate stories that happened to occur at the same time. The claim that tracing shows where time goes is true only for the time spent inside the closed loop of context propagation. The moment that loop breaks, the claim becomes false.

The Queue as a Context Breaker

Consider a common architecture: a web service receives a request, performs some validation, and then pushes a task onto a message queue for asynchronous processing. The web service creates a span for the HTTP request. It extracts the trace context from the incoming headers. It does its work. It serializes the task payload and publishes it to the queue. Here is the critical step: does it inject the trace context into the message metadata? If the application code forgets to do this, or if the queue client library does not support it, the context is lost.

The web service span ends when the message is acknowledged by the broker. The trace is now complete from the perspective of the tracing backend. It shows a fast request. The user, however, is waiting for the result of the background task. The worker service picks up the message. It creates a new span for the processing work. Because there was no parent context in the message, this new span has no parent. It starts a new trace. The tracing system sees two unrelated events: a fast web request and a slow background job. There is no link between them.

This is where the "where does the time go" question becomes unanswerable. The total user-perceived latency includes the queue wait time and the worker processing time. But the trace for the web request does not include either. The trace for the worker does not include the web request. You have a hole in the middle. The SRE book discusses the importance of correlating events to find root causes. Without the trace context bridging the queue, correlation is impossible. You cannot tell if the worker is slow because the queue is backed up, or because the worker itself is inefficient. You have two separate metrics, but no causal link. The gap is not just a missing data point; it is a missing relationship.

Third-Party Boundaries and Ownership Gaps

The problem is not limited to internal queues. It extends to any boundary where you do not control the instrumentation. This includes calls to third-party APIs, legacy systems, or services owned by other teams. When your service calls a third-party API, you inject the trace context into the HTTP headers. But if the third-party service does not respect or propagate that context, the trace ends at your service's boundary. You see the time spent waiting for the response, but you do not see what happened inside the third-party system.

This is often accepted as a limitation. "We can't see inside their box," the team says. But this framing is misleading. It suggests that the limitation is physical. In reality, the limitation is contractual and technical. The third-party service could propagate the context, but it chooses not to, or it lacks the capability. Your trace shows a black box. You know it took three seconds. You do not know if those three seconds were spent in network latency, database queries, or internal processing. The SRE book warns against assuming that black-box monitoring is sufficient for root cause analysis. A black box that does not propagate context is not just opaque; it is a break in the chain of evidence.

When this happens, the "where does the time go" question becomes a guess. You can measure the duration of the call, but you cannot decompose it. You are forced to rely on logs or metrics from the third party, which may not be available or may not be correlated. The trace, which was supposed to be the source of truth, is now just one piece of a puzzle that is missing its central piece. The team is left to infer the cause from incomplete data. This is not a failure of the tracing tool; it is a failure of the integration. The tool did what it was designed to do: it tracked the context. The context was not provided. The gap is a feature of the boundary, not a bug in the tracer.

Why Span Volume Is Not the Answer

The common response to these gaps is to add more spans. Teams instrument every function, every database call, every cache hit. They believe that if they have enough data, they can reconstruct the missing pieces. This is a mistake. Span volume does not create context propagation. If the context is dropped at the boundary, no amount of internal instrumentation will fix the link. You can have a million spans inside the worker service, but if they are not linked to the web service's trace, they are irrelevant to the original request.

The SRE book emphasizes the importance of actionable data. Data that cannot be correlated is noise. Adding spans without fixing propagation increases the cost of storage and processing without increasing the value of the trace. It makes the dashboard more cluttered, not more useful. The real work is not in instrumentation; it is in ensuring that the context survives every hop. This requires discipline. It requires that every producer injects the context, and every consumer extracts it. It requires that libraries support this, and that teams agree on the standard.

The claim that tracing shows where time goes is only true if you treat context propagation as a first-class concern. It is not an afterthought. It is the foundation. Without it, you are not tracing a request; you are collecting isolated events. The gaps are not small details. They are the handoffs. And the handoffs are where the system actually works. If you cannot see the handoffs, you cannot explain the system. The trace is not a map of the territory; it is a map of the parts of the territory that agree to be mapped. The rest is dark. And in the dark, you cannot find the cause of the delay. You can only guess.