Distributed Tracing: OpenTelemetry

On the first day after the microservices went live, everyone was pleased: the order, payment, inventory, and notification services each owned a clearly defined responsibility. At 3 a.m. on the third day, the on-call engineer was awakened because placing an order took five seconds and the client kept spinning. You check the order service logs and everything looks normal. You switch to the payment service, but its logs contain no errors. You ask the inventory team, and they say their service is fast. Every service "looks fine," yet the request is still slow.

This pain did not exist in the monolith era. In a monolith, a request runs from beginning to end within one process; run a profile and you can identify the slow line. Microservices split one request into N segments across N processes, with logs scattered over N machines. Each segment looks normal on its own, but together they are slow, and you have no tool to connect them. Distributed tracing solves this connection problem: it draws the complete call tree for a request and shows where time was spent, where errors occurred, and which services were crossed.

1. Core Concepts: Trace, Span, and Trace Context

Three terms are enough to understand distributed tracing.

Trace: the complete lifecycle of one request, from entry to response. The path from a user clicking "Submit" on an order to the page receiving a 200 response is one trace. It has a globally unique trace_id, which identifies the root of the tree.

Span: one operation, represented as a node in the trace tree. An HTTP call, SQL query, or Redis GET can each be a span. Every span has its own span_id, start and end times, parent pointer, attributes, and status (success/failure). A trace is a directed acyclic tree formed by connecting a collection of spans through parent-child relationships.

Trace Context: a trace_id does not simply appear independently in every service. It must be passed between services. The carrier being propagated is the trace context, which usually contains the trace_id, the current span's parent_span_id, and a sampling flag.

Here is what a real span tree looks like:

[order]      POST /api/order                          (520ms, trace root)
├── [order]    db.query  SELECT stock                 (12ms)
├── [payment]  POST /charge                           (480ms)
│   ├── [payment]   db.query  INSERT tx              (8ms)
│   └── [gateway]   HTTPS POST  /v1/charge           (460ms)   ← the bottleneck is here
└── [inventory] reserve                              (25ms)

At a glance, you can see that 460ms of the total 520ms was spent while the payment service called the external gateway. No guessing and no posting screenshots in a group chat to ask around; the tree speaks for itself. That is the value of tracing: it turns "distributed" back into something you can inspect on a single screen.

2. Context Propagation: How a Trace ID Crosses Services

A cross-service tree requires every span to share the same trace_id. How does this ID get from one service to the next? The answer is to put an HTTP header in the outbound request. The next service reads the header, uses it as the parent of its own span, and propagates it onward. This process is called Context Propagation.

The de facto industry standard is the traceparent header defined by W3C Trace Context. Its format fits on one line:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ^^
             version  trace-id (32 hex)           span-id (16 hex)   sampling flag

When the order service sends a request to the payment service, the HTTP client automatically adds this header. At the earliest point after receiving the request, the payment service framework parses traceparent, creates a child span from it, and continues propagating it. In addition to traceparent, W3C defines an optional tracestate header for vendor extensions, such as carrying information from a previous sampling decision. Most scenarios do not need it.

This step looks simple, but it is the part of distributed tracing most likely to fail. If context breaks at any point, every subsequent span becomes an orphan and the tree cannot be assembled. Common breakpoints include:

• Using a non-instrumented HTTP client: the framework's default instrumentation covers only mainstream clients. If you issue a raw request with fetch and do not propagate context manually, the trace breaks there.
• Losing context across an asynchronous boundary: after entering setTimeout, a message queue consumer, or setImmediate, the context may not follow. The OTel context API must propagate it explicitly.
• Using a non-HTTP protocol across processes: context must also be injected into and extracted from message headers for queues such as Kafka and RabbitMQ, which requires support from the corresponding instrumentation.

Checking whether context propagation works is straightforward: inspect a span from any downstream service. Its parent_span_id should point to a span in the upstream service. If the parent is empty or the trace_id changes at some point, propagation is broken. This is an almost inevitable issue during the first week of adopting distributed tracing.

3. OpenTelemetry: A Unified, Vendor-Neutral Standard

In the early days of distributed tracing, you had to choose among the Jaeger, Zipkin, and SkyWalking clients, binding your instrumentation code to the selected backend. Switching backends meant instrumenting every SDK again. This was classic vendor lock-in, and the industry had had enough.

That led to OpenTelemetry (OTel), the CNCF-backed standard for collecting observability data, created by merging the older OpenTracing and OpenCensus projects. Its core principle is: collection is separate from the backend. You instrument only with the OTel SDK and export the resulting data through OTLP. Whether the backend is Jaeger, Tempo, Datadog, or Alibaba Cloud ARMS, you can switch through OTLP or another exporter without changing a line of business code.

The complete OTel stack covers the three pillars (traces / metrics / logs), but traces remain its most mature and widely used signal. Its implementation has three layers:

1. API + SDK: your code calls APIs such as tracer.startSpan(). The SDK implements those APIs and generates and aggregates spans.
2. Instrumentation (automatic instrumentation): it patches mainstream libraries such as HTTP, gRPC, database drivers, and message queues to generate spans automatically. Most business code can achieve 80% trace coverage without changing a single line, which is OTel's greatest engineering benefit.
3. Collector: an independent process that receives exported application data, performs batching, resampling, redaction, and routing, then forwards the data to a backend. The Collector is a critical production component. It decouples applications from the backend and also handles tail sampling, covered in the next section.

A single OTel SDK emits all three signals, traces, metrics, and logs, and they share the same trace_id. Section 6 explains this further because it is the foundation for connecting the three pillars.

4. Sampling: Full Tracing Is Too Expensive, So How Do You Choose?

The largest cost of distributed tracing is not CPU but backend storage and ingestion. A reasonably active service can easily generate hundreds of millions of spans per day. Store all of them, and either the budget runs out or the backend fails first. Sampling is mandatory. The question is not whether to sample, but how to sample intelligently.

There are two basic strategies:

Head-based sampling: decide probabilistically at the trace entry point whether to keep the trace. After the decision, propagate it through the sampling flag in traceparent, and every downstream span follows that decision. It is simple, requires no additional coordination, and adds no latency. Its drawback is that the decision is made when the trace starts, before you know whether the request will eventually fail. A 1% sampling policy therefore misses 99% of error requests, even though errors are exactly what should be sampled. OTel's default ParentBased(AlwaysOn) / TraceIDRatioBased policies both belong to this category.

Tail-based sampling: buffer all spans from the complete trace in memory and wait until it ends, then decide whether to retain it based on whether it failed, whether it was slow, or whether it matched a business rule. This makes it possible to sample 1% of normal traces while retaining every error. With the same storage budget, the troubleshooting value is an order of magnitude higher. The cost is buffering tens of seconds of trace data in the Collector, which consumes memory and adds latency.

The standard production combination is: applications use head-based sampling to report 100% of traces to the Collector, and the Collector performs the final decision with tail-based sampling. Applications remain unaffected, all decisions are centralized in the Collector, and changing the rules does not require restarting business services. A typical tail sampling processor configuration looks like this:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Together, these three policies express a simple rule: keep errors, keep slow traces, and keep 5% of normal traces. This keeps costs under control while preserving useful data when problems occur, making it one of the most practical sampling patterns to adopt.

5. Using Traces to Locate Problems: Latency, Errors, and Call Relationships

Once instrumentation and sampling are in place, traces deliver their real value during incident response. There are three common use cases:

Locating slow calls. When a request is slow, inspect the trace timeline and find the longest span; that usually narrows the problem down substantially. An advanced technique is to inspect the gaps between spans. A 200ms blank interval between two child spans means the current service spent that time on its own CPU work, such as serialization, GC, or business logic, rather than waiting for a downstream service. Monitoring cannot see this time at all; only the trace timeline exposes it.

Locating error propagation. Every span in a trace has a status field (ERROR / OK) and exception events. When a 5xx surfaces at the gateway, follow parent spans backward through the trace to see which downstream service produced the error and which layer caught, wrapped, and rethrew it. The root cause is often the deepest ERROR span in the tree, so searching from the leaves toward the root is fastest.

Understanding call relationships. After a microservice system has run for a while, nobody can clearly explain which service calls which, how deep the dependencies are, or whether circular calls exist. Aggregate traces over a period into a Service Map, and the dependencies become immediately visible. It can also expose design smells such as "the order service is synchronously calling the notification service," which should have been asynchronous. For someone taking over a system, one service dependency map is worth more than ten pages of architecture documentation.

6. Correlating Logs and Metrics: trace_id Connects the Three Pillars

Viewed alone, tracing tells you which segment contains the problem. To ask why it was slow or why it failed, you must return to logs and metrics. This is where another role of the trace appears: it connects logs and metrics through trace_id.

The implementation is straightforward: whenever the application writes a log, include the current span's trace_id and span_id in the log line. OTel instrumentation usually injects both IDs into the logging context automatically; you only need to include them in the log format.

2026-07-17 03:14:22 [trace_id=0af7651916cd43dd8448eb211c80319c span_id=b7ad6b7169203331] payment charge failed: connection reset

With this line, troubleshooting becomes a straight path:

1. Find a slow trace in Grafana / Jaeger and identify the failing gateway span.
2. Copy the trace_id and paste it into the search box of the logging system (Loki / ELK).
3. Every log line corresponding to every service and span crossed by that trace appears in chronological order.

The path from "a high-level metric alert" → "the specific abnormal trace" → "the log lines for that trace" → "the stack and error text" is what Google SRE describes as correlating the three pillars of observability. Having all three pillars independently is not observability; observability means being able to move among them through one ID. Metrics tell you "there is a problem," traces tell you "which segment," and logs tell you "why." This workflow works out of the box in the Grafana stack (Prometheus + Tempo + Loki) and Elastic APM. The key is the shared trace_id.

7. Implementation: Add the OTel SDK and Choose a Backend

The implementation path has two parts: adding OTel to the application and choosing a backend.

Application side: in a Node.js project, initialize the SDK with @opentelemetry/sdk-node. It must be loaded before the application code. Start with a separate tracing.js using node --require ./tracing.js, or import it at the very top of the entry point. Otherwise, the instrumentation cannot hook mainstream libraries in time:

// tracing.js - must be loaded before app
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'order-service' }),
  traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4318/v1/traces' }),
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
});
sdk.start();

Once this is added, inbound and outbound HTTP, Express routes, and W3C traceparent injection and extraction are handled automatically without changing business code. To add business semantics, use trace.getSpan() to retrieve the current span and attach attributes such as order_id, user_id, and amount. Seeing these fields directly in a trace during troubleshooting is the key step that turns a "generic skeleton" into something useful to the business. Always configure service.name through Resource rather than relying on the hostname, or the downstream dependency graph will become inconsistent.

Backend selection: there are three categories.

• Open source / self-hosted: Jaeger (VendorNeutral, an established UI, and a mature community) and Tempo (part of the Grafana ecosystem, with the best integration with Loki / Prometheus and low-cost object storage). One of these is the usual self-hosted choice for small and medium-sized teams.
• Commercial APM: Datadog, New Relic, Dynatrace, Alibaba Cloud ARMS, and Tencent Cloud APM. They work out of the box and require almost no operational effort, but they cost money and move data outside your own environment.
• Cloud-managed OTel: AWS X-Ray, GCP Cloud Trace, and Azure Monitor. They integrate deeply within their respective clouds but become cumbersome across clouds.

The selection principle is simple: if you have the budget and want convenience and an SLA, choose a commercial APM; if you want data ownership, long-term retention, and custom analysis, self-host Jaeger / Tempo. Regardless of the choice, the OTel SDK layer remains unchanged. That is its greatest value as a standard: the backend can change without redoing the instrumentation.

Summary

• Distributed tracing solves the inability to connect activity across services: a trace is the complete call tree for one request, a span is one operation in the tree, and trace context propagates trace_id between services through the W3C traceparent header. If context breaks at any point, the tree cannot be assembled. This is an almost inevitable issue during the first week of adoption.
• OpenTelemetry is a unified, vendor-neutral standard: its three-layer architecture consists of the API, automatic instrumentation, and the Collector. It can provide 80% trace coverage with almost no changes to business code. Backends can change without redoing instrumentation.
• Sampling is an engineering tradeoff, not an on/off switch: head-based sampling is simple but misses errors, while tail-based sampling waits for the complete trace in the Collector before deciding. It can sample 5% of normal traces while retaining every error, delivering an order of magnitude more troubleshooting value with the same storage budget.
• Traces support three troubleshooting workflows: find the longest span to locate slow calls, work backward from the deepest ERROR span to identify the root cause, and aggregate a service dependency map to understand call relationships. Only the trace timeline exposes the monitoring blind spots represented by gaps between spans.
• trace_id is the shared key for the three pillars: include trace_id in logs to drill down from a metric alert to the exact error line. Metrics tell you "there is a problem," traces tell you "which segment," and logs tell you "why." Observability requires being able to move among them.

In one sentence: in the microservices era, debugging without distributed tracing is like defusing a bomb blindfolded. A single request crosses N services, and you cannot press ctrl+F in every service simultaneously, but trace_id can. That is the entire value of distributed tracing.