A checkout was intermittently taking eleven seconds. Not always — maybe one request in twenty. Four services were involved, each with its own logs, each looking completely healthy in isolation. Every dashboard was green.
We spent two days on it. The eventual cause was that one service called an address-validation vendor without a timeout, and that vendor was slow for a subset of postcodes. Two days, because reconstructing a single request's path across four services meant grepping four log streams by timestamp and hoping.
After we added distributed tracing, the same class of problem took four minutes: open a slow trace, look at the waterfall, see one span consuming ten of the eleven seconds. That difference — two days versus four minutes — is the entire argument for OpenTelemetry, and it is why I now add it before a system is distributed rather than after.
Three Signals, One Standard
OpenTelemetry is a vendor-neutral standard for producing telemetry. That is genuinely the important part: you instrument once, and you can send the data to Jaeger, Grafana, Datadog, Honeycomb or anything else without touching application code again. Before this, switching observability vendors meant reinstrumenting everything, which meant you never switched.
Traces follow one request across every service it touches. This is the signal that solves problems logs cannot, because it preserves causality and timing across process boundaries.
Metrics are aggregated numbers over time — request rate, error rate, latency percentiles, queue depth. Cheap to store, good for dashboards and alerts, useless for explaining an individual failure.
Logs are events with detail. Still necessary, and far more useful once they carry trace IDs, because then a log line and the trace it belongs to are one click apart.
The mental model I use: metrics tell you something is wrong, traces tell you where, logs tell you what. Teams that only have logs spend their time doing the first two jobs by hand.
Auto-Instrumentation Gets You Most of the Way
People assume this is a large project. For a Node service, the baseline is a file and a flag.
// tracing.ts — loaded before anything else via --require
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
new NodeSDK({
serviceName: process.env.OTEL_SERVICE_NAME ?? "orders-api",
traceExporter: new OTLPTraceExporter(), // endpoint from OTEL_EXPORTER_OTLP_ENDPOINT
instrumentations: [getNodeAutoInstrumentations({
"@opentelemetry/instrumentation-fs": { enabled: false }, // very noisy
})],
}).start();
That single file gives you spans for every HTTP request in and out, every database query, and every Redis call — with trace context propagated between services automatically through headers. No changes to business logic.
Turning off the filesystem instrumentation is not optional in my experience. It produces enormous numbers of spans nobody has ever needed.
The Manual Spans Worth Adding
Auto-instrumentation covers I/O. It knows nothing about your domain, and that is where the useful detail lives.
const tracer = trace.getTracer("orders");
export async function placeOrder(input: OrderInput, ctx: Ctx) {
return tracer.startActiveSpan("order.place", async (span) => {
// High-cardinality attributes: what you will actually filter on later.
span.setAttribute("tenant.id", ctx.tenantId);
span.setAttribute("order.item_count", input.items.length);
span.setAttribute("order.total_gbp", input.total);
span.setAttribute("payment.provider", input.provider);
try {
const order = await create(input);
span.setAttribute("order.id", order.id);
return order;
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end(); // always, or the trace never closes
}
});
}
The attributes are the point. A trace that says "this took 11 seconds" is mildly useful. A trace that says "this took 11 seconds, tenant X, payment provider Y, 47 items" lets you find the pattern across thousands of requests — which is how you discover it is one vendor and a subset of postcodes.
Be generous with high-cardinality attributes like IDs. That is exactly what tracing is good at, and it is the opposite of the advice for metrics, where high cardinality is expensive.
Correlate Your Logs
The cheapest high-value step, and it is often two lines. Put the trace and span ID into every log line:
const span = trace.getActiveSpan()?.spanContext();
logger.info({
msg: "payment authorised",
trace_id: span?.traceId,
span_id: span?.spanId,
order_id: order.id,
});
Now a log line links to its trace and a trace links to its logs. During an incident that is the difference between investigating and searching.
Sampling, Because You Cannot Keep Everything
Tracing every request at volume is expensive and mostly stores traces of things that worked. The naive fix is head sampling — keep 5% at random — which is cheap and has the obvious flaw: your rare failure is probably in the 95% you dropped.
Tail sampling is what you want. Buffer the complete trace, then decide: keep everything with an error, everything slower than a threshold, and a small percentage of the rest. You store the interesting traces and discard the boring ones. This runs in a collector rather than in your app, which is the main reason to deploy the OpenTelemetry Collector rather than exporting directly from services.
The collector is worth it for other reasons too — it batches, retries, scrubs PII before data leaves your network, and lets you change backends without redeploying anything.
Where It Goes Wrong
Instrumenting one service. A trace that stops at the first service boundary is a very expensive log. Context propagation across all your services is where the value is; do them together.
Spans that never end. A missing span.end() in an error path leaves traces open and incomplete. Always use finally.
PII in attributes. Email addresses, card details and names end up in a third-party observability platform and now you have a compliance question. Scrub at the collector.
Treating it as a dashboard project. Dashboards are for metrics. Tracing is for investigation, and its value shows up during incidents, not in a weekly review.
When to Add It
Honestly: before you need it. Tracing is the one piece of infrastructure that is nearly impossible to add usefully during an incident, because you need the data from before the incident started.
For a single service, structured logs with request IDs are probably enough and I would not push OpenTelemetry hard. The moment there are two services calling each other, the calculus changes completely — that is when "which service is slow?" stops being answerable by looking at any one of them.
Our eleven-second checkout was four services in, with no tracing, because nobody had wanted to spend a day on instrumentation. We spent two days on one bug instead.
