What is distributed tracing (often just called tracing), and what are its main components? More importantly: why should you invest time in integrating distributed tracing into your tech stack? There are many reasons, but in this post we'll cover the most significant ones.
It reveals service dependencies
The strategic decision has been made: your company has started migrating to a distributed architecture, the number of components has started to grow, and the ability to understand the organization's architecture has dropped. With distributed tracing, you can follow the path of a request as it moves through a complex system. Some tools even compute and draw a complete dependency graph. It can be useful both to get an overview of your architecture and to dive deep to better understand its dependencies.
Find the latency of each component along the path
You can detect latency by monitoring requests to edge services with monitoring systems. OK, you got the alert — but now you need to find out which component is causing that specific request to exceed the Service Level Objective (SLO). That is exactly why distributed tracing exists: to pinpoint the components along the path that are bottlenecks or likely points of failure.
Root cause analysis
Picture this – you get paged at 2 a.m., and a request that spans 5 different microservices is failing over and over. You desperately search the logs, still squinting at a blindingly bright screen, looking for errors around the time of the alert, but the volume of data is far too large to figure out what happened quickly and accurately. It's taking too long. With distributed tracing, you can find the first service that failed, pull the logs for that failure and a few other things besides (depending on how tracing was implemented in the system).
Collect events during the request
To help with debugging, you can attach extra data to the trace. For example, you can add every feature flag that was evaluated to the trace so that, if something fails, you know exactly which flags were evaluated in each service along the request path.
So what exactly is a tracing tool?
Let's start with the basics — where all distributed tracing solutions came from. Although a few distributed tracing solutions existed before it, the 14-page paper Google Dapper (2010) is, so to speak, the cornerstone of distributed tracing. The paper explains how Google built a production-grade tracing tool with 3 main goals in mind:
1) Low overhead – The tracing system should have a negligible performance impact on running services. In some highly optimized services, even small monitoring overheads are easily noticeable. For example, a single Google search query fans out across thousands of machines and services, so Google cannot afford anything but a negligible performance impact on each of those requests.
2) Application-level transparency – Tracing should not require active collaboration from programmers, since that can be fragile and would consume programmers' precious (and expensive) time to learn it.
3) Scalability – It must handle Google's scale for at least the next few years.
There is also a requirement that trace data be available within minutes of the request.
Service instrumentation and terminology
Dapper also introduced a terminology. It may vary among the solutions available on the market today, but these are the core ideas in most of Dapper's successors:
Trace: The description of a transaction as it moves through a distributed system.
Span: A named, timed operation that represents a piece of the workflow. Spans accept key-value tags as well as fine-grained, timestamped structured logs attached to that specific span instance.
(Span) Context: The identifying information that travels with the distributed transaction, including when it passes from service to service over the network or through a message bus. The span context contains the trace identifier, the span identifier and any other data the tracing system needs to propagate to the downstream service.
Instrumentation: Instrumentation is the process by which application code is extended to capture and report trace spans for the operations of interest.
Annotations (sometimes called baggage): Allow the developer to enrich the trace with user-defined data; they can be used to store counters, relevant logs and any data that might help investigate an incident.
The trace context is serialized and passed along on calls between instrumented services — whether that's an RPC, an event message, SMTP or any other communication channel the developer has in mind — with the client sending the trace data to the server (for example, via an HTTP header in REST calls). The server deserializes the span and starts a new one that points to its parent span, the span received from the client.
The evolution of Dapper
Now that we've covered the history of distributed tracing tools, let's look at the descendants available on the market today – many of them share the same concept or the same approach as Dapper.
Today's solutions fall into 2 groups:
Open source solutions:
Enterprise solutions:
Want to get started with tracing without turning it into a six-month project? The Elven team helps you choose the right instrumentation for your scenario and get your first traces into production.
Today's standard is called OpenTelemetry
When this post was first written, choosing distributed tracing also meant choosing a proprietary agent — and re-instrumenting everything whenever you switched tools. OpenTelemetry settled that debate: it is a CNCF project that standardizes the collection of traces, metrics and logs, and today it is supported by virtually every observability backend, Elven included.
In practice, this changes the order of decisions. Instrument with OpenTelemetry first, pick the backend later — and keep the freedom to switch without redoing the instrumentation work.
Already instrumented with OpenTelemetry and looking for a Brazilian backend for your data? Talk to the Elven team.
