In this article we will look at how to integrate Jaeger distrubuted tracing into Spring Boot microservices.
1.0 Introduction to Distributed Tracing & Jaeger
Microservices architecture has gained lot of popularity and acceptance due to its ability to solve problems like application scalability, availability, etc.
With time we have realized that developing microservices is not that difficult with frameworks like SpringBoot making our lives easier day by day. The real challenge comes after the microservies are deployed – the operation problems due to distributed nature of microservices.
Two most important challenges are
- Monitoring – how do we monitor all the services and their metrics
- Tracing – how do we trace all the requests and identify bottlenecks in our application?
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems.
Jaeger can help us in following areas
- Distributed transaction monitoring
- Service dependency analysis
- Performance / latency optimization – identify bottlenecks
Jaeger is compatible with OpenTracing, and has instrumentation libraries in multiple languages like Golang, Java, Node, Python, etc
2.0 Spans and Traces
The first thing that we need to understand is
Trace, as defined by OpenTracing Specifications.
Spans are timed operations representing small logical units of work. Each span contains some basic information like
Duration, etc. We can think of Spans as the building blocks of a distribute trace.
Trace is nothing but a complete execution path of a request and can be thought of as a collection of
Directed Acyclic Graph of Spans.
Suppose you are calling an API to get AllOrders for a Customer. In this case
- Trace is the entire HTTP Request/Response
- Spans could be
- Request from Loadbalance to Order Service
- Request Order Service to Product Service
- Product Service to Database
- Order Service to Database
When service A receives a request that contains no tracing information, Jaeger tracer will start a new trace, assign it a random trace ID.
Each trace starts by passing a
context that is carried forward in each of the spans. Take a look at below animation on how each call from A -> B -> C, etc each create an individual span for the trace.
3.0 Jaeger Architecture
This is how typical Jaeger architechture looks like
This is how the data flows from instrumented application to Jaeger
- It is a network daemon that listens to all spans send over UDP, and it batches the spans and sends to collector.
- It runs local to the application, so the application does not needs to know about routing and discovery of collectos
- Receives traces from all Jaeger agents
- Validates the traces, indexes them and also does any transformation if required.
Following storage options are supported by Jaeger –
Used to fetch the traces from storage and hosts an UI to diplay them.
3. Jaeger Demo – running it locally
3.1 Running Jaeger
For the local setup and demo of Jaeger, we will keep it very simple.
We will run the image –
jaegertracing/all-in-one – it launches the Jaeger UI, collector, query, and agent, with an in memory storage component.
This means no need to worry about setting up storage, or setting up of separate agents or collectors.
We can start the Jaeger all in one container using following docker command
docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ -p 5775:5775/udp \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 14268:14268 \ -p 14250:14250 \ -p 9411:9411 \ jaegertracing/all-in-one:1.19
You may be thinking why expose so many ports and what do these ports serve. Most important ports of our concern are
|6831||UDP||AGENT||– This is the port on which the application sends traces to agents. |
|6832||UDP||AGENT||– Accepts |
– Required by some client libraries that do not support compact
|16686||HTTP||QUERY||Serves the frontend for Jaeger UI|
|14268||HTTP||COLLECTOR||– Accepts |
– In some cases where it is not possible to deploy Jaeger Agent next to application (e.g AWS Lambda), Jaeger clients can submit spans directly to Collector over http/https.
– Url :
Since Jaeger 1.11, the official and recommended protocol between Agents and Collectors is
3.2 Instrumenting our application
Next step is to setup instrumentation of our applications, so that they emit traces to Jaeger collector.
Our sample Spring Boot Microservices application, that we will instrument looks like this
We will add Jaeger instrumentation to following components
- Spring Boot Services
- BFF Service
- Order Service
- Product Service
- Customer Service
- Traefik Loadbalancer
Adding Instrumentation to Spring Boot Services
- Add Dependency in Gradle
implementation group: 'io.opentracing.contrib', name: 'opentracing-spring-jaeger-cloud-starter', version: '3.1.2'
- Add following configuration to the
opentracing: jaeger: service-name: <name of application> enabled: true udp-sender: host: <agent-host> port: 6831
The name of the application can also be configured using Spring’s property –
Thats all configurations we need to do, to enable instrumentation in our Spring Boot application and emit spans to Jaeger agent.
Adding Instrumentation to Traefik Loadbalancer
Traefik also supports Jaeger instrumentation out of the box, and can be enabled by adding following properties to Traefik startup.
3.3 Viewing Jaeger traces
Lets run our application and view the traces.
Next we will hit an endpoint to get all orders of a customer –
Now go to Jaeger UI on http://localhost:16686/search, select
traefik service and click on “Find traces“
You will see a trace like this
As you can see that the API call results in all 4 services being called.
You can see the Dependency Graph by clicking on “Deep Dependency Graph” button
Now lets see the traces in more details, by clicking on any of the trace.
You can see complete trace, with each service call and the time it took
Dig further deep and you can even see the database queries being fired, and logs being logged by the application.
So far we have seen that Jaeger provides very powerful tracing capabilities that can help in tracing/debugging a microservices application.
We have also seen how easy it is to configure and instrument a Spring Boot application to emit traces to Jager.
With this we come to end of our current article. In future articles we will look at integration more telemetries with microservcies.