{"id":19190,"date":"2023-07-21T14:31:24","date_gmt":"2023-07-21T09:01:24","guid":{"rendered":"https:\/\/blogrevamp.cashfree.com\/?p=19190"},"modified":"2026-02-27T15:42:12","modified_gmt":"2026-02-27T10:12:12","slug":"distributed-tracing-apm-the-what-why-and-how","status":"publish","type":"post","link":"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/","title":{"rendered":"Distributed Tracing &#038; APM: The What, Why and How"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_81 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #005c31;color:#005c31\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #005c31;color:#005c31\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#What_are_the_Problem_Statements\" >What are the Problem Statements?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#Goals\" >Goals<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#First_Thing_First_Lets_Learn_The_Jargon\" >First Thing First, Let\u2019s Learn The Jargon<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#Working_of_Distributed_Tracing_Tools\" >Working of Distributed Tracing Tools<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#How_did_we_do_it_for_hundreds_of_Microservices\" >How did we do it for hundreds of Microservices<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#Wins_from_Distributed_Tracing\" >Wins from Distributed Tracing<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n\n<p class=\"wp-block-paragraph\">At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worrying about being able to measure and get near real-time stats of system execution\/insights.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Especially in distributed systems like ours, where one failure can cause cascading effects, these side effects can eventually lead to multiple system failures. Unless we can <a href=\"https:\/\/tech.cashfree.com\/payouts\/payouts-observability\" target=\"_blank\" rel=\"noreferrer noopener\"><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-purple-color\">observe<\/mark><\/a> a system, we cannot be confident enough of deploying services and will always have constraints of time\/traffic\/hours\/support.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Manual efforts\/coordinations are always error-prone and the only way observability\/monitoring scales is via automation.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" fetchpriority=\"high\" decoding=\"async\" width=\"770\" height=\"493\" data-attachment-id=\"19196\" data-permalink=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/dta_1\/\" data-orig-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?fit=1771%2C1133&amp;ssl=1\" data-orig-size=\"1771,1133\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"DTA_1\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?fit=770%2C493&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?resize=770%2C493&#038;ssl=1\" alt=\"\" class=\"wp-image-19196\" srcset=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?resize=1024%2C655&amp;ssl=1 1024w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?resize=300%2C192&amp;ssl=1 300w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?resize=768%2C491&amp;ssl=1 768w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?resize=1536%2C983&amp;ssl=1 1536w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_1.png?w=1771&amp;ssl=1 1771w\" sizes=\"(max-width: 770px) 100vw, 770px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-are-the-problem-statements\"><span class=\"ez-toc-section\" id=\"What_are_the_Problem_Statements\"><\/span><strong>What are the Problem Statements?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>How can I attribute latency factor per service?<\/li>\n\n\n\n<li>How can I know the error rate and do its tracking per service?<\/li>\n\n\n\n<li>How can I know upstream and downstream services for any given service?<\/li>\n\n\n\n<li>How can I know the API hit duration and span list for optimisation?<\/li>\n\n\n\n<li>How can I build alerts on the above metrics?<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-goals\"><span class=\"ez-toc-section\" id=\"Goals\"><\/span><strong>Goals<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">At Cashfree Payments, we wanted to move to a system where teams can seamlessly monitor and observe their services. This includes being able to look at traffic patterns, latencies and the ability to trace the requested journey.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With the growing number of services, we needed to move to either a managed solution or build enough capability and allocate bandwidth to be able to support different needs. We used open-tracing libraries and APIs for instrumenting tracing in our applications and configured alerting on top of it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-first-thing-first-let-s-learn-the-jargon\"><span class=\"ez-toc-section\" id=\"First_Thing_First_Lets_Learn_The_Jargon\"><\/span><strong>First Thing First, Let\u2019s Learn The Jargon<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Request Tracing &amp; Span <\/strong>&#8211; A trace<a href=\"https:\/\/en.wikipedia.org\/wiki\/Tracing_(software)\" target=\"_blank\" rel=\"noreferrer noopener nofollow\"><sup>1<\/sup><\/a> is a representation of a series of related distributed events that encode the end-to-end request flow through a distributed system. This is critical for distributed systems<a href=\"https:\/\/en.wikipedia.org\/wiki\/Distributed_computing#:~:text=A%20distributed%20system%20is%20a,science%20that%20studies%20distributed%20systems.\" target=\"_blank\" rel=\"noreferrer noopener nofollow\"><sup>2<\/sup>,<\/a> as it allows debugging requests spanning multiple services, helping identify the source of latency\/increase in resource utilisation. Each span in the trace represents a single unit of work during that journey, such as an API call or database query.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In a nutshell, distributed tracing is the method of tracking application requests as they flow from frontend applications to backend services and databases. Developers can use distributed tracing to troubleshoot requests that exhibit high latency or errors.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img data-recalc-dims=\"1\" decoding=\"async\" width=\"770\" height=\"529\" data-attachment-id=\"19200\" data-permalink=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/dta_2\/\" data-orig-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?fit=1724%2C1184&amp;ssl=1\" data-orig-size=\"1724,1184\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"DTA_2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?fit=770%2C529&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?resize=770%2C529&#038;ssl=1\" alt=\"\" class=\"wp-image-19200\" style=\"width:750px;height:514px\" srcset=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?resize=1024%2C703&amp;ssl=1 1024w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?resize=300%2C206&amp;ssl=1 300w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?resize=768%2C527&amp;ssl=1 768w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?resize=1536%2C1055&amp;ssl=1 1536w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_2.png?w=1724&amp;ssl=1 1724w\" sizes=\"(max-width: 770px) 100vw, 770px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>APM<\/strong> &#8211; Application Performance Monitoring System (APM) gives deep visibility into applications for services, queues, and databases to monitor requests, number of API hits, errors tracking, latency, infrastructure metrics across hosts, containers, proxies, and serverless functions.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" decoding=\"async\" width=\"770\" height=\"544\" data-attachment-id=\"19202\" data-permalink=\"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/dta_4\/\" data-orig-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?fit=1735%2C1227&amp;ssl=1\" data-orig-size=\"1735,1227\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"DTA_4\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?fit=770%2C544&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?resize=770%2C544&#038;ssl=1\" alt=\"\" class=\"wp-image-19202\" srcset=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?resize=1024%2C724&amp;ssl=1 1024w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?resize=300%2C212&amp;ssl=1 300w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?resize=768%2C543&amp;ssl=1 768w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?resize=1536%2C1086&amp;ssl=1 1536w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_4.png?w=1735&amp;ssl=1 1735w\" sizes=\"(max-width: 770px) 100vw, 770px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Working_of_Distributed_Tracing_Tools\"><\/span><strong>Working of Distributed Tracing Tools<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Distributed tracing tools typically support three phases of request&nbsp; &nbsp; &nbsp; &nbsp; tracing:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Instrumentation<\/strong><br>First, you modify your code so requests can be recorded as they pass through your stack. In our case we have added the custom metric tracing library (open-tracing libraries) to the codebase repository. Below is the code snippet of golang<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-code\">\n\t<div class=\"cm-editor\">\n\t\t<div class=\"cm-scroller\">\n\t\t\t\n<pre>\n<code><div class=\"cm-line\">func InjectTracing(ctx context.Context, req *http.Request) (*http.Request, *HttpSpan) {<\/div><div class=\"cm-line\">  parentSpan := opentracing.SpanFromContext(ctx)<\/div><div class=\"cm-line\">  tracer := opentracing.GlobalTracer()<\/div><div class=\"cm-line\">  if parentSpan != nil {<\/div><div class=\"cm-line\">    carrier := opentracing.HTTPHeadersCarrier(req.Header)<\/div><div class=\"cm-line\">    ctx, _ := tracer.Extract(opentracing.HTTPHeaders, carrier)<\/div><div class=\"cm-line\">    op := &quot;[external.http]_&quot; + req.Host<\/div><div class=\"cm-line\">    sp := tracer.StartSpan(op, ext.RPCServerOption(ctx), opentracing.ChildOf(parentSpan.Context()))<\/div><div class=\"cm-line\">    ext.HTTPMethod.Set(sp, req.Method)<\/div><div class=\"cm-line\">    ext.HTTPUrl.Set(sp, req.URL.String())<\/div><div class=\"cm-line\">    ext.Component.Set(sp, op)<\/div><div class=\"cm-line\">    req = req.WithContext(opentracing.ContextWithSpan(req.Context(), sp))<\/div><div class=\"cm-line\">    err := tracer.Inject(sp.Context(), opentracing.HTTPHeaders, carrier)<\/div><div class=\"cm-line\">    if err != nil {<\/div><div class=\"cm-line\">      fmt.Printf(&quot;error injecting trace %v&quot;, err)<\/div><div class=\"cm-line\">    }<\/div><div class=\"cm-line\">    return req, &amp;HttpSpan{&amp;sp}<\/div><div class=\"cm-line\"> }<\/div><div class=\"cm-line\">  return req, &amp;HttpSpan{nil}<\/div><div class=\"cm-line\">}<\/div><\/code><\/pre>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data collection<br><\/strong>Second, configure the metrics scrapper agent as code is already instrumented in the above step. Now, traces will be scrapped from the instrumented services by the metrics scrapper agent and will be sent to the backend.<br><\/li>\n\n\n\n<li><strong>Analysis and visualisation<\/strong><br>Finally, the spans are unified into a single distributed trace and encoded with business-relevant tags for analysis. Depending on the distributed tracing tool being used, traces may be visualised as flame graphs or other types of diagrams.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_did_we_do_it_for_hundreds_of_Microservices\"><\/span><strong>How did we do it for hundreds of Microservices<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">At Cashfree Payments, we have 150+ codebase repositories supporting different coding languages such as Java, Golang, Python etc.&nbsp; which require a code change for the instrumentation of tracing and alert monitoring.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s see how we solved this challenge seamlessly:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We went ahead and created a core team of 2 people who would be responsible for end-to-end execution of this project, right from POC to code change to take this live. Needless to say, this includes support in case of any production issues occurring due to these changes for a period of one quarter.<\/li>\n\n\n\n<li>To optimise the above activity, we created an internal custom metric tracing library whose dependency was added in the client of each codebase repository. By doing this, we have not only avoided duplicated code across all repositories, but also saved our effort and time as well.<\/li>\n\n\n\n<li>For configuring alerts and its routing rule as per the team in each code repository based on its collected metrics, we have created an automation script which added desired alerts based on language and domain in each service code repository, and raised a pull request<sup><a href=\"https:\/\/www.atlassian.com\/git\/tutorials\/making-a-pull-request\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">4<\/a><\/sup> for service owners to review and deploy it to production. This Automation script further saved our time and effort plus avoided any human error.<\/li>\n\n\n\n<li>Rather than being reactive about the occurrence of issues with this change, we proactively enabled SonarQube<sup><a href=\"https:\/\/www.sonarsource.com\/products\/sonarqube\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">5<\/a><\/sup> so that any issues can be caught in the code review phase itself.<\/li>\n\n\n\n<li>&nbsp;To be double sure about not breaking anything in production, we first tried deploying services in the Beta environment and then post successfully observing the behavior of the services, and shipped the code to the production environment.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Wins_from_Distributed_Tracing\"><\/span><strong>Wins from Distributed Tracing<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Reduced time to Detect and Respond<br><\/strong>Now,<strong> <\/strong>if a customer reports that a feature in an application is slow or broken, the support team reviews distributed traces to determine if this is a backend issue. Engineers then analyse the traces generated by the affected service to quickly troubleshoot the pinpointed problem.<br><\/li>\n\n\n\n<li><strong>Alerts via Monitors and Watchdogs<\/strong><br>Now alerts have been set up on the metrics\/log patterns or APM and it helps in proactively checking the system&#8217;s health.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Gaining deep insight into services&nbsp;<\/strong><br>Understanding service dependencies has become very easy with an auto-generated service map from traces alongside service performance metrics and monitor alert statuses. Now, we can analyse:<\/li>\n<\/ul>\n\n\n\n<ol class=\"wp-block-list\">\n<li>&nbsp;Individual database queries&nbsp;<\/li>\n\n\n\n<li>&nbsp;Endpoints correlated with infrastructure<\/li>\n\n\n\n<li>&nbsp;Monitor service performance<\/li>\n\n\n\n<li>&nbsp;Compare between versions for rolling, blue\/green, shadow, or canary deployments.<\/li>\n<\/ol>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Understand latency breakups and service metrics<\/strong><strong><br><\/strong>By viewing distributed traces, developers now understand cause-and-effect relationships between services and optimise their performance. Tracing gives us insights into how services communicate, the time taken in executions etc.<\/li>\n\n\n\n<li><strong>Improve collaboration and productivity<\/strong><strong><br><\/strong>In microservice architecture, different teams own the services that are involved in completing a single request. Distributed tracing makes it clear where an error occurred and which team is responsible for fixing it.<\/li>\n\n\n\n<li><strong>Maintain Service Level Agreements (SLAs)<br><\/strong>Distributed tracing tools aggregate performance data from specific services, so teams can readily evaluate if they comply with SLAs<sup><a href=\"https:\/\/en.wikipedia.org\/wiki\/Service-level_agreement\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">6<\/a><\/sup>.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span><strong>Conclusion<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">While implementing distributed tracing does introduce some overhead and complexity, the benefits it brings far outweigh the challenges.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As software systems continue to evolve and become more complex and distributed, distributed tracing plays an increasingly vital role in ensuring the smooth operation and performance of these systems.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this blog, we have shared a sneak peak of our journey with distributed tracing and we can definitely say this is not the end of our tech evolution as at Cashfree Payments where we embrace the challenge of exceeding yesterday&#8217;s success!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Does this sound exciting and intriguing to you? Well then, there are some great opportunities in store for intrepid engineers just like you!<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.cashfree.com\/careers?utm_source=blog&amp;utm_medium=organic&amp;utm_campaign=distributed-tracing-apm-blog\/\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"770\" height=\"172\" data-attachment-id=\"13055\" data-permalink=\"https:\/\/blogrevamp.cashfree.com\/reporting-systems-to-ensure-enhanced-customer-satisfaction\/tech-blog\/\" data-orig-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?fit=1600%2C358&amp;ssl=1\" data-orig-size=\"1600,358\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"Tech Jobs at Cashfree Payments\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?fit=770%2C172&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?resize=770%2C172&#038;ssl=1\" alt=\"Jobs at Cashfree Payments\" class=\"wp-image-13055\" srcset=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?resize=1024%2C229&amp;ssl=1 1024w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?resize=300%2C67&amp;ssl=1 300w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?resize=768%2C172&amp;ssl=1 768w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?resize=1536%2C344&amp;ssl=1 1536w, https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2022\/10\/Tech-Blog.png?w=1600&amp;ssl=1 1600w\" sizes=\"(max-width: 770px) 100vw, 770px\" \/><\/a><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.<\/p>\n","protected":false},"author":87,"featured_media":19194,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_themeisle_gutenberg_block_has_review":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_wpcom_ai_launchpad_first_post":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1497],"tags":[],"class_list":["post-19190","post","type-post","status-publish","format-standard","has-post-thumbnail","category-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Distributed Tracing &amp; APM: The What, Why and How<\/title>\n<meta name=\"description\" content=\"At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Distributed Tracing &amp; APM: The What, Why and How\" \/>\n<meta property=\"og:description\" content=\"At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/\" \/>\n<meta property=\"og:site_name\" content=\"Cashfree Payments Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-07-21T09:01:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-02-27T10:12:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"3402\" \/>\n\t<meta property=\"og:image:height\" content=\"2252\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Priyanka Agrawal\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Priyanka Agrawal\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Distributed Tracing & APM: The What, Why and How","description":"At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/","og_locale":"en_US","og_type":"article","og_title":"Distributed Tracing & APM: The What, Why and How","og_description":"At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.","og_url":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/","og_site_name":"Cashfree Payments Blog","article_published_time":"2023-07-21T09:01:24+00:00","article_modified_time":"2026-02-27T10:12:12+00:00","og_image":[{"width":3402,"height":2252,"url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","type":"image\/jpeg"}],"author":"Priyanka Agrawal","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Priyanka Agrawal","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#article","isPartOf":{"@id":"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/"},"author":{"name":"Priyanka Agrawal","@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/e94de233414e12d9d2bc095f070df77f"},"headline":"Distributed Tracing &#038; APM: The What, Why and How","datePublished":"2023-07-21T09:01:24+00:00","dateModified":"2026-02-27T10:12:12+00:00","mainEntityOfPage":{"@id":"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/"},"wordCount":1199,"commentCount":0,"image":{"@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","articleSection":["Engineering"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blogrevamp.cashfree.com\/distributed-tracing-apm-the-what-why-and-how\/","url":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/","name":"Distributed Tracing & APM: The What, Why and How","isPartOf":{"@id":"https:\/\/blogrevamp.cashfree.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#primaryimage"},"image":{"@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","datePublished":"2023-07-21T09:01:24+00:00","dateModified":"2026-02-27T10:12:12+00:00","author":{"@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/e94de233414e12d9d2bc095f070df77f"},"description":"At Cashfree Payments, we have millions of transactions happening everyday and each of these transactions has a journey across tens of microservices. Our motivation was to make any software-related changes at scale without worry.","breadcrumb":{"@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#primaryimage","url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","contentUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","width":3402,"height":2252,"caption":"Distributed Tracing"},{"@type":"BreadcrumbList","@id":"https:\/\/www.cashfree.com\/blog\/distributed-tracing-apm-the-what-why-and-how\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blogrevamp.cashfree.com\/"},{"@type":"ListItem","position":2,"name":"Engineering","item":"https:\/\/blogrevamp.cashfree.com\/category\/engineering\/"},{"@type":"ListItem","position":3,"name":"Distributed Tracing &#038; APM: The What, Why and How"}]},{"@type":"WebSite","@id":"https:\/\/blogrevamp.cashfree.com\/#website","url":"https:\/\/blogrevamp.cashfree.com\/","name":"Cashfree Payments Blog","description":"Cashfree Payments- Payment Gateway for India","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blogrevamp.cashfree.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/e94de233414e12d9d2bc095f070df77f","name":"Priyanka Agrawal","url":"https:\/\/blogrevamp.cashfree.com\/author\/priyanka-agrawal\/"}]}},"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9MjXo-4Zw","jetpack_likes_enabled":false,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2023\/07\/DTA_Tn.jpg?fit=3402%2C2252&ssl=1","_links":{"self":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/19190","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/users\/87"}],"replies":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/comments?post=19190"}],"version-history":[{"count":13,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/19190\/revisions"}],"predecessor-version":[{"id":33035,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/19190\/revisions\/33035"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/media\/19194"}],"wp:attachment":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/media?parent=19190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/categories?post=19190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/tags?post=19190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}