Concepts of tracing

Callstacks

A callstack is an ordered list of code executions. Whenever code invokes other code, the new code is put onto the topmost of the stack. Callstacks are used by runtimes of all programming languages and are usually printed as a stacktrace. When an error occurs, the stacktrace traces back to the calls that led to the error.

For example, the following error message states Apple is not a number. Combined with the callstack, it's possible to narrow down in a complex system where the error occurred. The message alone is usually insufficient, as the NumberUtil algorithm might be used in many places.

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              NumberUtil.convert()  <-- Error: "Apple is not a number"

To understand why the error occurred, use the callstack to trace back from the error to the relevant business method, which in this case is ShoppingCart.parseQuantity().

Callstacks themselves are insufficient for monitoring. They are not easy to read and do not provide information to correlate the performance and availability of a system to overall health. To see what happens on a code execution and to correlate, consider information like process activity, resource usage, queuing, access patterns, load and throughput, system, and application health.

Distributed tracing

With the introduction of service oriented architectures (SOA), the callstack is broken apart. For example, the ShoppingCart logic might now reside on server A, while NumberUtil resides on server B. An error trace on server B contains only the short callstack of the parse error. While on server A, a new error is produced stating that something went wrong on server B, but not stating the problem itself.

Instead of a single error callstack that is easy to troubleshoot, you end up with two callstacks with two errors. Also, because no connection exists between the two, it makes it impossible to have access to both at the same time.

Server A:

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              RestClient.invokeConversion() <-- Error: Unkown

Server B:

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      NumberUtil.convert()  <-- Error: "Apple is not a number"

The idea behind distributed tracing is to fix this problem by connecting the two error call stacks with each other. Most implementations use a simple mechanism to do so; when server A calls server B, the application performance monitoring (APM) tool adds an identifier to the call that serves as a common reference point between the callstacks in the APM system. This mechanism is called correlation and to produce one error, it joins the two callstacks.

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              RestClient.invokeConversion()
                Thread.run()
                  HttpFramework.service()
                    HttpFramework.dispatch()
                      NumberUtil.convert()  <-- Error: "Apple is not a number"

By understanding where the remote call takes place and on which server parts of the callstack were executed, you can find out that the ShoppingCart was the context of the error, and the NumberUtil caused the shopping cart activity to fail.

Measuring performance

However, the preceding examples illustrate that the APM tools trace errors by using the same mechanism that is used for taking and presenting performance measurements. The trace is annotated with performance numbers as shown:

413 Thread.run()
413   HttpFramework.service()
413     HttpFramework.dispatch()
412       ShoppingCart.update()
411         ShoppingCart.updateCart()
211           ShoppingCart.parseQuantity()
210             ShoppingCart.convertParams()
200               RestClient.invokeConversion()
 10                 Thread.run()
 10                   HttpFramework.service()
 10                     HttpFramework.dispatch()
  5                       NumberUtil.convert()

The total time for executing the shopping cart update is approximately 413 ms. The number conversion (NumberUtil.convert()) took 5 ms. The time that is spent in between is distributed among many calls, so you are looking for bigger cliffs. In the example, updating the cart (ShoppingCart.updateCart()) took a total of 411 ms, while the parsing (ShoppingCart.parseQuantity()) only required 211 ms, which itself spent most of the time in doing the remote call.

Trace continuity

Trace continuity means that calls triggered by one external request are collected into one trace. Instana employs protocol-specific means to add metadata, such as HTTP headers, gRPC metadata, Kafka message headers, AMQP headers, JMS headers, and more. Adding metadata ensures trace continuity across all protocols and services.

Communication protocols without support for any metadata do not support trace continuity, which means that when you call another service over such a protocol, the outgoing call is a leaf in the trace tree. The work that happens in the receiver of the call is not part of that trace. Instead, receiving the call starts a new trace and all subsequent calls that are triggered in the receiver belong to this new trace.

Trace continuity is not supported in the following cases:

• Kafka up to version 0.10 (Kafka introduced headers in version 0.11)
• Sending or receiving Kafka messages with the Node.js package kafka-node (that is, the package does not support headers. When you work with Kafka in Node.js, use the npm package kafkajs instead of kafka-node because kafkajs supports trace continuity. For more information, see additional remarks for continuing the trace for incoming messages)
• NATS and NATS streaming messaging
• Microsoft Message Queue

W3C Trace Context Support

The following Instana tracers support the W3C trace context specification for HTTP or HTTPS communication in addition to the proprietary headers like X-INSTANA-T or X-INSTANA-S:

• .NET Core
• .NET
• Apache HTTPd
• Clojure
• Go
• Haskell
• Java
• Kotlin
• Node.js
• Python
• Ruby
• Scala

The following Instana tracers currently do not support the W3C trace context specification. Only the proprietary headers like X-INSTANA-T or X-INSTANA-S are supported:

• Envoy
• NGINX
• PHP

Tracing headers

To ensure the trace continuity across different services, Instana tracers use different headers or metadata properties, depending on the protocol.