Dear Infinispan community,

With the Infinispan 14 development release 04, we started to support tracing with OpenTelemetry.

If configured, Infinispan Server produces cache events tracing spans and sends them to a remote tracing collector server, such as Jaeger or Zipkin.

Moreover, if a Java client application with the HotRot or the Rest client produces some tracing spans, these spans can be correlated as parent spans of the corresponding spans events produced by the Infinispan Server.

The new version of Infinispan Server comes with a gRPC OpenTelemetry Protocol (OTLP) Exporter, which is now supported by the majority of tracing servers.

For instance, with the newer Jaeger server versions, you can enable data collection through the OTLP protocol with the following option:

The server opens a port to import gRPC OTLP tracing data at port 4317.

Configure tracing on the Infinispan Server by setting system properties or environment variables:

The first property is Infinispan specific and enables the tracing capability of the Infinispan Server. The further properties belong to the OpenTelemetry SDK Autoconfigure project that Infinispan uses to configure the tracing exporter. In this case, OTLP gRPC Exporter protocol is used and the server runs on the same machine as the Infinispan Server.

Starting the server with these parameters:

The following log is produced when the server starts:

This indicates that the OpenTelemetrySdk is correctly configured.

Any OpenTelemetry tracing context present on HotRot client applications will be automatically propagated by the new Hot Rod v4 client to the server tracing context.

For instance, for a client that defines some tracing spans containing cache operations, such as the following:

The client span sub-bulk-1 will be correlated to any related server spans, in this case the three put operations.

Opening the Jaeger console, we can see that client and server spans are correctly aggregated:

You can find a complete application example here: https://github.com/fax4ever/infinispan-play/tree/main/tracing-hotrod-client

You can achieve the same with a REST client by putting manually in the HTTP headers the requests to provide information about the current tracing context using a standard OpenTelemetry instance of W3CTraceContextPropagator.

Opening the Jaeger console, you can see that client and server spans are correctly aggregated:

You can find a complete application example here: https://github.com/fax4ever/infinispan-play/tree/main/tracing-rest-client