In order to send traces you will need to set up a Jaeger deployment. A deployment includes either the jaeger all-in-one binary, or else a distributed system of agents, collectors, and queriers. If running on Kubernetes, Jaeger Kubernetes is an excellent resource.
In order to configure Cortex to send traces you must do two things:
- Set the
JAEGER_AGENT_HOSTenvironment variable in all components to point to your Jaeger agent. This defaults to
- Enable sampling in the appropriate components:
- The Ingester and Ruler self-initiate traces and should have sampling explicitly enabled.
- Sampling for the Distributor and Query Frontend can be enabled in Cortex or in an upstream service such as your frontdoor.
To enable sampling in Cortex components you can specify either
JAEGER_SAMPLER_MANAGER_HOST_PORT for remote sampling, or
JAEGER_SAMPLER_PARAM to manually set sampling
configuration. See the Jaeger Client Go
for the full list of environment variables you can configure.
Note that you must specify one of
JAEGER_SAMPLER_MANAGER_HOST_PORT in each component for Jaeger to be enabled,
even if you plan to use the default values.
In order to send traces you will need to set up an OpenTelemetry Collector. The collector will be able to send traces to multiple destinations such AWS X-Ray, Google Cloud, DataDog and others. OpenTelemetry Collector provides a helm chart to set up the environment.
See document on the tracing section in Configuration file.
Cortex is maintaining backward compatibility with Jaeger support, Cortex has not fully migrated from OpenTracing to OpenTelemetry and is currently using the OpenTracing bridge.