Cortex uses Jaeger to implement distributed tracing. We have found Jaeger invaluable for troubleshooting the behavior of Cortex in production.
In order to send traces you will need to set up a Jaeger deployment. A deployment includes either the jaeger all-in-one binary, or else a distributed system of agents, collectors, and queriers. If running on Kubernetes, Jaeger Kubernetes is an excellent resource.
In order to configure Cortex to send traces you must do two things:
- Set the
JAEGER_AGENT_HOSTenvironment variable in all components to point to your Jaeger agent. This defaults to
- Enable sampling in the appropriate components:
- The Ingester and Ruler self-initiate traces and should have sampling explicitly enabled.
- Sampling for the Distributor and Query Frontend can be enabled in Cortex or in an upstream service such as your frontdoor.
To enable sampling in Cortex components you can specify either
JAEGER_SAMPLER_MANAGER_HOST_PORT for remote sampling, or
JAEGER_SAMPLER_PARAM to manually set sampling
configuration. See the Jaeger Client Go
for the full list of environment variables you can configure.
Note that you must specify one of
JAEGER_SAMPLER_MANAGER_HOST_PORT in each component for Jaeger to be enabled,
even if you plan to use the default values.