Overrides Exporter

Since Cortex is a multi-tenant system, it supports applying limits to each tenant to prevent any single one from using too many resources. In order to help operators understand how close to their limits tenants are, the overrides-exporter module can expose limits as Prometheus metrics.

Context

To update configuration without restarting, Cortex allows operators to supply a runtime_config file that will be periodically reloaded. This file can be specified under the runtime_config section of the main configuration file or using the -runtime-config.file command-line flag. This file is used to apply tenant-specific limits.

Example

The overrides-exporter is not enabled by default; it must be explicitly enabled. We recommend only running a single instance of it in your cluster due to the cardinality of the metrics emitted.

With a runtime.yaml file given below:

# file: runtime.yaml
# In this example, we're overriding ingestion limits for a single tenant.
overrides:
  "user1":
    ingestion_burst_size: 350000
    ingestion_rate: 350000
    max_global_series_per_metric: 300000
    max_global_series_per_user: 300000
    max_series_per_metric: 0
    max_series_per_user: 0
    max_fetched_series_per_query: 100000

The overrides-exporter is configured to run as follows:

cortex -target overrides-exporter -runtime-config.file runtime.yaml -server.http-listen-port=8080

After the overrides-exporter starts, you can use curl to inspect the tenant overrides.

curl -s http://localhost:8080/metrics | grep cortex_overrides
# HELP cortex_overrides Resource limit overrides applied to tenants
# TYPE cortex_overrides gauge
cortex_overrides{limit_name="ingestion_burst_size",user="user1"} 350000
cortex_overrides{limit_name="ingestion_rate",user="user1"} 350000
cortex_overrides{limit_name="max_global_series_per_metric",user="user1"} 300000
cortex_overrides{limit_name="max_global_series_per_user",user="user1"} 300000
cortex_overrides{limit_name="max_local_series_per_metric",user="user1"} 0
cortex_overrides{limit_name="max_local_series_per_user",user="user1"} 0
cortex_overrides{limit_name="max_samples_per_query",user="user1"} 100000

With these metrics, you can set up alerts to know when tenants are close to hitting their limits before they exceed them.

Exposed Metrics

Prior to version v1.20.0, the exporter only exposed the fixed, hardcoded list of limits shown in the example section above.

As of version v1.20.0, the exporter automatically discovers and exposes all fields from the limit_config configuration that are numerically representable.

The exposes metrics based on the following rules:

  • Numerical types (int, int64, uint, uint64, float64) are converted directly to a float64.
  • bool types are converted to 1.0 (for true) or 0.0 (for false).
  • model.Duration types (which are internally int64) are converted to the total number of seconds as a float64 (e.g, 1h becomes 3600.0).

Fields with types such as string, slice, map, or other nested structs (like metric_relabel_configs) are ignored and not exported as metrics.