Use OpenTelemetry Collector to send metrics to Cortex

This guide explains how to configure open-telemetry collector and OTLP(OpenTelemetry Protocol) configurations in the Cortex.

Context

The open-telemetry collector can write collected metrics to the Cortex with the Prometheus and OTLP formats.

Push with Prometheus format

To push metrics via the Prometheus format, we can use prometheusremotewrite exporter in the open-telemetry collector. In the exporters and service sections in the open-telemetry collector yaml file, we can add as follows:

exporters:
  prometheusremotewrite:
    endpoint: http://<cortex-endpoint>/api/v1/push
    headers:
      X-Scope-OrgId: <orgId>

...

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [prometheusremotewrite]

Please refer to Authentication and Authorisation section for the X-Scope-OrgId explanation.

Push with OTLP format

To push metrics via the OTLP format, we can use otlphttp exporter in the open-telemetry collector. In the exporters and service sections in the open-telemetry collector yaml file, we can add as follows:

exporters:
  otlphttp:
    endpoint: http://<cortex-endpoint>/api/v1/otlp
    headers:
      X-Scope-OrgId: <orgId>

...

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [otlphttp]

Cortex configurations for ingesting OTLP metrics

You can configure OTLP-related flags in the config file.

limits:
 promote_resource_attributes: <list of string>
...
distributor:
  otlp:
    convert_all_attributes: <boolean>
    disable_target_info: <boolean>
    allow_delta_temporality: <boolean>
    enable_type_and_unit_labels: <boolean>

Ingest `target_info` metric

By default, the target_info is enabled to write and can be disabled via -distributor.otlp.disable-target-info=true.

Resource attributes conversion

The conversion of all resource attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true.

You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. It is supported only if -distributor.otlp.convert-all-attributes=false.

These flags can be configured via yaml:

limits:
 promote_resource_attributes: <list of string>
...
distributor:
  otlp:
    convert_all_attributes: <boolean>
    disable_target_info: <boolean>

These are the yaml examples:

Example 1: All of the resource attributes are converted, and the target_info metric is disabled to push.

distributor:
  otlp:
    convert_all_attributes: true
    disable_target_info: true

Example 2: Only service.name and service.instance.id resource attributes are converted to labels and the target_info metric is enabled to push.

limits:
 promote_resource_attributes: ["service.name", "service.instance.id"]
distributor:
  otlp:
    convert_all_attributes: false
    disable_target_info: false

Ingest delta temporality OTLP metrics

OpenTelemetry supports two temporalities, Delta and Cumulative. By default, only the cumulative metrics can be ingested via OTLP endpoint in Cortex. To enable the ingestion of OTLP metrics with delta temporality, set the distributor.otlp.allow-delta-temporality flag to true.

Enable `type` and `unit` label

The __type__ and __unit__ labels are added to OTLP metrics if distributor.otlp.enable-type-and-unit-labels is set to true. This flag is disabled by default.

Configure promote resource attributes per tenants

The promote_resource_attributes is a runtime config so you can configure it per tenant.

For example, this yaml file specifies attr1 being converted to label in both user-1 and user-2. But, the attr2 is converted only for user-2.

overrides:
  user-1:
    promote_resource_attributes: ["attr1"]
  user-2:
    promote_resource_attributes: ["attr1", "attr2"]
`