Use OpenTelemetry Collector to send metrics to Cortex

This guide explains how to configure open-telemetry collector and OTLP(OpenTelemetry Protocol) configurations in the Cortex.

Context

The open-telemetry collector can write collected metrics to the Cortex with the Prometheus and OTLP formats.

Push with Prometheus format

To push metrics via the Prometheus format, we can use prometheusremotewrite exporter in the open-telemetry collector. In the exporters and service sections in the open-telemetry collector yaml file, we can add as follows:

exporters:
  prometheusremotewrite:
    endpoint: http://<cortex-endpoint>/api/v1/push
    headers:
      X-Scope-OrgId: <orgId>

...

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [prometheusremotewrite]

Please refer to Authentication and Authorisation section for the X-Scope-OrgId explanation.

Push with OTLP format

To push metrics via the OTLP format, we can use otlphttp exporter in the open-telemetry collector. In the exporters and service sections in the open-telemetry collector yaml file, we can add as follows:

exporters:
  otlphttp:
    endpoint: http://<cortex-endpoint>/api/v1/otlp
    headers:
      X-Scope-OrgId: <orgId>

...

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [otlphttp]

Push with OTLP over gRPC

The distributor can also receive OTLP metrics over gRPC. This is experimental, and it is disabled by default. Enable it with -distributor.otlp.grpc-enabled=true. Then use the otlp exporter with the distributor gRPC server port (-server.grpc-listen-port, default 9095):

exporters:
  otlp:
    endpoint: <cortex-distributor>:9095
    compression: gzip
    tls:
      insecure: true
    headers:
      X-Scope-OrgId: <orgId>

...

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [otlp]

Notes:

  • The otlp exporter uses TLS by default. Set tls to match the Cortex gRPC server configuration.
  • The maximum request size is set by -server.grpc-max-recv-msg-size-bytes (default 4 MiB). This limit applies to all gRPC traffic of the distributor. If the Collector sends larger batches, increase the limit, or decrease the batch size in the Collector.
  • The gRPC port is often only reachable inside the cluster, with no authenticating gateway in front of it. The distributor trusts the X-Scope-OrgId metadata as it is sent. Do not expose the port to clients that you do not trust.

Cortex configurations for ingesting OTLP metrics

You can configure OTLP-related flags in the config file.

limits:
 promote_resource_attributes: <list of string>
...
distributor:
  otlp:
    convert_all_attributes: <boolean>
    disable_target_info: <boolean>
    allow_delta_temporality: <boolean>
    enable_type_and_unit_labels: <boolean>
    add_metric_suffixes: <boolean>

Ingest target_info metric

By default, the target_info is enabled to write and can be disabled via -distributor.otlp.disable-target-info=true.

Resource attributes conversion

The conversion of all resource attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true.

You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. It is supported only if -distributor.otlp.convert-all-attributes=false.

These flags can be configured via yaml:

limits:
 promote_resource_attributes: <list of string>
...
distributor:
  otlp:
    convert_all_attributes: <boolean>
    disable_target_info: <boolean>

These are the yaml examples:

  • Example 1: All of the resource attributes are converted, and the target_info metric is disabled to push.
distributor:
  otlp:
    convert_all_attributes: true
    disable_target_info: true
  • Example 2: Only service.name and service.instance.id resource attributes are converted to labels and the target_info metric is enabled to push.
limits:
 promote_resource_attributes: ["service.name", "service.instance.id"]
distributor:
  otlp:
    convert_all_attributes: false
    disable_target_info: false

Ingest delta temporality OTLP metrics

OpenTelemetry supports two temporalities, Delta and Cumulative. By default, only the cumulative metrics can be ingested via OTLP endpoint in Cortex. To enable the ingestion of OTLP metrics with delta temporality, set the distributor.otlp.allow-delta-temporality flag to true.

Enable __type__ and __unit__ label

The __type__ and __unit__ labels are added to OTLP metrics if distributor.otlp.enable-type-and-unit-labels is set to true. This flag is disabled by default.

Configure promote resource attributes per tenants

The promote_resource_attributes is a runtime config so you can configure it per tenant.

For example, this yaml file specifies attr1 being converted to label in both user-1 and user-2. But, the attr2 is converted only for user-2.

overrides:
  user-1:
    promote_resource_attributes: ["attr1"]
  user-2:
    promote_resource_attributes: ["attr1", "attr2"]
`

Configure adding suffixes to metrics

The flag add_metric_suffixes allows control to add suffixes to the metrics for name normalization. This flag is enabled by default.