Cortex – Blog

Blog: Introducing READONLY State: Gradual and Safe Ingester Scaling

Fri, 17 Oct 2025 00:00:00 +0000

Introduction

Scaling down ingesters in Cortex has traditionally been a complex and risky operation. The conventional approach required setting querier.query-store-after=0s, which forces all queries to hit storage directly, significantly impacting performance. With Cortex 1.19.0, we introduced a new READONLY state for ingesters that changes how you can safely scale down your Cortex clusters.

Why Traditional Scaling Falls Short

The legacy approach to ingester scaling had several issues:

Performance Impact: Setting querier.query-store-after=0s forces all queries to bypass ingesters entirely, increasing query latency and storage load.

Operational Complexity: Traditional scaling required coordinating configuration changes across multiple components, precise timing, manual monitoring of bucket scanning intervals, and scaling ingesters one by one with waiting periods between each shutdown.

Risk of Data Loss: Without proper coordination, scaling down could result in data loss if in-memory data wasn’t properly flushed to storage before ingester termination.

What is the READONLY State?

The READONLY state addresses these challenges. When an ingester transitions to READONLY state:

Stops accepting new writes - Push requests are rejected and redistributed to ACTIVE ingesters
Continues serving queries - Existing data remains available, maintaining query performance
Gradually ages out data - Data naturally expires according to your retention settings
Enables safe removal - Ingesters can be terminated once data has aged out

How to Use READONLY State

Step 1: Transition to READONLY

# Set multiple ingesters to READONLY simultaneously
curl -X POST "http://ingester-1:8080/ingester/mode" -d 'mode=READONLY'
curl -X POST "http://ingester-2:8080/ingester/mode" -d 'mode=READONLY'
curl -X POST "http://ingester-3:8080/ingester/mode" -d 'mode=READONLY'

Step 2: Monitor Data Status (Optional)

# Check user statistics and loaded blocks on the ingester
curl http://ingester-1:8080/ingester/all_user_stats

Step 3: Choose Removal Strategy

You have three options:

Immediate removal: Safe for service availability but may impact query performance
Conservative removal: Wait for querier.query-ingesters-within duration (recommended)
Complete data aging: Wait for full retention period

Step 4: Remove Ingesters

# Terminate the ingester processes
kubectl delete pod ingester-1 ingester-2 ingester-3

Timeline Example

For a cluster with querier.query-ingesters-within=5h:

T0: Set ingesters to READONLY state
T1: Ingesters stop receiving new data but continue serving queries
T2 (T0 + 5h): Ingesters no longer receive query requests (safe to remove)
T3 (T0 + retention_period): All blocks naturally removed from ingesters

Any time after T2 is safe for removal without service impact.

Benefits

Performance Preservation

Unlike the traditional approach, READONLY ingesters continue serving queries, maintaining performance during the scaling transition.

Operational Simplicity

No configuration changes required across multiple components
Batch operations supported - multiple ingesters can transition simultaneously (no more “one by one” requirement)
No waiting periods between ingester transitions
Flexible timing - remove ingesters when convenient
Reversible operations - ingesters can return to ACTIVE state if needed

Enhanced Safety

Gradual data aging without manual intervention
Data remains available during transition
Monitoring capabilities with /ingester/all_user_stats endpoint

Practical Examples

Basic READONLY Scaling

#!/bin/bash
INGESTERS_TO_SCALE=("ingester-1" "ingester-2" "ingester-3")
WAIT_DURATION="5h"

# Set ingesters to READONLY
for ingester in "${INGESTERS_TO_SCALE[@]}"; do
 echo "Setting $ingester to READONLY..."
 curl -X POST http://$ingester:8080/ingester/mode -d 'mode=READONLY'
done

# Wait for safe removal window
echo "Waiting $WAIT_DURATION for safe removal..."
sleep $WAIT_DURATION

# Remove ingesters
for ingester in "${INGESTERS_TO_SCALE[@]}"; do
 echo "Removing $ingester..."
 kubectl delete pod $ingester
done

Advanced: Check for Empty Users Before Removal

#!/bin/bash
check_ingester_ready() {
 local ingester=$1
 local response=$(curl -s http://$ingester:8080/ingester/all_user_stats)

 # Empty array "[]" indicates no users/data remaining
 if [[ "$response" == "[]" ]]; then
 return 0 # Ready for removal
 else
 return 1 # Still has user data
 fi
}

INGESTERS_TO_SCALE=("ingester-1" "ingester-2" "ingester-3")

# Set ingesters to READONLY
for ingester in "${INGESTERS_TO_SCALE[@]}"; do
 echo "Setting $ingester to READONLY..."
 curl -X POST http://$ingester:8080/ingester/mode -d 'mode=READONLY'
done

# Wait and check for data removal
for ingester in "${INGESTERS_TO_SCALE[@]}"; do
 echo "Waiting for $ingester to be ready for removal..."
 while ! check_ingester_ready $ingester; do
 echo "$ingester still has user data, waiting 30s..."
 sleep 30
 done

 echo "Removing $ingester (no user data remaining)..."
 kubectl delete pod $ingester
done

Best Practices

Test in non-production first to validate the process with your configuration
Scale gradually - don’t remove too many ingesters simultaneously
Monitor throughout - watch metrics during the entire process
Understand your query patterns - know your querier.query-ingesters-within setting

Emergency Rollback

If issues arise, return ingesters to ACTIVE state:

# Revert to ACTIVE state
curl -X POST "http://ingester-1:8080/ingester/mode" -d 'mode=ACTIVE'

Conclusion

The READONLY state improves Cortex’s operational capabilities. This feature makes scaling operations safer, simpler, more flexible, and more performant than the traditional approach. Configuration changes across multiple components are no longer required - set ingesters to READONLY and remove them when convenient.

For detailed information and examples, check out our Ingesters Scaling Guide.

Blog: Query Priority in Cortex

Mon, 08 Sep 2025 00:00:00 +0000

Introduction

In high-scale monitoring environments, not all queries are created equal. Some queries power critical dashboards that need sub-second response times, while others are exploratory analytics that can tolerate delays. However, the queries from a tenant is handled FIFO (first-in-first-out), which could lead to a noisy-neighbor problem within the user.

Cortex’s query priority feature addresses this challenge by allowing operators to reserve querier resources for high-priority queries.

What is Query Priority?

Query priority in Cortex enables you to classify queries based on various attributes and allocate dedicated querier resources to different priority levels. The system works by:

Matching queries against configurable attributes (regex patterns, time ranges, API types, user agents, dashboard UIDs)
Assigning priority levels to matched queries
Reserving querier capacity as a percentage for each priority level

Configuration Example

query_priority:
enabled: true
default_priority: 0
priorities:
- priority: 100
reserved_queriers: 0.1 # Reserve 10% of queriers
query_attributes:
- regex: ".*alert.*" # Alert queries
- priority: 50
reserved_queriers: 0.05 # Reserve 5% of queriers
query_attributes:
- api_type: "query_range"
time_range_limit:
max: "1h" # Dashboard queries (short range)
user_agent_regex: "Grafana.*"

Benefits

1. Preventing Resource Starvation

The most compelling use case is protecting critical queries from resource-hungry analytical workloads. Without priority, a few expensive queries scanning months of data can starve dashboard queries, causing user-facing alerts to timeout.

2. SLA Differentiation

Organizations can offer different service levels:

Tier 1: Real-time dashboards and alerts (high priority)
Tier 2: Business intelligence queries (medium priority)
Tier 3: Ad-hoc exploration and data exports (low priority)

Drawbacks

1. Resource Underutilization

Reserved queriers sit idle when high-priority queries aren’t running. If you reserve 30% capacity for dashboard queries that only use 10% during off-peak hours, you’re wasting 20% of your infrastructure.

2. Configuration Complexity

Query attributes require careful tuning:

Regex patterns can be brittle and hard to maintain
Time window matching needs constant adjustment as usage patterns evolve
Dashboard UID matching breaks when dashboards are recreated

Best Practices

When to use query priority

High query volume with mixed workload types
Clear SLA requirements that justify the complexity
Stable query patterns that won’t require frequent reconfiguration

When to avoid it:

Homogeneous workloads where all queries have similar requirements
Unstable environments where query patterns change frequently

Conclusion

If your users have different SLA requirements depending on their query patterns that’s consistent, this is a power feature that helps towards meeting expected availability and performance for queries. There is also plenty of room for this logic to be improved in the future, such that it self-adapts to the query pattern of the users and dynamically assign priorities to balance SLAs with fairness across different query patterns.

Blog: Efficient Query Parallelism in Cortex with Dynamic Query Splitting

Tue, 12 Aug 2025 00:00:00 +0000

Introduction

Cortex traditionally relied on static query splitting and static vertical sharding to optimize the execution of long-range PromQL queries. Static query splitting divides a query into fixed time intervals, while vertical sharding—when applicable—splits the query across subsets of time series. These techniques offered improved parallelism and reduced query latency but were limited by their one-size-fits-all approach. They did not account for differences in query range, lookback behavior, and cardinality—leading to inefficiencies like over-sharding, redundant data fetches, and storage pressure in large or complex queries.

To address those gaps, Cortex introduced dynamic query splitting and dynamic vertical sharding—two adaptive mechanisms that intelligently adjust how queries are broken down based on query semantics.

This article explores the motivations behind this evolution, how the dynamic model works, and how to configure it for more efficient scalable PromQL query execution.

Query Splitting

Query splitting breaks a single long-range query into smaller subqueries based on a configured time interval. For example, given this configuration, a 30-day range query will be split into 30 individual 1-day subqueries:

query_range:
split_queries_by_interval: 24h

These subqueries are processed in parallel by different queriers, and the results are merged at query-frontend before returning to client. This improved performance for long range queries and helped prevent timeouts and resource exhaustion.

Vertical Sharding

Unlike query splitting, which divides a query over time intervals, vertical sharding divides a query across shards of the time series data itself. Each shard processes only a portion of the matched series, reducing memory usage and computational load per querier. This is especially useful for high-cardinality queries where the number of series can reach hundreds of thousands or more.

limits:
query_vertical_shard_size: 4

For example, suppose the label selector in the following query http_requests_total{job="api"} matches 500,000 distinct time series, each corresponding to different combinations of labels like instance, path, status, and method.

sum(rate(http_requests_total{job="api"}[5m])) by (instance)

Without vertical sharding, a single querier must fetch and aggregate all 500,000 series. With vertical sharding enabled and configured to 4, the query would be split into 4 shards each processing ~125,000 series. The results are finally merged at query frontend before returning to client.

Introducing Dynamic Query Splitting

Cortex dynamic query splitting was introduced to address the limitations of static configurations. Instead of applying a fixed split interval uniformly across all queries, the dynamic logic computes an optimal split interval per query—as a multiple of the configured base interval—based on query semantics and configurable constraints. If dynamic vertical sharding is also enabled, then both split interval and vertical shard size will be dynamically adjusted for every query.

The goal is to maximize query parallelism through both horizontal splitting and vertical sharding, while staying within safe and configurable limits that prevent system overload or inefficiency. This is best explained by how dynamic splitting solves two problems:

1. Queuing and Merge Bottlenecks from Over-Splitting

While increasing parallelism through splitting and sharding is generally beneficial, it becomes counterproductive when the number of subqueries or shards far exceeds the number of available queriers.

The number of splits when using a fixed interval increases with the query’s time range, which is under the user’s control. For example:

With a static split interval of 24 hours, a 7-day query results in 7 horizontal splits
If the user increased query range to 100 days, the query is split into 100 horizontal splits.
If vertical sharding is also enabled and configured to use 5 shards (query_vertical_shard_size: 5), each split is further divided—leading to a total of 500 individual shards.

When the number of shards grows too large:

Backend workers become saturated, causing subqueries to queue and increasing total latency.
The query-frontend merges hundreds of partial results, which introduces overhead that can outweigh the benefits of parallelism.
Overall system throughput degrades, especially when multiple large queries are executed concurrently.

To solve this issue, a new configuration max_shards_per_query is introduced for the maximum parallelism per query. Given the same example above:

With a static split interval of 24 hours and max_shards_per_query set to 75, a 7-day query still results in 7 splits.
If the user increased query range to 100 days, the dynamic splitting algorithm will adjust the split interval to be 48 hours, producing 50 horizontal splits—keeping the total within the target of 75 shards.
If vertical sharding is enabled and configured to use up to 5 shards, the dynamic logic selects the optimal combination of split interval and vertical shards to maximize parallelism without exceeding 75 shards.
- In this case, a 96-hour (4-day) split interval with 3 vertical shards yields exactly 75 total shards—the most efficient combination.
- Note: enable_dynamic_vertical_sharding must be set to true; otherwise, only the split interval will be adjusted.

In summary, with dynamic splitting enabled, you can define a target total number of shards, and Cortex will automatically adjust time splitting and vertical sharding to maximize parallelism without crossing that limit.

2. Parallelism Cost with Query Lookback

In PromQL, some functions like rate(), increase(), or max_over_time() use a lookback window, meaning each query must fetch samples from before the evaluation timestamp to execute.

Consider the following query that calculates the maximum container memory usage over a 90-day lookback window:

max_over_time(container_memory_usage_bytes{cluster="prod", namespace="payments"}[90d])

Suppose this query is evaluated over a 30-day range and static query splitting is configured to split it into 1-day intervals. This produces 30 subqueries, each corresponding to a single day. However, due to the [90d] range vector:

Each subquery must fetch the full 90 days of historical data to evaluate correctly.
The same data blocks are repeatedly fetched across all subqueries
The total duration of data fetched is query range + (lookback window x total shards), which results in 30 + 90 x 30 = 2,730 days.

As a result, store-gateways must handle a large amount of mostly redundant reads, repeatedly fetching data blocks for each subquery. This puts additional pressure on the storage layer, and the cumulative effect slows down query execution and degrades overall system performance. In this case, splitting the query further doesn’t reduce backend load—it amplifies it.

With dynamic splitting, you can define a target max_fetched_data_duration_per_query—the maximum cumulative duration of historical data that a single query is allowed to fetch. If the lookback window is long, the algorithm automatically increases the split interval or reduces the vertical shard size to lower the shard count and protect the storage layer.

For example, with max_fetched_data_duration_per_query set to 500d:

A larger split interval of 120-hour (5-day) is used to split the query into 5 splits.
- This is the optimal split interval that results in highest parallelism without crossing the limit of 500 days fetched.
The total duration fetched from storage layer becomes 30 + 90 x 5 = 480 days—lower than the target of 500 days.

How to Configure Dynamic Query Splitting

Dynamic query splitting is configured under the dynamic_query_splits section in the query_range block of Cortex’s configuration. Keep in mind that it works in conjunction with static split_queries_by_interval and query_vertical_shard_size which are required to be configured as well:

Dynamic query splitting considers the following configurations:

max_shards_per_query: Defines the maximum number of total shards (horizontal splits × vertical shards) that a single query can generate. If enable_dynamic_vertical_sharding is set, the dynamic logic will adjust both the split interval and vertical shard size to find the most effective combination that results in the highest degree of sharding without exceeding this limit.
max_fetched_data_duration_per_query: Sets a target for the maximum total time duration of data that can be fetched across all subqueries. To keep the duration fetched below this target, a larger split interval and/or less vertical sharding is used. This is especially important for queries with long lookback windows, where excessive splitting can lead to redundant block reads, putting pressure on store-gateways and the storage layer.
enable_dynamic_vertical_sharding: When enabled, vertical sharding becomes dynamic per query. Instead of using a fixed shard count, an optimal vertical shard size ranging from 1 (no sharding) to the tenant’s configured query_vertical_shard_size will be used.

Example Configuration

Let’s explore how two different queries will be handled given the following configuration:

query_range:
split_queries_by_interval: 24h
dynamic_query_splits:
max_shards_per_query: 100
max_fetched_data_duration_per_query: 8760h # 365 day
enable_dynamic_vertical_sharding: true
limits:
query_vertical_shard_size: 4

Query #1

sum by (pod) (
rate(container_cpu_usage_seconds_total{namespace="prod"}[1m])
)

Query time range: 60 days
Lookback window: 1 minute

Since the query has a short lookback window of 1 min, the total duration of data fetched by each shard is not going to be limiting factor. The limiting factor to consider here is maintaining less than 100 total shards. Both dynamic splitting and dynamic vertical sharding are enabled. Cortex finds the most optimal combination that results in the highest number of shards below 100. In this case:

Number of splits by time: 30 (2 day interval)
Vertical shard size: 3
Total shards: 90

Query #2

sum by (pod) (
max_over_time(container_memory_usage_bytes{namespace="prod"}[30d])
)

Query time range: 14 days
Lookback window: 30 days

This query can be split into 14 splits and sharded vertically by 4 resulting in a total of 56 shards, which is below the limit of 100 total shards. However, since each shard is going to have to fetch all 30 days of the lookback window to evaluate in addition to the interval itself, this would result in 56 shards each fetching 31 days of data for a total of 1736 days. This is not optimal and will cause a heavy load on the backend storage layer.

Luckily we configured max_fetched_data_duration_per_query to be 365 days. This will limit query sharding to achieve highest parallelism without crossing the duration fetched limit. In this case:

Number of splits by time: 5 (3 day interval)
Vertical shard size: 2
Total shards: 10

The total duration of data fetched for query evaluation is calculated using (interval + lookback window) x total shards. In this case (3 + 30) x 10 = 330 days fetched, which is below our limit of 365 days.

Conclusion

Dynamic query splitting and vertical sharding make Cortex smarter about how it executes PromQL queries. By adapting to each query’s semantics and backend constraints, Cortex avoids the limitations of static configurations—enabling efficient parallelism across diverse query patterns and consistently delivering high performance at scale.

Blog: Block the Blast: How Query Rejection Protects Your Cortex Cluster

Mon, 04 Aug 2025 00:00:00 +0000

Introduction

Although Cortex includes various safeguards to protect against overload, they can’t prevent every failure scenario. In some environments, a small set of seemingly harmless-looking dashboard queries have repeatedly slipped just under the limits yet still OOM-killed the querier pods. Built-in protections weren’t enough, and the only available option was to throttle all incoming traffic. These queries often came from a specific dashboard or followed a predictable pattern. There was no way to block just those without affecting everything else. This inspired the introduction of query rejection, a last-resort safety net for operators running multi-tenant Cortex clusters.

Why Limits Aren’t Enough

Cortex already includes resource limiting, throttling and other resource safeguards, but these protections can’t cover every edge case. Some limits are enforced too late in the query lifecycle to matter; others are broad and can’t target specific bad actors. As a result, a single heavy query can bypass all service limits and still:

Cause OOM kills that disrupt other tenants.
Degrade availability or spike latency for everyone.
Require manual operator intervention to restore normal service.

We needed a more precise tool—one that lets operators proactively block specific query patterns without harming legitimate traffic.

What Is Query Rejection?

Think of query rejection as an “emergency stop” in a factory. It sits in front of the query engine and checks each request against a set of operator-defined rules. If a request matches criteria, it’s rejected immediately. This allows you to target the handful of queries that cause trouble without imposing a blanket slowdown on everyone else.

Key features:

Per-tenant control: It’s defined in the tenant limit configuration, which only targets queries from specific tenant.
Precise matching: You can specify different query attributes to narrow down to specific queries. All fields within a rejection rule must match (AND logic). If needed, you can define multiple independent rejection rules to target different types of queries.
Pre-processing enforcement: Query rejection is applied before the query is executed, allowing known-bad patterns to be blocked before consuming any resources.

Matching Criteria

Heavy queries often share identifiable traits. Query rejection lets you match on a variety of attributes and reject only those requests. You can combine as many of these as needed:

API type: query, query_range, series, etc.
Query string (regex): Match by pattern, e.g., any query containing “ALERT”.
Time range: Match queries whose range falls between a configured min and max.
Time window: Match queries based on how far their time window is from now by specifying relative min and max boundaries. This is often used to distinguish queries that hit hot storage versus cold storage.
Step value (resolution): Block extremely fine resolutions (e.g., steps under 5s).
Headers: Match by User-Agent, Grafana dashboard UID (X-Dashboard-Uid) or panel ID (X-Panel-Id).

By combining these fields, you can zero in on the exact query patterns causing problems without over-blocking.

Configuring Query Rejection

You define query rejection rules per tenant in a runtime config file. Each rejection rule specifies a set of attributes that must all match for the query to be rejected. The configuration supports multiple such rules.

Here’s an example configuration:

# runtime_config.yaml
overrides:
 <tenant_id>:
 query_rejection:
 enabled: true
 query_attributes:
 - api_type: query_range
 regex: .*ALERT.*
 query_step_limit:
 min: 6s
 max: 20s
 dashboard_uid: "dash123"

What this does:

enabled This allows you to temporarily turn off query rejection without removing the configuration. It can help verify whether previously blocked queries are still causing issues.
query_attributes is a list of rejection rules. A query will only be rejected if it matches all attributes within one rejection rule.

In the example above, the single rejection rule requires the following:

API type must be query_range.
The query string must contain the word ALERT.
The step must be between 6 and 20 seconds.
The request must come from a dashboard with UID dash123.

If all of these conditions match, the request is rejected with a 422 response. If even one condition doesn’t match, the query is allowed to run. You can define additional rejection rules in the list to target different patterns.

Practical Example

Imagine a dashboard panel that repeatedly hits your cluster with a query like this:

curl \
 'http://localhost:8005/prometheus/api/v1/query?query=customALERTquery&start=1718383304&end=1718386904&step=7s' \
 -H "User-Agent: other" \
 -H "X-Dashboard-Uid: dash123"

Because this request matches all the configured attributes, it will be blocked. But if even one field is different—such as a longer step duration or a different dashboard UID—the query will go through.

Best Practices and Cautions

Start with narrow rules. Use the most specific fields first—like panel ID or dashboard UID—to reduce risk of over-blocking.
Monitor rejections. Use the cortex_query_frontend_rejected_queries_total metric to track rejected queries, and check logs for detailed query information.
Communicate with tenants. Let affected tenants know if their queries are being blocked, and help them adjust their dashboards accordingly.

Ruler Queries

Query rejection only applies to API queries and does not apply to ruler queries. However, Ruler queries are typically instant and lightweight, so a complex query‑rejection mechanism isn’t required for them. In situations where a rule group contains heavy queries and no other mitigations are effective, operators can disable the entire rule group. This is especially helpful to prevent Rulers from getting OOMKilled due to resource-intensive rules.

Rule group disabling is configured per tenant, similar to query rejection. When you disable a rule group, Cortex stops evaluating the rules within that group, removing the problematic queries altogether. For example:

# runtime_config.yaml
overrides:
 <tenant_id>:
 disabled_rule_groups:
 - namespace: "keep_firing_for_test"
 name: "smallsteps"

This makes it easy to mitigate issues from the ruler without introducing query rejection logic for those queries.

Conclusion

When traditional safeguards fall short, query rejection gives operators precise control to block only what’s harmful; without slowing down everything else.

If you operate a shared Cortex environment, consider learning how to use query rejection effectively. It might just save you from the next incident; by preventing OOM kills, degraded performance, or disruption to other tenants.

Blog: Optimizing PromQL queries: A deep dive

Tue, 29 Apr 2025 00:00:00 +0000

Introduction

This guide explains how Cortex evaluates PromQL queries, details how time series data is stored and retrieved, and offers strategies to write performant queries — particularly in high-cardinality environments.

Note: If you are new to PromQL, it is recommended to start with the Querying basics documentation.

Prometheus Concepts

Data Model

Prometheus employs a straightforward data model:

Each time series is uniquely identified by a metric name and a set of label-value pairs.
Each sample includes:
- A millisecond precision timestamp
- A 64 bit floating point value.

Label Matchers

Label matchers define the selection criteria for time series within the TSDB. Consider the following PromQL expression:

http_requests_total{cluster="prod", job="envoy"}

the label matchers are:

__name__="http_requests_total"
cluster="prod"
job="envoy"

Prometheus supports four types of label matchers:

Type	Syntax	Example
Equal	label=“value”	job=“envoy”
Not Equal	label!=“value”	job!=“prometheus”
Regex Equal	label=~“regex”	job=~“env.*”
Regex Not Equal	label!~“regex”	status!~“4..”

Time Series Storage in Cortex

Cortex uses Prometheus’s Time Series Database (TSDB) for storing time series data. The Prometheus TSDB is time partitioned into blocks. Each TSDB block is made up of the following files:

ID - ID of the block (ULID)
meta.json - Contains the metadata of the block
index - A binary file that contains the index
chunks - Directory containing the chunk segment files

More details: TSDB format docs

Index File

The index file contains two key mappings for query processing:

Postings Offset Table and Postings: Maps label-value pairs to Series IDs
Series Section: Maps series IDs to label sets and chunk references

Example

Given the following time series:

http_requests_total{cluster="prod", job="envoy", status="200"} -> SeriesID(1)
http_requests_total{cluster="prod", job="envoy", status="400"} -> SeriesID(2)
http_requests_total{cluster="prod", job="envoy", status="500"} -> SeriesID(3)
http_requests_total{cluster="prod", job="prometheus", status="200"} -> SeriesID(4)

The index file would store mappings such as:

__name__=http_requests_total → [1, 2, 3, 4]
cluster=prod → [1, 2, 3, 4]
job=envoy → [1, 2, 3]
job=prometheus → [4]
status=200 → [1, 4]
status=400 → [2]
status=500 → [3]

Chunks

Each chunk segment file can store up to 512MB of data. Each chunk in the segment file typically holds up to 120 samples.

Query Execution in Cortex

To optimize PromQL queries effectively, it is essential to understand how queries are executed within Cortex. Consider the following example:

sum(rate(http_requests_total{cluster="prod", job="envoy"}[5m]))

Block Selection

Cortex first identifies the TSDB blocks that fall within the query’s time range. This process is very fast in Cortex and will not add a huge overhead on query execution.

Series Selection

Next, Cortex uses the inverted index to retrieve the set of matching series IDs for each label matcher. For example:

__name__="http_requests_total" → [1, 2, 3, 4]
cluster="prod" → [1, 2, 3, 4]
job="envoy" → [1, 2, 3]

The intersection of these sets yields:

http_requests_total{cluster=“prod”, job=“envoy”, status=“200”}
http_requests_total{cluster=“prod”, job=“envoy”, status=“400”}
http_requests_total{cluster=“prod”, job=“envoy”, status=“500”}

Sample Selection

The mapping from series to chunks is used to identify the relevant chunks from the chunk segment files. These chunks are decoded to retrieve the underlying time series samples.

PromQL evaluation

Using the retrieved series and samples, the PromQL engine evaluates the query. There are two modes of running queries:

Instant queries – Evaluated at a single timestamp
Range queries – Evaluated at regular intervals over a defined time range

Common Causes of Slow Queries and Optimization Techniques

Several factors influence the latency and resource usage of PromQL queries. This section highlights the key contributors and practical strategies for improving performance.

Query Cardinality

High cardinality increases the number of time series that must be scanned and evaluated.

Recommendations

Eliminate unnecessary labels from metrics.
Use selective label matchers to reduce the number of series returned.

Number of samples processed

The number of samples fetched impacts both memory usage and CPU time for decoding and processing.

Recommendations

Until downsampling is implemented, reducing the scrape interval can help lower the amount of samples to be processed. But this comes at the cost of reduced resolution.

Number of evaluation steps

The number of evaluation steps for a range query is computed as:

num of steps = 1 + (end - start) / step

Example: A 24-hour query with a 1-minute step results in 1,441 evaluation steps.

Recommendations

Grafana can automatically set the step size based on the time range. If a query is slow, manually increasing the step parameter can reduce computational overhead.

Time range of the query

Wider time ranges amplify the effects of cardinality, sample volume, and evaluation steps.

Recommendations

Use shorter time ranges (e.g., 1h) in dashboards.
Default to instant queries during metric exploration to reduce load.

Query Complexity

Subqueries, nested expressions, and advanced functions may lead to substantial CPU consumption.

Recommendations

Simplify complex expressions where feasible.

Regular Expressions

While Prometheus has optimized regex matching, such queries remain CPU-intensive.

Recommendations

Avoid regex matchers in high-frequency queries.
Where possible, use equality matchers instead.

Query Result Size

Queries returning large datasets (>100MB) can incur significant serialization and network transfer costs.

Example

pod_container_info #No aggregation
sum by (pod) (rate(container_cpu_seconds_total[1m])) # High cardinality result

Recommendations

Scoping the query using additional label matchers reduces result size and improves performance.

Summary

The key optimization techniques are:

Use selective label matchers to limit cardinality.
Increase the step value in long-range queries.
Simplify complex or nested PromQL expressions.
Avoid regex matchers unless strictly necessary.
Favor instant queries for interactive use cases.
Scope queries to minimize the result size.

Blog: Introducing the Cortex Blog: Sharing Our Journey

Sat, 26 Apr 2025 00:00:00 +0000

Welcome to the official Cortex blog!

We’re kicking things off here to share updates, best practices, technical deep-dives, and community highlights around everything Cortex. Whether you’re operating a Cortex cluster, integrating it into your observability platform, or just starting to explore scalable time-series databases — you’re in the right place.

In the coming weeks, you can expect posts on:

Real-world Cortex deployment strategies and lessons learned
Tips for running Cortex efficiently at scale
Deep dives into key Cortex concepts like blocks storage, ruler sharding, and query federation
Guides to help new contributors get involved with the project
Interviews with maintainers and users from across the community
Roadmap insights and upcoming features we’re excited about

Cortex has grown a lot thanks to a vibrant community of operators, contributors, and partners. This blog will be another space for us to connect, learn from each other, and push the project even further.

Stay tuned — the first technical post is coming soon!

If there’s a topic you’d love to see covered, feel free to reach out or open a discussion in our Cortex community forums.

Thanks for being part of the journey! 🚀

— The Cortex Team