Running Cortex in Production
This document assumes you have read the architecture document.
In addition to the general advice in this document, please see these platform-specific notes:
If you will run Cortex as a multi-tenant system, you need to give each tenant a unique ID - this can be any string. Managing tenants and allocating IDs must be done outside of Cortex. You must also configure Authentication and Authorisation.
Cortex requires a scalable storage back-end. Commercial cloud options are DynamoDB and Bigtable: the advantage is you don’t have to know how to manage them, but the downside is they have specific costs. Alternatively you can choose Cassandra, which you will have to install and manage.
Every Cortex installation will need Distributor, Ingester and Querier. Alertmanager, Ruler and Query-frontend are optional.
Cortex needs a KV store to track sharding of data between processes. This can be either Etcd or Consul.
If you want to configure recording and alerting rules (i.e. if you will run the Ruler and Alertmanager components) then a Postgres database is required to store configs.
Memcached is not essential but highly recommended.
Ingester replication factor
The standard replication factor is three, so that we can drop one replica and be unconcerned, as we still have two copies of the data left for redundancy. This is configurable: you can run with more redundancy or less, depending on your risk appetite.
Schema periodic table
The periodic table from argument (
using command line flags, the
from field for the first schema entry if using
YAML) should be set to the date the oldest metrics you will be sending to
Cortex. Generally that means set it to the date you are first deploying this
instance. If you use an example date from years ago table-manager will create
hundreds of tables. You can also avoid creating too many tables by setting a
reasonable retention in the table-manager
Choose schema version 9 in most cases; version 10 if you expect hundreds of thousands of timeseries under a single name. Anything older than v9 is much less efficient.
Standard choice would be Bigchunk, which is the most flexible chunk encoding. You may get better compression from Varbit, if many of your timeseries do not change value from one day to the next.
You will want to estimate how many nodes are required, how many of each component to run, and how much storage space will be required. In practice, these will vary greatly depending on the metrics being sent to Cortex.
Some key parameters are:
- The number of active series. If you have Prometheus already you
prometheus_tsdb_head_seriesto see this number.
- Sampling rate, e.g. a new sample for each series every 15 seconds. Multiply this by the number of active series to get the total rate at which samples will arrive at Cortex.
- The rate at which series are added and removed. This can be very high if you monitor objects that come and go - for example if you run thousands of batch jobs lasting a minute or so and capture metrics with a unique ID for each one. Read how to analyse this on Prometheus
- How compressible the time-series data are. If a metric stays at the same value constantly, then Cortex can compress it very well, so 12 hours of data sampled every 15 seconds would be around 2KB. On the other hand if the value jumps around a lot it might take 10KB. There are not currently any tools available to analyse this.
- How long you want to retain data for, e.g. 1 month or 2 years.
Other parameters which can become important if you have particularly high values:
- Number of different series under one metric name.
- Number of labels per series.
- Rate and complexity of queries.
Now, some rules of thumb:
- Each million series in an ingester takes 15GB of RAM. Total number
of series in ingesters is number of active series times the
replication factor. This is with the default of 12-hour chunks - RAM
required will reduce if you set
-ingester.max-chunk-agelower (trading off more back-end database IO)
- Each million series (including churn) consumes 15GB of chunk storage and 4GB of index, per day (so multiply by the retention period).
- Each 100,000 samples/sec arriving takes 1 CPU in distributors. Distributors don’t need much RAM.
If you turn on compression between distributors and ingesters (for example to save on inter-zone bandwidth charges at AWS) they will use significantly more CPU (approx 100% more for distributor and 50% more for ingester).
Cortex can retain data in-process or in Memcached to speed up various queries by caching:
- individual chunks
- index lookups for one label on one day
- the results of a whole query
You should always include Memcached in your Cortex install so results from one process can be re-used by another. In-process caching can cut fetch times slightly and reduce the load on Memcached.
Ingesters can also be configured to use Memcached to avoid re-writing index and chunk data which has already been stored in the back-end database. Again, highly recommended.
Because Cortex is designed to run multiple instances of each component (ingester, querier, etc.), you probably want to automate the placement and shepherding of these instances. Most users choose Kubernetes to do this, but this is not mandatory.
If using Kubernetes, each container should specify resource requests so that the scheduler can place them on a node with sufficient capacity.
For example an ingester might request:
resources: requests: cpu: 4 memory: 10Gi
The specific values here should be adjusted based on your own experiences running Cortex - they are very dependent on rate of data arriving and other factors such as series churn.
Take extra care with ingesters
Ingesters hold hours of timeseries data in memory; you can configure Cortex to replicate the data but you should take steps to avoid losing all replicas at once: - Don’t run multiple ingesters on the same node. - Don’t run ingesters on preemptible/spot nodes. - Spread out ingesters across racks / availability zones / whatever applies in your datacenters.
You can ask Kubernetes to avoid running on the same node like this:
affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: name operator: In values: - ingester topologyKey: "kubernetes.io/hostname"
Give plenty of time for an ingester to hand over or flush data to store when shutting down; for Kubernetes this looks like:
Ask Kubernetes to limit rolling updates to one ingester at a time, and signal the old one to stop before the new one is ready:
strategy: rollingUpdate: maxSurge: 0 maxUnavailable: 1
Ingesters provide an http hook to signal readiness when all is well; this is valuable because it stops a rolling update at the first problem:
readinessProbe: httpGet: path: /ready port: 80
We do not recommend configuring a liveness probe on ingesters - killing them is a last resort and should not be left to a machine.
Remote writing Prometheus
To configure your Prometheus instances for remote writes take a look at
the Prometheus Remote Write Config. We recommend to tune the following
parameters of the
remote_write: - queue_config: capacity: 5000 max_shards: 20 min_shards: 5 max_samples_per_send: 1000
Please take note that these values are tweaked for our use cases and may be necessary to adapt depending on your workload. Take a look at the remote write tuning docs.
If you experience a rather high delay for your metrics to appear in
Cortex (15s+) you can try increasing the
min_shards in your remote
write config. Sometimes Prometheus does not increase the number of
shards even though it hasn’t caught up the lag. You can monitor the
delay with this Prometheus query:
time() - sum by (statefulset_kubernetes_io_pod_name) (prometheus_remote_storage_queue_highest_sent_timestamp_seconds)
These ingester options reduce the chance of storing multiple copies of the same data:
Add a chunk cache via
-memcached.hostname to allow writes to be de-duplicated.
As recommended under Chunk encoding, use Bigchunk:
-ingester.chunk-encoding=3 # bigchunk