config-yamlCLI flag). This is the same schema config the queriers consume. The schema is only required to use this option.
server.grpc-max-concurrent-streams(for both query-frontends and queriers)
config_storefield in favor of a nested
frontend.cache-split-intervalin favor of
querier.split-queries-by-intervalboth to reduce configuration complexity and guarantee alignment of these two configs. Starting from now,
-querier.cache-resultsmay only be enabled in conjunction with
-querier.split-queries-by-interval(previously the cache interval default was
24hso if you want to preserve the same behaviour you should set
-ingester.normalise-tokens=false), such ingesters will now be completely invisible to distributors and need to be either switched to Cortex 0.6.0 or later, or be configured to use normalised tokens. #2034
--store.min-chunk-ageto the Querier config as
--querier.query-store-after, allowing the store to be skipped during query time if the metrics wouldn’t be found. The YAML config option
ingestermaxquerylookbackhas been renamed to
query_ingesters_withinto match its CLI flag. #1893
--store.min-chunk-agehas been removed
--querier.query-store-afterhas been added in it’s place.
-querier.max-concurrentworking, Active Query Tracker is enabled by default, and is configured to store its data to
active-query-trackerdirectory (relative to current directory when Cortex started). This can be changed by using
-querier.active-query-tracker-diroption. Purpose of Active Query Tracker is to log queries that were running when Cortex crashes. This logging happens on next Cortex start. #2088
-experimental.ruler.enable-apito enable the ruler api which implements the Prometheus API
/api/v1/alertsendpoints under the configured
cortex_alertmanager_configsmetric to gauge the number of valid and invalid configs. #2125
custom_authenticatorsconfig option that allows users to authenticate with cassandra clusters using password authenticators that are not approved by default in gocql #2093
-experimental.tsdb.bucket-store.tenant-sync-concurrencyto configure the maximum number of concurrent tenants for which blocks are synched. #2026
cortex_<component>_thanos_objstore_, component being one of
retry_max_backoffconfiguration options to enable retrying recoverable errors. #2054
v1.3.0to support PostgreSQL SCRAM-SHA-256 authentication. #2097
<all keyspaces>if given keyspace exists. #2032
password_fileconfiguration options to enable reading Cassandra password from file. #2096
-memberlist.dead-node-reclaim-timeoptions to control how memberlist library handles dead nodes and name reuse. #2131
/api/prom/user_statsendpoints when using the experimental TSDB blocks storage. #2042
invalid chunk checksumerrors. #2074
cortex_overrides_last_reload_successfulis now only exported by components that use a
RuntimeConfigManager. Previously, for components that do not initialize a
RuntimeConfigManager(such as the compactor) the gauge was initialized with 0 (indicating error state) and then never updated, resulting in a false-negative permanent error state. #2092
urlin config-file-reference. #2148
Cortex 0.4.0 is the last version that can write denormalised tokens. Cortex 0.5.0 and above always write normalised tokens.
Cortex 0.6.0 is the last version that can read denormalised tokens. Starting with Cortex 0.7.0 only normalised tokens are supported, and ingesters writing denormalised tokens to the ring (running Cortex 0.4.0 or earlier with
-ingester.normalise-tokens=false) are ignored by distributors. Such ingesters should either switch to using normalised tokens, or be upgraded to Cortex 0.5.0 or later.
Note that the ruler flags need to be changed in this upgrade. You’re moving from a single node ruler to something that might need to be sharded. Further, if you’re using the configs service, we’ve upgraded the migration library and this requires some manual intervention. See full instructions below to upgrade your PostgreSQL.
Cache-Controlheader and if one of its values is
ruler.configs.client-timeoutin order to match
-ruler.group-timeouthas been removed.
-ruler.num-workershas been removed.
-ruler.rule-pathhas been added to specify where the prometheus rule manager will sync rule files.
-ruler.storage.typehas beem added to specify the rule store backend type, currently only the configdb.
-ruler.poll-intervalhas been added to specify the interval in which to poll new rule groups.
-ruler.evaluation-intervaldefault value has changed from
1mto match the default evaluation interval in Prometheus.
-ingester.normalise-tokensis now deprecated, and ignored. If you want to switch back to using denormalised tokens, you need to downgrade to Cortex 0.4.0. Previous versions don’t handle claiming tokens from normalised ingesters correctly. #1809
-runtime-config.file(defaults to empty) and
-runtime-config.reload-period(defaults to 10 seconds), which replace previously used
-limits.per-user-override-periodoptions. Old options are still used if
-runtime-config.fileis not specified. This change is also reflected in YAML configuration, where old
limits.per_tenant_override_periodfields are replaced with
ha-tracker/in order to not clash with other keys (ie. ring) stored in the same key-value store. #1940
--ingester.wal-enabled: Setting this to
trueenables writing to WAL during ingestion.
--ingester.wal-dir: Directory where the WAL data should be stored and/or recovered from.
--ingester.checkpoint-enabled: Set this to
trueto enable checkpointing of in-memory chunks to disk.
--ingester.checkpoint-duration: This is the interval at which checkpoints should be created.
--ingester.recover-from-wal: Set this to
trueto recover data from an existing WAL.
debug.mutex-profile-fractionto enable mutex profiling #1969
globalingestion rate limiter strategy. Deprecated
/readyto queriers. #1934
/seriesAPI endpoint support with TSDB blocks storage. #1830
compactorcomponent, which iterates over users blocks stored in the bucket and compact them according to the configured block ranges. #1942
cortex_ingester_flush_reasonsgets a new
-ingester.spread-flushesoption is enabled. #1978
enable_tlsoptions to redis cache configuration. Enables usage of Microsoft Azure Cache for Redis service. #1923
-experimental.tsdb.bucket-store.sync-interval(0 disables the sync)
cortex_querier_blocks_sync_secondsmetric for the initial sync too
DROP TABLE schema_migrations;.
Run the migrate command:
migrate -path <absolute_path_to_cortex>/cmd/cortex/migrations -database postgres://localhost:5432/database force 2
cortex_prometheus_rule_group_last_evaluation_timestamp_seconds metric, tracked by the ruler, is not unregistered for rule groups not being used anymore. This issue will be fixed in the next Cortex release (see 2033).
Write-Ahead-Log (WAL) does not have automatic repair of corrupt checkpoint or WAL segments, which is possible if ingester crashes abruptly or the underlying disk corrupts. Currently the only way to resolve this is to manually delete the affected checkpoint and/or WAL segments. Automatic repair will be added in the future releases.
-ruler.configs.urlhas been now deprecated. #1579
Deltaencoding. Any old chunks with
Deltaencoding cannot be read anymore. If
ingester.chunk-encodingis set to
Deltathe ingester will fail to start. #1706
-ingester.max-transfer-retriesto 0 now disables hand-over when ingester is shutting down. Previously, zero meant infinite number of attempts. #1771
dynamohas been removed as a valid storage name to make it consistent for all components.
aws-dynamoremain as valid storage names.
--querier.split-queries-by-intervalis not provided request splitting is disabled by default.
--querier.split-queries-by-dayis still accepted for backward compatibility but has been deprecated. You should now use
--querier.split-queries-by-interval. We recommend a to use a multiple of 24 hours.
-distributor.shard-by-all-labelsset for the ingesters too
In this release we updated the following dependencies:
This release adds support for Redis as an alternative to Memcached, and also includes many optimisations which reduce CPU and memory usage.
--alertmanager.configs.auto-slack-rootflag was dropped as auto Slack root is not supported anymore. #1597
/shutdownendpoint for ingester to shutdown all operations of the ingester. #1746
Full list of changes: https://github.com/cortexproject/cortex/compare/v0.2.0...v0.3.0
This release has several exciting features, the most notable of them being setting
-ingester.spread-flushes to potentially reduce your storage space by upto 50%.
alertmanager.mesh.peer.servicecan be replaced by
cortex_cache_metrics is now
chunksmemcache(before it was
-ingester.spread-flushes. This means multiple replicas of a chunk are very likely to contain the same contents which cuts chunk storage space by up to 66%. #1578
http_configon alert receivers #929