How integration tests work
Cortex integration tests are written in Go and based on a custom framework running Cortex and its dependencies in Docker containers and using the Go
testing package for assertions. Integration tests run in CI for every PR, and can be easily executed locally during development (it just requires Docker).
How to run integration tests
When integration tests run in CI, we build the Cortex docker image based on the PR code and then run the integration tests against it. When running tests locally you should build the Cortex Docker image first:
This will locally build the
quay.io/cortexproject/cortex:latest image used by integration tests. Whenever the Cortex code changes (
pkg/ or vendors) you should rebuild the Cortex image, while it’s not necessary to rebuild it while developing integration tests.
Once the Docker image is built, you can run integration tests:
go test -v -tags=integration,requires_docker,integration_alertmanager,integration_memberlist,integration_querier,integration_ruler,integration_query_fuzz ./integration/...
If you want to run a single test you can use a filter. For example, to only run
go test -v -tags=integration,integration_ruler -timeout 2400s -v -count=1 ./integration/... -run "^TestRulerAPISharding$"
Supported environment variables
Docker image used to run Cortex in integration tests (defaults to
The absolute path of the Cortex repository local checkout (defaults to
The absolute path to a directory where the integration test will create an additional temporary directory to store files generated during the test.
Name of the docker network to create and use for integration tests. If no variable is set, defaults to
Integration tests have
requires_docker tag (
// +build requires_docker line followed by empty line on top of Go files), to avoid running them unintentionally as they require Docker, e.g. by running
go test ./... in main Cortex package.
Each integration test runs in isolation. For each integration test, we do create a Docker network, start Cortex and its dependencies containers, push/query series to/from Cortex and run assertions on it. Once the test has done, both the Docker network and containers are terminated and deleted.