Securing communication between Cortex components with TLS
Cortex is a distributed system with significant traffic between its services. To allow for secure communication, Cortex supports TLS between all its components. This guide describes the process of setting up TLS.
Generation of certs to configure TLS
The first step to securing inter-service communication in Cortex with TLS is generating certificates. A Certifying Authority (CA) will be used for this purpose which should be private to the organization, as any certificates signed by this CA will have permissions to communicate with the cluster.
We will use the following script to generate self signed certs for the cluster:
# Refer: github.com/cortexproject/cortex/integration/certs/genCerts.sh # keys openssl genrsa -out root.key openssl genrsa -out client.key openssl genrsa -out server.key # root cert / certifying authority openssl req -x509 -new -nodes -key root.key -subj "/C=US/ST=KY/O=Org/CN=root" -sha256 -days 100000 -out root.crt # csrs - certificate signing requests openssl req -new -sha256 -key client.key -subj "/C=US/ST=KY/O=Org/CN=client" -out client.csr openssl req -new -sha256 -key server.key -subj "/C=US/ST=KY/O=Org/CN=localhost" -out server.csr # certificates openssl x509 -req -in client.csr -CA root.crt -CAkey root.key -CAcreateserial -out client.crt -days 100000 -sha256 openssl x509 -req -in server.csr -CA root.crt -CAkey root.key -CAcreateserial -out server.crt -days 100000 -sha256
Note that the above script generates certificates that are valid for 100000 days.
This can be changed by adjusting the
-days option in the above commands.
It is recommended that the certs be replaced atleast once every 2 years.
The above script generates keys
client.key, server.key and certs
client.crt, server.crt for both the client and server. The CA cert is
Load certs into the HTTP/GRPC server/client
Every HTTP/GRPC link between Cortex components supports TLS configuration through the following config parameters:
# Path to the TLS Cert for the HTTP Server -server.http-tls-cert-path=/path/to/server.crt # Path to the TLS Key for the HTTP Server -server.http-tls-key-path=/path/to/server.key # Type of Client Auth for the HTTP Server -server.http-tls-client-auth="RequireAndVerifyClientCert" # Path to the Client CA Cert for the HTTP Server -server.http-tls-ca-path="/path/to/root.crt" # Path to the TLS Cert for the GRPC Server -server.grpc-tls-cert-path=/path/to/server.crt # Path to the TLS Key for the GRPC Server -server.grpc-tls-key-path=/path/to/server.key # Type of Client Auth for the GRPC Server -server.grpc-tls-client-auth="RequireAndVerifyClientCert" # Path to the Client CA Cert for the GRPC Server -server.grpc-tls-ca-path=/path/to/root.crt
Client flags are component specific.
For an HTTP client in the Alertmanager:
# Path to the TLS Cert for the HTTP Client -alertmanager.configs.tls-cert-path=/path/to/client.crt # Path to the TLS Key for the HTTP Client -alertmanager.configs.tls-key-path=/path/to/client.key # Path to the TLS CA for the HTTP Client -alertmanager.configs.tls-ca-path=/path/to/root.crt
For a GRPC client in the Querier:
# Path to the TLS Cert for the GRPC Client -querier.frontend-client.tls-cert-path=/path/to/client.crt # Path to the TLS Key for the GRPC Client -querier.frontend-client.tls-key-path=/path/to/client.key # Path to the TLS CA for the GRPC Client -querier.frontend-client.tls-ca-path=/path/to/root.crt
Similarly, for the GRPC Ingester Client:
# Path to the TLS Cert for the GRPC Client -ingester.client.tls-cert-path=/path/to/client.crt # Path to the TLS Key for the GRPC Client -ingester.client.tls-key-path=/path/to/client.key # Path to the TLS CA for the GRPC Client -ingester.client.tls-ca-path=/path/to/root.crt
TLS can be configured in a similar fashion for other HTTP/GRPC clients in Cortex.