Securing communication between Cortex components with TLS
Cortex is a distributed system with significant traffic between its services. To allow for secure communication, Cortex supports TLS between all its components. This guide describes the process of setting up TLS.
Generation of certs to configure TLS
The first step to securing inter-service communication in Cortex with TLS is generating certificates. A Certifying Authority (CA) will be used for this purpose which should be private to the organization, as any certificates signed by this CA will have permissions to communicate with the cluster.
We will use the following script to generate self signed certs for the cluster:
# keys
openssl genrsa -out root.key
openssl genrsa -out client.key
openssl genrsa -out server.key
# root cert / certifying authority
openssl req -x509 -new -nodes -key root.key -subj "/C=US/ST=KY/O=Org/CN=root" -sha256 -days 100000 -out root.crt
# csrs - certificate signing requests
openssl req -new -sha256 -key client.key -subj "/C=US/ST=KY/O=Org/CN=client" -out client.csr
openssl req -new -sha256 -key server.key -subj "/C=US/ST=KY/O=Org/CN=localhost" -out server.csr
# certificates
openssl x509 -req -in client.csr -CA root.crt -CAkey root.key -CAcreateserial -out client.crt -days 100000 -sha256
openssl x509 -req -in server.csr -CA root.crt -CAkey root.key -CAcreateserial -out server.crt -days 100000 -sha256
Note that the above script generates certificates that are valid for 100000 days.
This can be changed by adjusting the -days
option in the above commands.
It is recommended that the certs be replaced at least once every 2 years.
The above script generates keys client.key, server.key
and certs
client.crt, server.crt
for both the client and server. The CA cert is
generated as root.crt
.
Load certs into the HTTP/GRPC server/client
Every HTTP/GRPC link between Cortex components supports TLS configuration through the following config parameters:
Server flags
# Path to the TLS Cert for the HTTP Server
-server.http-tls-cert-path=/path/to/server.crt
# Path to the TLS Key for the HTTP Server
-server.http-tls-key-path=/path/to/server.key
# Type of Client Auth for the HTTP Server
-server.http-tls-client-auth="RequireAndVerifyClientCert"
# Path to the Client CA Cert for the HTTP Server
-server.http-tls-ca-path="/path/to/root.crt"
# Path to the TLS Cert for the GRPC Server
-server.grpc-tls-cert-path=/path/to/server.crt
# Path to the TLS Key for the GRPC Server
-server.grpc-tls-key-path=/path/to/server.key
# Type of Client Auth for the GRPC Server
-server.grpc-tls-client-auth="RequireAndVerifyClientCert"
# Path to the Client CA Cert for the GRPC Server
-server.grpc-tls-ca-path=/path/to/root.crt
Client flags
Client flags are component specific.
For an HTTP client in the Alertmanager:
# Path to the TLS Cert for the HTTP Client
-alertmanager.configs.tls-cert-path=/path/to/client.crt
# Path to the TLS Key for the HTTP Client
-alertmanager.configs.tls-key-path=/path/to/client.key
# Path to the TLS CA for the HTTP Client
-alertmanager.configs.tls-ca-path=/path/to/root.crt
For a GRPC client in the Querier:
# Path to the TLS Cert for the GRPC Client
-querier.frontend-client.tls-cert-path=/path/to/client.crt
# Path to the TLS Key for the GRPC Client
-querier.frontend-client.tls-key-path=/path/to/client.key
# Path to the TLS CA for the GRPC Client
-querier.frontend-client.tls-ca-path=/path/to/root.crt
Similarly, for the GRPC Ingester Client:
# Path to the TLS Cert for the GRPC Client
-ingester.client.tls-cert-path=/path/to/client.crt
# Path to the TLS Key for the GRPC Client
-ingester.client.tls-key-path=/path/to/client.key
# Path to the TLS CA for the GRPC Client
-ingester.client.tls-ca-path=/path/to/root.crt
TLS can be configured in a similar fashion for other HTTP/GRPC clients in Cortex.