Certificate Hot Reload

Feature Overview

• The native certificate hot reload capabilities provided by Kubernetes, etcd, and CoreDNS are incomplete. Kubernetes and etcd provide only partial certificate hot reload capabilities, and CoreDNS does not support certificate hot reload. This feature aims to improve the certificate hot reload capabilities of all components to meet the certificate hot loading requirements of services.

• To simplify the description, certificate hot reload in the following refers to the hot reload of the certificate, private key, and public key files.

Application Scenarios

• After the certificate, private key, and CA certificate for communication in the Kubernetes cluster are updated, certificate hot reload is automatically performed for all components, communication between all components is normal, and service components can access kube-apiserver properly.
• After the SA key is updated, certificate hot reload is automatically performed forkube-apiserver, kube-controller-manager, and kubelet, and service components can access kube-apiserver properly.

Supported Capabilities

Table 1 Support and usage of Kubernetes component certificate hot reload

Component	Certificate/Public Key/Private Key Parameter	Hot Reload Support	Usage
kube-apiserver	--client-ca-file --requestheader-client-ca-file --tls-cert-file --tls-private-key-file --tls-sni-cert-key	Supported	Certificate required by the server, which is used for communication with the API server client.
kube-apiserver	--etcd-cafile --etcd-certfile --etcd-keyfile	Not supported by CA certificates	Certificate required by the client, which is used for communication with etcd.
kube-apiserver	--kubelet-certificate-authority --kubelet-client-certificate --kubelet-client-key	Not supported by CA certificates	Certificate required by the client, which is used for communication with kubelet.
kube-apiserver	--peer-ca-file --proxy-client-cert-file --proxy-client-key-file	Not supported by CA certificates	Certificate required by the client, which is used for communication with the peer (for example, user API server).
kube-apiserver	--service-account-key-file --service-account-signing-key-file	Not supported	Used for SA token signing and authentication.
kube-apiserver	--oidc-ca-file	Not supported	Certificate required by the OpenID Connect (OIDC) client, which is used to authenticate the OpenID service.
kube-controller-manager	--client-ca-file --requestheader-client-ca-file --tls-cert-file --tls-private-key-file --tls-sni-cert-key	Supported	Certificate required by the server, which is used for communication with the controller-manager client.
kube-controller-manager	--root-ca-file	Not supported	Root CA certificate, which is included in the kube-root-ca.crt ConfigMap and in TokenSecret and injected tokens.
kube-controller-manager	--cluster-signing-key-file --cluster-signing-cert-file --cluster-signing-kube-apiserver-client-cert-file --cluster-signing-kube-apiserver-client-key-file --cluster-signing-kubelet-client-cert-file --cluster-signing-kubelet-client-key-file --cluster-signing-kubelet-serving-cert-file --cluster-signing-kubelet-serving-cert-file --cluster-signing-legacy-unknown-cert-file --cluster-signing-legacy-unknown-key-file	Supported	Used for certificate signing during certificate rotation.
kube-controller-manager	--service-account-private-key-file	Not supported	Used for SA token signing.
kube-controller-manager	--kubeconfig -certificate-authority -client-certificate -client-key	Not supported by CA certificates	Certificate required by the client, which is used for communication with the API server.
kube-scheduler	--client-ca-file --requestheader-client-ca-file --tls-cert-file --tls-private-key-file --tls-sni-cert-key	Supported	Certificate required by the server, which is used for communication with the client.
kube-scheduler	--kubeconfig -certificate-authority -client-certificate -client-key	Not supported by CA certificates	Certificate required by the client, which is used for communication with the API server.
kubelet	--client-ca-file --tls-cert-file --tls-private-key-file	Not supported	Certificate required by the server, which is used for communication with the client.
kubelet	--kubeconfig -certificate-authority -client-certificate -client-key --bootstrap-kubeconfig -certificate-authority -client-certificate -client-key	Not supported by CA certificates	Certificate required by the client, which is used for communication with the API server.
kube-proxy	--kubeconfig: -certificate-authority -client-certificate -client-key	Not supported by CA certificates	Certificate required by the client, which is used for communication with the API server.
etcd	--trusted-ca-file --cert-file --key-file --client-cert-file --client-key-file	Not supported by CA certificates	Certificate required for communication between the etcd client and server.
etcd	--peer-trusted-ca-file --peer-cert-file --peer-key-file --peer-client-cert-file --peer-client-key-file	Not supported by CA certificates	Certificate required for communication between etcd instances.
etcd	--cacert --cert --key	Not supported by CA certificates	Certificate required for the etcd health self-check client.
CoreDNS	kubernetes { tls: }	Not supported by CA certificates	Certificate required by the client, which is used for secure communication with the API server.
CoreDNS	prometheus { tls: }	Not supported	Certificate required by the server, which is used for secure communication with the metrics API caller.
CoreDNS	health { tls: }	Not supported	Certificate required by the server, which is used for secure communication with the health interface caller.

Highlights

• After the certificate, private key, CA certificate, and SA key for communication in the kubernetes cluster are updated, certificate hot reload is automatically performed for all components.
• Services are not interrupted, communication between all components is normal, and service components can access kube-apiserver properly.

Implementation Principles

• The implementation solution includes the server certificate, private key, and client CA certificate (optional).

  tls.Config supports the GetConfigForClient method, which is called each time when a TLS connection is created to obtain a tls.Config data entity. This method is registered for tls.Config and the following functions are implemented in the method for hot reload:

  Create a tls.Config data entity and set the server certificate and CA certificate content in the memory to the Certificates and ClientCAs fields. The server certificate and CA certificate content are monitored by another coroutine and dynamically updated to the memory.

  type Config struct {
      ... ...
      // Device certificate (server certificate saved for the server and client certificate saved for the client)
      Certificates []Certificate
      // Client CA certificate
      ClientCAs *x509.CertPool
      // tls.Config is provided dynamically.
  	GetConfigForClient func(*ClientHelloInfo) (*Config, error)    
      ... ...
  }

• Certificates required for TLS client communication include the client certificate (optional), private key (optional), and server CA certificate (optional).

  Register the GetClientCertificate method in tls.Config and implement the following functions in the method:

  Set the client certificate content in the memory to the Certificates field. The client certificate content is monitored by another coroutine and dynamically updated to the memory.

  type Config struct {
      ... ...
      // Server CA certificate
      RootCAs *x509.CertPool
      // The client certificate is provided dynamically.
  	GetClientCertificate func(*CertificateRequestInfo) (*Certificate, error)   
      ... ...
  }

  The CA file changes are monitored. After the CA file is updated, create the x509.CertPool object again to replace the original x509.CertPool object in tls.Config.

• SA public key and private key (kube-apiserver, kube-controller-manager, and kubelet) are monitored.

  The SA public key and private key file changes are monitored. After the file content is updated, update the objects that use the SA public key or private key, including the SA token generator and SA token verifier. In addition, update the allocated token.

1. token secret: It is updated by kube-controller-manager.

2. Token injected to the Pod: The token is updated by kubelet. kubelet is not configured with the SA key file and cannot detect the change. In this case, the following operations are required:

   (1) kube-apiserver monitors the changes of the SA private key file and releases the private key hash as a ConfigMap.

   (2) kubelet monitors the changes of the key hash ConfigMap. Once the key hash changes, kubelet injects the SA token to all Pods on the current node.

• Root CA certificate (kube-controller-manager or kubelet)

  The root certificate file changes are monitored. After the file content is updated, kube-controller-manager needs to update the kube-root-ca.crt files in all namespaces and the CA certificates in all token secrets. kubelet re-injects the SA token that contains the latest CA certificate to all Pods.

Relationship with Related Features

None

Installation

This function is automatically enabled. No manual installation is required.

Using Certificate Hot Reload

Prerequisites

None

Context

• After the certificate, private key, and CA certificate for communication in the Kubernetes cluster are updated, certificate hot reload is automatically performed for all components, communication between all components is normal, and service components can access kube-apiserver properly.

Constraints

• The certificates and private keys embedded in the kubeconfig file cannot be hot reloaded.
• CoreDNS only supports host reload of certificates required for intra-cluster interaction. The certificates of plug-ins such as forward, gRPC, and etcd cannot be hot reloaded.
• After the certificates and private keys are updated externally, the CaaSCore component completes certificate hot reload within 1 minute and SA token refreshing within 5 minutes. Before the certificates are hot reloaded, the cluster functions may be unavailable. Before the SA token is refreshed, service Pods may fail to access kube-apiserver. If a component fails to access the server, the backoff retry mechanism is used. The retry logic of requests from different components and clients may be different. You are advised to retry every 30 seconds for 5 minutes.

Procedure

No manual operation is required. After the SA key, CA certificate, or cluster communication certificate is updated, the Kubernetes component certificates are automatically updated.