Renewing Kubernetes 1.14.x cluster certificates

This topic applies only when you have Kuberenetes 1.14.x. To find the Kubernetes version, enter the following command:

kubectl version --short

The Kubernetes cluster certificates have a lifespan of one year. It is important to know when your certificate expires. To determine the expiry date, run the following command as root user on the Kubernetes master:

find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'

For example, if your certificate expires in May 1, 2021, your output will resemble the following:

bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver.crt|grep After
            Not After : May  1 00:25:47 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-kubelet-client.crt|grep After
            Not After : May  1 00:30:35 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/front-proxy-client.crt|grep After
            Not After : May  1 00:31:02 2021 GMT

You should renew before the expiry date. If the Kubernetes cluster certificate expires on the Kubernetes master, then the kubelet service will fail. Issuing a kubectl command, such as kubectel get pods or kubectl exec -it container_name bash, will result in a message similar to Unable to connect to the server: x509: certificate has expired or is not yet valid.

If your Kubernetes cluster certificate has not expired and your system is still operational, you do not need to plan for a system outage because IBM Financial Crimes Insight will remain operational during the following procedure.

Procedure

To regenerate a new certificate and update worker nodes:

  1. If you have already renewed a Kubernetes certificate before, validate that the Kubernetes version in the kubeadm.yaml file matches the Kubernetes version that you are using. Otherwise, create a configuration file in /root named kubeadm.yaml with advertiseAddress set to the IP address of your Kubernetes master node. For example: 
    apiVersion: kubeadm.k8s.io/v1beta1
 kind: InitConfiguration
 localAPIEndpoint:
   advertiseAddress: <kube_master_ip>
---
 apiVersion: kubeadm.k8s.io/v1beta1
 kind: ClusterConfiguration
 imageRepository: <kube_master_hostname>:5000
 networking:
   dnsDomain: cluster.local
   podSubnet: 172.30.48.0/20
   serviceSubnet: 172.30.32.0/20
 kubernetesVersion: v1.14.1
apiServer:
  certSANs:
  - <kube_master_ip>
    Note: The kubernetesVersion must match the Kubernetes version that you are using. To determine the value of <kube_master_ip>, use the CLUSTER-IP value from this command: 
    kubectl get svc -l'component=apiserver'
    If the CLUSTER-IP matches the advertiseAddress, you can omit the following lines: 
    apiServer:
  certSANs:
  - <kube_master_ip>
    If you have more than one IP address, the advertiseAddress is the internal IP address that the worker nodes use to communicate back to the Kubernetes master. <kube_master_ip> is the external IP address that end users use to connect to the Kubernetes master.
  2. Back up the existing Kubernetes certificates by running the following commands:
    mkdir -p $HOME/fci103-k8s-old-certs/pki
/bin/cp -p /etc/kubernetes/pki/*.* $HOME/fci103-k8s-old-certs/pki
ls -l $HOME/fci103-k8s-old-certs/pki/
    The output should resemble the following:
    [root@fcitest231 ~]# mkdir -p $HOME/fci103-k8s-old-certs/pki
[root@fcitest231 ~]# /bin/cp -p /etc/kubernetes/pki/*.* $HOME/fci103-k8s-old-certs/pki
[root@fcitest231 ~]# ls -l $HOME/fci103-k8s-old-certs/pki/
total 56
-rw-r--r-- 1 root root 1261 Mar  2  2020 apiserver.crt
-rw-r--r-- 1 root root 1094 Mar  2  2020 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Mar  2  2020 apiserver-etcd-client.key
-rw------- 1 root root 1675 Mar  2  2020 apiserver.key
-rw-r--r-- 1 root root 1099 Mar  2  2020 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Mar  2  2020 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 Mar  2  2020 ca.crt
-rw------- 1 root root 1679 Mar  2  2020 ca.key
-rw-r--r-- 1 root root 1025 Mar  2  2020 front-proxy-ca.crt
-rw------- 1 root root 1675 Mar  2  2020 front-proxy-ca.key
-rw-r--r-- 1 root root 1050 Mar  2  2020 front-proxy-client.crt
-rw------- 1 root root 1675 Mar  2  2020 front-proxy-client.key
-rw------- 1 root root 1675 Mar  2  2020 sa.key
-rw------- 1 root root  451 Mar  2  2020 sa.pub
  3. Remove the existing certificate and key files: 
    /bin/rm /etc/kubernetes/pki/apiserver.key
/bin/rm /etc/kubernetes/pki/apiserver.crt
/bin/rm /etc/kubernetes/pki/apiserver-kubelet-client.crt
/bin/rm /etc/kubernetes/pki/apiserver-kubelet-client.key
/bin/rm /etc/kubernetes/pki/front-proxy-client.crt
/bin/rm /etc/kubernetes/pki/front-proxy-client.key
  4. Renew the certificates: 
    kubeadm --config /root/kubeadm.yaml alpha certs renew all
    Check the certificates under /etc/kubernetes/pki and /etc/kubernetes/pki/etcd to ensure that they have been renewed successfully.
  5. Back up the existing configuration files by running the following commands:
    /bin/cp -p /etc/kubernetes/*.conf $HOME/fci103-k8s-old-certs
ls -ltr $HOME/fci103-k8s-old-certs
    The output should resemble the following:
    [root@fcitest231 ~]# ls -ltr $HOME/fci103-k8s-old-certs
total 36
-rw------- 1 root root 5451 Mar  2  2020 admin.conf
-rw------- 1 root root 5599 Mar  2  2020 kubelet.conf
-rw------- 1 root root 5487 Mar  2  2020 controller-manager.conf
-rw------- 1 root root 5431 Mar  2  2020 scheduler.conf
drwxr-xr-x 2 root root 4096 Sep  2 20:40 pki
  6. Remove the old configuration files: 
    /bin/rm /etc/kubernetes/admin.conf
/bin/rm /etc/kubernetes/kubelet.conf
/bin/rm /etc/kubernetes/controller-manager.conf
/bin/rm /etc/kubernetes/scheduler.conf
  7. Generate new configuration files: 
    kubeadm --config /root/kubeadm.yaml init phase kubeconfig all
    The output should resemble the following:
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
  8. Renew the kubelet server certificates /var/lib/kubelet/pki/kubelet.crt by leveraging csr and Kubernetes master CA certificate. The kubelet-client.pem is automatically renewed.
    1. Back up /var/lib/kubelet/config.yaml. 
      cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak
    2. Insert the following code snippet into /var/lib/kubelet/config.yaml to enable rotation of the kubelet server/client certificate. 
      serverTLSBootstrap: true
FeatureGates:
  RotateKubeletClientCertificate: true
   RotateKubeletServerCertificate: true
    3. Run kubectl edit cm kubelet-config-1.14 -n kube-system, and add following code snippet under kubelet section: 
      serverTLSBootstrap: true
FeatureGates:
  RotateKubeletClientCertificate: true
  RotateKubeletServerCertificate: true
    4. Back up /etc/kubernetes/kubelet.conf. 
      cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak.new
    5. Use the current kubelet-client certificate in /etc/kubernetes/kubelet.conf.

      In versions of kubeadm before 1.17, there is a bug where you manually have to modify the contents of kubelet.conf.

      After kubeadm init completes, update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with the following code snippet. 

      client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
    6. Update $HOME/.kube/config with latest admin.conf. 
      /bin/cp $HOME/.kube/config $HOME/.kube/config.orig
/bin/cp /etc/kubernetes/admin.conf $HOME/.kube/config
export KUBECONFIG=.kube/config
    7. Restart kubelet systemctl restart kubelet to trigger csr.
    8. Run kubectl get csr to get the corresponding Kubernetes master VM’s csr and approve the csr. 
      kubectl certificate approve <csr_name>
    9. Run the following command to ensure that the kubelet server cert has been created successfully: 
      ls -ld /var/lib/kubelet/pki/kubelet-server*

      The output should be similar to the following:

      kubelet-server-2020-06-16-20-06-09.pem
kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2020-06-16-20-06-09.pem
    10. Run kubectl get csr to get the corresponding Kubernetes master VM’s csr and approve the csr. 
      kubectl certificate approve <csr_name>
    11. Run the following command to ensure that the kubelet server cert has been created successfully: 
      ls -ld /var/lib/kubelet/pki/kubelet-server*

      The output should be similar to the following:

      kubelet-server-2020-06-16-20-06-09.pem
kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2020-06-16-20-06-09.pem
  9. For Kubernetes worker VMs, upgrade the kubelet server certificate kubelet.crt. The kubelet-client.crt is renewed automatically.
    1. Back up /var/lib/kubelet/config.yaml. 
      cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak
    2. To enable rotation of the kubelet server certificate, insert the following code snippet into /var/lib/kubelet/config.yaml: 
      serverTLSBootstrap: true
 FeatureGates:
   RotateKubeletClientCertificate: true
   RotateKubeletServerCertificate: true
    3. Restart the kubelet. 
      systemctl restart kubelet
    4. Log in to the Kubernetes master VM and get the corresponding Kubernetes worker VM’s csr. 
      kubectl get csr
    5. Approve the csr. 
      kubectl certificate approve <csr_name>
    6. Run the following command to ensure that the kubelet server cert has been created successfully: 
      ls -ld /var/lib/kubelet/pki/kubelet-server*

      The output should be similar to the following:

      kubelet-server-2020-06-16-20-06-09.pem
kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2020-06-16-20-06-09.pem
  10. Stop the Kubernetes cluster:
    1. As the root user, stop all worker nodes, simultaneously or individually. If running in VMWare vSphere, use Shutdown Guest OS. Otherwise, run the following command from the terminal session: 
      shutdown -h now
    2. After all the worker nodes are shut down, shut down the Kubernetes master node.

      If the NFS server is on a different host than the Kubernetes master, you can shut down the Kubernetes master when you shut down the worker nodes.

    3. Stop the NFS server next. By default, this is located on the Kubernetes master node and will be shutdown when the Kubernetes master node host or virtual machine is powered off.
    4. Stop the server or virtual machine running the Docker registry last. The Docker registry is normally running on the Kubernetes Master node and will get stopped when the master node is powered off.
  11. Start the Kubernetes cluster.
    1. Start the Docker registry, normally the Docker registry is on the same node as the Kubernetes master and will get started automatically when the server or virtual machine is started that contains the Docker registry is started.
    2. Start the NFS server and wait two minutes after the operating system has started before proceeding to ensure that the NFS server has started. Normally the NFS server is on the Kubernetes master and the NFS Server will start automatically when the server or virtual machine is started that contains the NFS Server.
    3. If not already running, start the Kubernetes master and all worker nodes at the same time. If the Kubernetes master was already running, start all worker nodes at the same time or start the worker nodes individually in rapid succession.
  12. Run the following commands after all VMs up and running, to check the health of the system: 
    kubectl get pods
kubectl get nodes

    On the Kubernetes master, run the following command:

    run "echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate"

    Log in to the FCI application.