Question & Answer
Question
We have been reported for a security vulnerability in our ICP cluster, related to weak ciphers currently in use (SWEET32).
How can this be manually fixed?
Answer
Following instructions have been successfully tested for an ICP 3.1.2 environment.
The changes must be implemented on ETCD, KUBERNETES CONTROL PLANE and KUBELET nodes.
- ETCD and KUBERNETES CONTROL PLANE: on all master nodes only
- KUBELET: on master nodes and also all cluster nodes
ETCD
1. Copy and back up etcd static pod manifest file on master node. cp /etc/cfc/pods/etcd.json ~/icp-backup/ cp ~/icp-backup/etcd.json ~/icp-backup/etcd.json.bak 2. Update etcd static pod manifest file vim ~/icp-backup/etcd.json to add following options "--cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" By default etcd is using TLSv1.2. No need to update TLS min version. 3. Copy the updated etcd static pod manifest back. Kubelet will help restart etcd service. cp ~/icp-backup/etcd.json /etc/cfc/pods/ 4. Verify etcd service is started. # docker ps |grep etcd 416e7e7ed2a5 33bdcac177c2 "etcd --name=etcd0 -…" 2 minutes ago Up 2 minutes k8s_etcd_k8s-etcd-9.21.55.15_kube-system_ae53b0c24e347e2f786003f83ab595b7_0 Run above steps on all master nodes.
KUBELET
1. Update kubelet service config vim /etc/cfc/kubelet/kubelet-service-config and add following option tlsCipherSuites: ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256"] By default TLSv1.2 is used. No need to specify tls min version. 2. Restart kubelet service systemctl restart kubelet systemctl status kubelet 3. Verify kubelet is started. Check kubelet log journalctl -u kubelet.service -f Check node status kubectl get nodes Run above steps on all cluster nodes.
KUBERNETES CONTROL PLANE
1. Copy and back up Kubernetes static pod manifest file on master node. cp /etc/cfc/pods/master.json ~/icp-backup/ cp ~/icp-backup/master.json ~/icp-backup/master.json.bak 2. Update Kubernetes static pod manifest file vim ~/icp-backup/master.json to add following options. There are 3 containers in the static pod, kube-controller-manager, kube-apiserver and kube-scheduler. Add same option for all 3 container. ( NOTE: "tls-cipher-suites" can be ignored for "kube-scheduler" container ) "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" 3. Copy the updated Kubernetes static pod manifest back. Kubelet will help restart Kubernetes service. cp ~/icp-backup/master.json /etc/cfc/pods/ 4. Verify Kubernetes service is started. # docker ps |grep hyperkube b4844586cc1a a28dcbcae557 "/hyperkube schedule…" 14 minutes ago Up 14 minutes k8s_scheduler_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_3 4472e7a9f4bd a28dcbcae557 "/hyperkube controll…" 14 minutes ago Up 14 minutes k8s_controller-manager_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_3 bbc2b79cee2e a28dcbcae55 7 "/hyperkube apiserve…" 20 hours ago Up 14 minutes k8s_apiserver_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_0 Verify kubernetes service kubectl get pods --all-namespaces Run above steps on all master nodes.
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.1.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Product Synonym
ICP
Was this topic helpful?
Document Information
Modified date:
15 October 2019
UID
ibm11087504