Troubleshooting
Problem
After the upgrading of the Kubernetes on the "IBM Cloud Private cluster," following pods were not started
kubectl get pods --all-namespaces | grep -v Running
Show non-Running pods
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system k8s-proxy-qkmvf 0/1 ContainerCreating 0 1h
kube-system k8s-proxy-wvbws 0/1 ContainerCreating 0 1h
kube-system k8s-proxy-xws79 0/1 ContainerCreating 0 1h
Symptom
Apiserver logs
I0429 02:03:24.307185 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io
E0429 02:03:25.939799 1 available_controller.go:316] v1beta1.servicecatalog.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0429 02:03:30.945853 1 available_controller.go:316] v1beta1.servicecatalog.k8s.io failed with: Get https://10.0.0.33:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0429 02:03:35.952120 1 available_controller.go:316] v1beta1.servicecatalog.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0429 02:03:40.964045 1 available_controller.go:316] v1beta1.servicecatalog.k8s.io failed with: Get https://10.0.0.33:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Cause
There was a backup of the master.jason file under /etc/cfs/pods on all the master nodes. While starting the Kubernetes, old versions were used which cause the API servers not available for other pods to access.
"name": "apiserver",
"image": "mycluster.icp:8500/ibmcom/hyperkube:v1.11.3-ee",
Environment
IBM Cloud Private upgrade which includes K8S upgrade
Diagnosing The Problem
kubectl version shows mismatch and is still at the older version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12+icp-ee2003", GitCommit:"8d24284de4f4c9a00ad4fa2c4370fbbc13fdf3ac", GitTreeState:"clean", BuildDate:"2020-03-19T06:05:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1+icp-ee", GitCommit:"5803c3b1f9422c43a963e0610b3a4cad565e127e", GitTreeState:"clean", BuildDate:"2018-09-04T09:29:02Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
[root@louapplts200 cluster]# getpodsnrw
Resolving The Problem
1- Make sure there are no backup files under the /etc/cfc/pods directory on all the master nodes
ls -l /etc//cfc/pods
total 16
-rw------- 1 root root 3262 Oct 5 2020 etcd.json
-rw------- 1 root root 2077 Oct 15 2020 kmsplugin.json
-rw------- 1 root root 7434 Oct 15 2020 master.json
2-- make sure the imaged pulled are the latest version in the master.jason file
# grep image -B1 m /etc/cfc/pods/master.json
grep: m: No such file or directory
/etc/cfc/pods/master.json- "name": "controller-manager",
/etc/cfc/pods/master.json: "image": "mycluster.icp:8500/ibmcom/hyperkube:v1.13.12-ee.2003",
/etc/cfc/pods/master.json: "imagePullPolicy": "IfNotPresent",
--
/etc/cfc/pods/master.json- "name": "apiserver",
/etc/cfc/pods/master.json: "image": "mycluster.icp:8500/ibmcom/hyperkube:v1.13.12-ee.2003",
/etc/cfc/pods/master.json: "imagePullPolicy": "IfNotPresent",
--
/etc/cfc/pods/master.json- "name": "scheduler",
/etc/cfc/pods/master.json: "image": "mycluster.icp:8500/ibmcom/hyperkube:v1.13.12-ee.2003",
/etc/cfc/pods/master.json: "imagePullPolicy": "IfNotPresent",
3-- you can confirm the image version by running the following command
# docker images | grep hyperkube
mycluster.icp:8500/ibmcom/hyperkube-amd64 v1.13.12-ee.2003 3df37a524d2a 2 years ago 761MB
mycluster.icp:8500/ibmcom/hyperkube v1.13.12-ee.2003 3df37a524d2a 2 years ago 761MB
mycluster.icp:8500/ibmcom/hyperkube-amd64 v1.13.9-ee 7bf58e3b5e59 2 years ago 633MB
mycluster.icp:8500/ibmcom/hyperkube v1.13.9-ee 7bf58e3b5e59 2 years ago 633MB
After the upgrade the later version should be updated in the /etc/cfc/pods/master.json file
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"ARM Category":[{"code":"a8m50000000Ck50AAC","label":"IBM Cloud Private-\u003EUpgrade"}],"ARM Case Number":"TS009195149","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.2.0;3.2.1;3.2.2"}]
Product Synonym
ICP
Was this topic helpful?
Document Information
Modified date:
02 May 2022
UID
ibm16579125