Troubleshooting
Problem
Multiple operators in degraded state due to failed automatic operator upgrades which failed.
Symptom
Worker node going to cordoned state and failing to drain. Result is a node which is unschedulable and degraded operators.
Cause
The update strategy for operators is automatic it and when performing this it encountered an issue updating several operators. When performing this action OpenShift will cordon and drain nodes, whilst draining one node it fails to drain the node fully and applies the operator update.
Accounts used to install OpenShift on the respective Cloud or VMware the password, or API keys, change. Installing on VSphere, using a specific account in VCenter to do that install, but then locking the account and changing the password after the successful install of OCP. That causes the API calls, between OCP and VMware or the cloud providers, with the respective permissions and accounts to fail. Thus, the entire OCP install fails.
Resolving The Problem
Manually drain the affected node which was running into issues:
oc adm drain {$NODE} --delete-emptydir-data --ignore-daemonsets --grace-period=0 --force
Note: Replace {node} with the target IP address.
Result
The node comes back up without issue and all degraded operators are updated.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m3p000000PCQWAA4","label":"OpenShift-\u003ECertificates"},{"code":"a8m3p000000PCQ7AAO","label":"OpenShift-\u003EUpdate"}],"ARM Case Number":"TS009759997","Platform":[{"code":"PF116","label":"VMware vSphere"},{"code":"PF118","label":"Red Hat OpenShift Platform"},{"code":"PF119","label":"Microsoft Azure"},{"code":"PF121","label":"Amazon Web Services"}],"Version":"1.9.0"}]
Was this topic helpful?
Document Information
Modified date:
18 August 2023
UID
ibm17027679