The CartridgeRequirements webhook fails with x509 certificate error
An upgrade might fail with the following CartridgeRequirements webhook certificate error:
error: cartridgerequirements.base.automation.ibm.com “my-cartridge" could not be patched: Internal error occurred: failed calling webhook "vcartridgerequirements.kb.io": Post "https://iaf-operator-controller-manager-service.my.svc:9443/validate-base-automation-ibm-com-v1beta1-cartridgerequirements?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Red Hat, Inc."
Cause
Upon analyzing the logs of the kube-controller and etcd, you might notice a few entries that are related to network disruption.
While upgrading the IBM® Automation foundation operator, a network disruption creates an incompatibility between iaf-operator-controller-manager
certificate and the vcartridgerequirments
validating webhook.
Red Hat's Operator Lifecycle Manager (OLM) is responsible for the lifecycle of Operator and Webhook certificates and creates the required certificates at CSV install time. Netwrok disruption during this process can cause an incompatibility between the operator and it's webhook.
Resolving the problem
Perform the following steps to resolve the issue:
-
Uninstall the IBM® Automation foundation Operator. The uninstall operation should remove both the
Subscription
and theCSV
(UI uninstall removes both. However, you have to manually remove the subscription and CSV if you are uninstalling through CLI). -
Delete the validatingWebhookConfiguration named
vcartridgerequirements.kb.io-*
. -
Install the IBM® Automation foundation Operator. Your installation would be successful once the following logs appear in
iaf-operator-controller-manager
indicating that the controller manager can talk to the webhook and the certs have been refreshed by OLM.
{"level":"info","ts":"2022-03-16T14:40:48.514Z","logger":"cartridgerequirements-resource","msg":"validate update","name":"my-cartridge”}