The CartridgeRequirements webhook fails with x509 certificate error

An upgrade might fail with the following CartridgeRequirements webhook certificate error:

error: cartridgerequirements.base.automation.ibm.com “my-cartridge" could not be patched: Internal error occurred: failed calling webhook "vcartridgerequirements.kb.io": Post "https://iaf-operator-controller-manager-service.my.svc:9443/validate-base-automation-ibm-com-v1beta1-cartridgerequirements?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Red Hat, Inc."

Cause

Upon analyzing the logs of the kube-controller and etcd, you might notice a few entries that are related to network disruption.

While upgrading the IBM® Automation foundation operator, a network disruption creates an incompatibility between iaf-operator-controller-manager certificate and the vcartridgerequirments validating webhook.

Red Hat's Operator Lifecycle Manager (OLM) is responsible for the lifecycle of Operator and Webhook certificates and creates the required certificates at CSV install time. Netwrok disruption during this process can cause an incompatibility between the operator and it's webhook.

Resolving the problem

Perform the following steps to resolve the issue:

  1. Uninstall the IBM® Automation foundation Operator. The uninstall operation should remove both the Subscription and the CSV (UI uninstall removes both. However, you have to manually remove the subscription and CSV if you are uninstalling through CLI).

  2. Delete the validatingWebhookConfiguration named vcartridgerequirements.kb.io-*.

  3. Install the IBM® Automation foundation Operator. Your installation would be successful once the following logs appear in iaf-operator-controller-manager indicating that the controller manager can talk to the webhook and the certs have been refreshed by OLM.

{"level":"info","ts":"2022-03-16T14:40:48.514Z","logger":"cartridgerequirements-resource","msg":"validate update","name":"my-cartridge”}