Operator installation hangs during upgrade

When you upgrade IBM Cloud Pak foundational services, the installation of some operators does not complete.

Symptom

The catalog operator logs show messages similar to the following message:

Following is the command to get the catalog operator logs:

oc -n openshift-operator-lifecycle-manager logs deploy/catalog-operator

Following is a sample content:

E1201 07:54:37.617295       1 queueinformer_operator.go:290] sync "ibm-common-services" failed: found more than one head for channel

Cause

This is a known issue in the Operator Lifecycle Manager (OLM), which might be fixed in a future release of OpenShift Container Platform. During upgrade, there is an operator with replace in CSV that is missing skip range, which causes it to hang.

For more information about the issue, see Red Hat Bugzilla – Bug 1869441 Opens in a new tab.

Resolving the problem

If the operator that is causing the error is either IBM NamespaceScope Operator or Operand Deployment Lifecycle Manager (ODLM):

  1. From your OpenShift console, uninstall the hanging operator.

    1. From the navigation pane, click Operators > Installed Operators.

    2. For the operator with Pending status, click the overflow menu icon and click Uninstall.

      (Or)

    Delete the subscription and CSV of the operator by using the oc CLI.

    1. Log in to the cluster by using the oc login command.

    2. Run the following commands to delete the subscription and CSV.

        oc -n ibm-common-services delete sub <hanging-operator-sub>
  oc -n ibm-common-services delete csv <hanging-operator-csv>

  2. Delete ibm-common-service-operator pod

    oc delete pod <ibm-common-service-operator-pod-name> -n ibm-common-services

If the error is caused by other operator:

  1. From your OpenShift console, uninstall the hanging operator.

    1. From the navigation pane, click Operators > Installed Operators.

    2. For the operator with Pending status, click the overflow menu icon and click Uninstall.

      (Or)

    Delete the subscription and CSV of the operator by using the oc CLI.

    1. Log in to the cluster by using the oc login command.

    2. Run the following commands to delete the subscription and CSV.

      oc -n ibm-common-services delete sub <hanging-operator-sub>
oc -n ibm-common-services delete csv <hanging-operator-csv>

  2. Delete the ODLM pod.

    oc delete pod <operand-deployment-lifecycle-manager-pod-name> -n ibm-common-services