Upgrading API Connect on Kubernetes

Upgrade your API Connect subsystems to a newer version on Kubernetes.

1. If you did not run the pre-upgrade health checks recently, run them now.
2. If you have a two data center disaster recovery deployment, then upgrade the warm-standby data center first.
   See the special steps for 2DCDR upgrades:

   • Upgrading a 2DCDR deployment on Kubernetes and OpenShift from V10.0.5
   • Upgrading a 2DCDR deployment on Kubernetes and OpenShift from V10.0.7
3. Apply the API Connect CRDs that you downloaded and extracted in the pre-upgrade steps:

   kubectl apply --server-side --force-conflicts -f ibm-apiconnect-crds.yaml

4. Apply the new DataPower Operator YAML into the namespace where the DataPower operator is running.
   1. Edit the ibm-datapower.yaml file and set the following properties:the namespace property to the namespace where you deployed the operator.
      • Set all instances of the namespace property to the namespace where you deployed the operator.
      • Set the containers.image property to the location of the datapower-operator image, either uploaded to your own registry or pulled from a public registry.
   2. Run the following command:

      kubectl -n <namespace> apply -f ibm-datapower.yaml

      The gateway CR goes to Pending state when the operator is updated, and then changes to Running after installation of the API Connect operator in the next step.

      Troubleshooting: If your DataPower operator pod fails to start, see Troubleshooting upgrade.

5. Upgrade cert-manager to version 1.12.10.
   1. Run the following command to back up the existing certificates and issuers to a file called backup.yaml:

      kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert,secret > backup.yaml

   2. Run the following command to verify the version of cert-manager that you are upgrading:
      • API Connect 10.0.5.4: verify that you are upgrading cert-manager 1.9.1:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.9.1

      • API Connect 10.0.5.5 or 10.0.7.0: verify that you are upgrading cert-manager 1.11.5:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.11.5

      • API Connect 10.0.5.6: verify that you are upgrading cert-manager 1.12.7:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.7

      • API Connect 10.0.5.7: verify that you are upgrading cert-manager 1.12.9:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.9

   3. Run the following command to upgrade cert-manager to version 1.12.10:

      kubectl apply -f helper_files/cert-manager-1.12.10.yaml

6. Clean up the webhook configuration before you deploy the newer API Connect operator:
   1. Run the following command to get the webhook configuration:

      kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep ibm-apiconnect

   2. Use the kubectl delete command to delete the webhook configuration; for example:

      kubectl delete mutatingwebhookconfiguration ibm-apiconnect-mutating-webhook-configuration
      kubectl delete validatingwebhookconfiguration ibm-apiconnect-validating-webhook-configuration
      

7. Apply the new API Connect operator YAML into the namespace where the API Connect operator is running.
   • For a single-namespace deployment:
     1. If the operator is not running in the default namespace, edit the ibm-apiconnect.yaml and replace all references to default with the name of the namespace where you deployed API Connect.
        Note: If you are using Operator Lifecycle Manager (OLM), then skip this step.
     2. Edit ibm-apiconnect.yaml file and replace the value of each image property with the location of the apiconnect operator images (from the ibm-apiconnect container and the ibm-apiconnect-init container), either uploaded to your own registry or pulled from a public registry.
     3. Run the following command:

        kubectl -n <namespace> apply -f ibm-apiconnect.yaml

   • For a multi-namespace deployment:
     1. Edit the ibm-apiconnect-distributed.yaml file and replace each occurrence of $OPERATOR_NAMESPACE with the namespace for the deployment.
     2. Also in ibm-apiconnect-distributed.yaml, set the values of the containers.image properties. Replace the placeholder values REPLACE-DOCKER-REGISTRY with the docker registry host location of the API Connect operator image (either uploaded to own registry or pulled from public registry).
     3. Install ibm-apiconnect-distributed.yaml with the following command:

        kubectl apply -f ibm-apiconnect-distributed.yaml

8. Verify that the ibm-datapower-operator and the ibm-apiconnect operators are restarted.
9. Ensure that the apiconnect operator re-created the necessary microservices:

   kubectl get apic -n <namespace>

   Do not proceed to upgrade the API Connect subsytems until the operator upgrade is complete.

   Note: If analytics backups are configured, then after the API Connect operator is upgraded, the analytics CR reports a status of Blocked until the analytics subsystem is upgraded to V10.0.8. With top-level CR deployments, the apiconnectcluster CR reports Pending until the analytics CR is no longer Blocked. For more information, see Analytics backup changes.
10. Upgrade the API Connect subsystems:
    1. Upgrading the management subsystem on Kubernetes
    2. Upgrading the portal subsystem on Kubernetes
    3. Upgrading the analytics subsystem on Kubernetes
    4. Upgrading the gateway subsystem on Kubernetes
11. Verify that the upgraded subsystems report as Running and the RECONCILED VERSION displays the new version of API Connect.

    Run the following command:

    kubectl get apic --all-namespaces

    Example response:

    NAME                                                READY   STATUS    VERSION              RECONCILED VERSION      AGE
    analyticscluster.analytics.apiconnect.ibm.com/analytics      8/8     Running   10.0.8.0   10.0.8.0-1074   121m
    
    NAME                                     PHASE     READY   SUMMARY                           VERSION    AGE
    datapowerservice.datapower.ibm.com/gw1   Running   True    StatefulSet replicas ready: 1/1   10.0.8.0   100m
    
    NAME                                     PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
    datapowermonitor.datapower.ibm.com/gw1   Running                false          false              100m
    
    NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
    gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   10.0.8.0   10.0.8.0-1074  100m
    
    NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
    managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   10.0.8.0   110.0.8.0-1074   162m
    
    
    NAME                                             READY   STATUS    VERSION              RECONCILED VERSION      AGE
    portalcluster.portal.apiconnect.ibm.com/portal   3/3     Running   10.0.8.0   10.0.8.0-1074   139m

    Important: If you need to restart the deployment, wait until all portal sites complete the upgrade.

    After the portal subsystem upgrade is complete, each portal site is upgraded. You can monitor the site upgrade progress from the MESSAGE column in the oc get ptl output. You can still use the portal while sites are upgrading, although a maintenance page is shown for any sites that are being upgraded. When the site upgrades are complete, the oc get ptl output shows how many sites the portal is serving:

    NAME     READY   STATUS    VERSION          RECONCILED VERSION   MESSAGE              AGE
    portal   3/3     Running   <version>        <version>            Serving 2 sites      22h

    On two data center disaster recovery deployments, the sites are not upgraded until both data centers are upgraded.

12. Optional: Provide a custom endpoint for the Consumer Catalog.

    Beginning with version 10.0.8.0, API Connect includes the Consumer Catalog feature. If your deployment uses cert-manager, then during the upgrade from 10.0.5.x or 10.0.7.0, a new endpoint is added to the management CR for the Custom Catalog.

    The following definition is generated and automatically added to the management CR during the upgrade, using a certificate from cert-manager:

      consumerCatalogEndpoint:
        annotations:
          cert-manager.io/issuer: ingress-issuer
        hosts:
        - name: consumer-catalog.<domain>
          secretName: <name>-consumer-catalog-endpoint

    1. Make the following optional changes to the endpoint definition in the CR:
       • Change the name to a host name of your own choosing.
       • Specify a custom cert-manager certificate.
    2. Run the following command to save and apply the CR: wq
13. Optional: For the optional components API Connect Toolkit and API Connect Local Test Environment, install the latest version of each after you complete the upgrade of the subsystems.