Upgrading Management subsystem on native Kubernetes

Upgrade the management subsystem to the latest version of API Connect.

1. Update the management CR for the new version of API Connect.
   1. Run the following command to edit the CR:

      kubectl -n $NAMESPACE edit mgmt <CLUSTERNAME>

      where CLUSTERNAME is the name specified in the subsystem CR at installation time.

   2. If upgrading from v10.0.4-ifix3, or upgrading from v10.0.1.7-eus (or higher): In the spec section, add a new allowUpgrade attribute and set it to true:

      spec:
        allowUpgrade: true

      The allowUpgrade attribute enables the upgrade to 10.0.5.0 or later. Because the upgrade to 10.0.5.x deletes your existing analytics data, the attribute is required to prevent an accidental upgrade. See Additional considerations for upgrading analytics from a version prior to 10.0.5.0. After the Management subsystem has been upgraded, the Analytics subsystem must also be upgraded.

   3. Update the API Connect version in the CR:

      For example:

      version: 10.0.5.9

   4. If you are upgrading to a version of API Connect that requires a new license, update the license value now.

      For example:

      license: L-GVEN-GFUPVE

      For the list of licenses, see API Connect licenses.

   5. Run the following command to save and apply the CR: wq

      When you save the updated CR, the upgrade starts automatically.

      Note: If you see an error message when you attempt to save the CR, check if it is one of the following known issues:

      • Webhook error for incorrect license.

        If you did not update the license ID in the CR, then when you save your changes, the following webhook error might display:

        admission webhook "vmanagementcluster.kb.io" denied the request: 
        ManagementCluster.management.apiconnect.ibm.com "management" is invalid: 
        spec.license.license: Invalid value: "L-RJON-BYGHM4": 
        License L-RJON-BYGHM4 is invalid for the chosen version 10.0.5.9. 
        Please refer license document https://ibm.biz/apiclicenses

        To resolve the error, see API Connect licenses for the list of the available license IDs and select the appropriate License IDs for your deployment. Update the CR with the new license value as in the following example, and then save and apply your changes again.

      • Webhook error: Original postgres primary is running as replica
        If you see the error message:

        Original PostgreSQL primary is running as replica, 
        please perform failover to original primary before attempting upgrade. Refer - https://ibm.biz/BdPLWU"

        you must complete a Postgres failover before the upgrade can proceed, see Postgres failover steps. After you apply the Postgres failover steps, the upgrade resumes automatically.
      • Webhook error: Original PostgreSQL primary not found
        If you see the error message:

        Original PostgreSQL primary not found. Upgrade is blocked. Set apiconnect-operator/db-primary-not-found-allow-upgrade: true annotation 
        to unblock the upgrade. A warning will be issued after the upgrade with further steps.
        

        take the following actions to complete the upgrade and fix the cause of the error message:

        1. Edit the management CR:

           kubectl -n $NAMESPACE edit mgmt <CLUSTERNAME>

        2. Add the following annotation:

           apiVersion: management.apiconnect.ibm.com/v1beta1
           kind: ManagementCluster
           metadata:
             annotations:
               apiconnect-operator/db-primary-not-found-allow-upgrade: true
               ...

        3. Continue with the upgrade. When the upgrade is complete, the management CR reports the warning:

           Original PostgreSQL primary not found. Run apicops upgrade:pg-health-check to check the health of the database and to ensure pg_wal symlinks exist. If database health check passes please perform a management database backup and restore to restore the original PostgreSQL primary pod

        4. Take a new management database backup.
        5. Immediately restore from the new backup taken in the previous step. The action of taking and restoring a management backup results in the establishment of a new Postgres primary, eliminating the CR warning message. Be careful to restore from the backup that is taken after the upgrade, and not from a backup taken before upgrade.
2. Run the following command to verify that the upgrade was successful:

   kubectl get apic -n <namespace>

   Example output after upgrading the management subsystem:

   NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
   managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   <version>   <version-build>   162m
   
   NAME                                                                    STATUS    MESSAGE                                              AGE
   managementdbupgrade.management.apiconnect.ibm.com/management-up-8c28g   Running   Upgrade is running (DB Schema/data are up-to-date)   13m

3. Restart all nats-server pods by running the following command:

   kubectl -n <namespace> delete po -l app.kubernetes.io/name=natscluster