Upgrading management, portal, and analytics on VMware

Upgrade your management, portal, and analytics subsystems on VMware.

1. If some time has passed since you completed the Pre-upgrade preparation, repeat these pre-upgrade steps to be sure that your subsystems are ready for upgrade.
2. Block all external traffic to the management server.

   Upgrading API Connect requires downtime while files are transferred and updated. All external traffic to the management server from all sources (Cloud Manager UI, API Manager UI, CLI, and REST API) must be blocked. As a result, the Cloud Manager and API Manager UIs cannot be used during the upgrade. In addition, the developer portal cannot be used to create, update, or delete any applications, subscriptions, memberships, or consumer organizations during the upgrade.

3. Verify that the new apicup command is in your operating system's PATH variable:

   apicup version

   The version that is returned should be the version that you are upgrading to. If not, then refer to step 14 on Pre-upgrade preparation and checks VMware.

4. Upgrade the management subsystem:
   1. If you are upgrading to v10.0.7.0 and are not able to configure S3 object-store backups for your management subsystem, then set accept-no-backups to true to allow upgrade to proceed:

      apicup subsys set <subsystem name> accept-no-backups=true

      For more information about this limitation, see Planning your API Connect upgrade on VMware.
   2. Start the upgrade of the management subsystem:
      1. Install the management subsystem files.
         Note:

         If you have a two data center disaster recovery deployment and are upgrading to v10.0.7.0, all data on the warm-standby management subsystem is deleted during the upgrade, to be replaced by data from the upgraded active management subsystem.

         Include the flag --accept-warm-standby-upgrade-data-deletion in the apicup subsys install <mgmt subsystem> <upgrade_tar> command to acknowledge this temporary data deletion.

         If you have a two data center disaster recovery deployment, and are upgrading to v10.0.6.0, do not follow this step. Instead, follow the steps in Management subsystem two data center upgrade from v10.0.5.

         For more information about two data center disaster recovery upgrade, see Upgrading a two data center deployment.

         If you are upgrading to v10.0.7.0, include the --skip-health-check argument:

         apicup subsys install <subsystem name> <subsystem_upgrade_tar_archive> --skip-health-check

         For all other upgrade paths, use:

         apicup subsys install <subsystem name> <subsystem_upgrade_tar_archive>

         Notes:

         • If you are adding one or more control plane files, specify the file path to each control plane file:

           apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive> <path_to_control_plane_file_1> <path_to_control_plane_file_n>

         • If the upgrade gets stuck, run the following command to restart the appliance:

            sudo systemctl restart appliance-manager

      2. Verify that the upgrade was successful by running the apicup health check:

         apicup subsys health-check <subsystem_name>

      3. If upgrading to 10.0.7.0, verify that the health check did not return a false-positive response.
         Note: This step only applies if you upgraded to API Connect 10.0.7.0 and is due to a known issue that will be corrected in a later release.
         Run the following command to get the current version of API Connect on the upgraded subsystem:

         1. Log in to the VM:

            ssh apicadm@<hostname>

         2. Switch to root user:

            sudo -i

         3. Get the API Connect version:

            kubectl get apic

         If the health check indicates a successful upgrade but the subsystem is still running the older version of API Connect, skip the remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.

         If the subsystem is running the newer version of API Connect then the upgrade was successful; continue with the remaining steps in this procedure.

      4. If the health check indicates that a reboot is needed, complete the following steps on all management VMs:
         1. Login to the VM:

            ssh apicadm@<hostname>

         2. Switch to root user:

            sudo -i

         3. systemctl reboot

   3. Use apicops to validate the certificates.

      For more information about the apicops tool, see The API Connect operations tool: apicops.

      1. Login to the VM:

         ssh apicadm@<hostname>

      2. Switch to root user:

         sudo -i

      3. Run the following command:

         apicops upgrade:stale-certs

      4. Delete any stale cert-manager certificates.
         If a cert-manager certificate failed validation, delete the stale certificate secret to trigger cert-manager to regenerate it. Run the following command:

         kubectl delete secret <stale-secret>

      5. Restart the corresponding pod so that it can pick up the new secret.
         To determine which pod to restart, see the following topics:

         • List of ingress certificates
         • List of intra-subsystem certificates
5. If you have multiple management subsystems in the same project, set the portal subsystem's platform-api and consumer-api certificates to match the certificates used by their corresponding management subsystems.

   This step applies only if you installed more than one management subsystem into a single project.

   A portal subsystem can be associated with only one management subsystem. To associate the upgraded portal subsystem with the appropriate management subsystem, manually set the mgmt-platform-api and mgmt-consumer-api certificates to match the ones used by the management subsystem.

   1. Run the following commands to get the certificates from the management subsystem:

      apicup certs get mgmt-subsystem-name platform-api -t cert > platform-api.crt
      apicup certs get mgmt-subsystem-name platform-api -t key > platform-api.key
      apicup certs get mgmt-subsystem-name platform-api -t ca > platform-api-ca.crt
      
      apicup certs get mgmt-subsystem-name consumer-api -t cert > consumer-api.crt
      apicup certs get mgmt-subsystem-name consumer-api -t key > consumer-api.key
      apicup certs get mgmt-subsystem-name consumer-api -t ca > consumer-api-ca.crt

      where mgmt-subsystem-name is the name of the specific management subsystem that you want to associate the new portal subsystem with.

   2. Run the following commands to set the portal's certificates to match the certificates used by the management subsystem:

      apicup certs set ptl-subsystem-name platform-api Cert_file_path Key_file_path CA_file_path
      
      apicup certs set ptl-subsystem-name consumer-api Cert_file_path Key_file_path CA_file_path

6. Upgrade the portal subsystem.
   Attention: If you are upgrading a two data center disaster recovery deployment, upgrade the warm-standby data center first.

   For more information about two data center disaster recovery upgrade, see Upgrading a two data center deployment.

   1. Install the portal subsystem files:

      apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>

      Note: If you are adding one or more control plane files, specify the file path of each control plane file:

      apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive> <path_to_control_plane_file_1> <path_to_control_plane_file_n>

   2. Verify that the upgrade was successful by running a health check with the following command:

      apicup subsys health-check <subsystem_name>

      Important: If the health check indicates that a reboot is required, do not reboot at this stage. Reboot after all sites are upgraded, see step 7.
   3. Verify that the health check did not return a false-positive response.
      Run the following command to get the current version of API Connect on the upgraded subsystem:

      1. Log in to the VM:

         ssh apicadm@<hostname>

      2. Switch to root user:

         sudo -i

      3. Get the API Connect version:

         kubectl get apic

      If the health check indicates a successful upgrade but the subsystem is still running the older version of API Connect, skip the remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.

      If the subsystem is running the newer version of API Connect then the upgrade was successful; continue with the remaining steps in this procedure.

   4. If you have two data center disaster recovery deployment and upgraded the portal subsystem in the warm-standby data center, then start the upgrade on the active data center.
      Note: In a two data center disaster recovery deployment, after the warm-standby is upgraded, the active data center might enter a non-READY state. If this happens, do not be concerned, proceed to upgrade the portal subsystem in the active data center.
7. Verify portal sites are upgraded.

   Run the following toolkit commands to ensure that the portal site upgrades are complete:

   1. Log in as an admin user:

      apic login -s <server_name> --realm admin/default-idp-1 --username admin --password <password>

   2. Get the portal service ID and endpoint:

      apic portal-services:get -o admin -s <management_server_endpoint> \
                   --availability-zone availability-zone-default <portal-service-name> \
                   --output - --format json

   3. List the sites:

      apic --mode portaladmin sites:list -s <management_server_endpoint> \ 
                   --portal_service_name <portal-service-name> \ 
                   --format json

      Any sites currently upgrading display the UPGRADING status; any site that completed its upgrade displays the INSTALLED status and the new platform version. Verify that all sites display the INSTALLED status before proceeding.

      For more information about the sites command, see apic sites:list and Using the sites commands.

   4. After all sites are in INSTALLED state and show the new platform listed, run:

      apic --mode portaladmin platforms:list -s <server_name> --portal_service_name <portal_service_name>
      

      Verify that the new version of the platform is the only platform listed.

      For more information about the platforms command, see apic platforms:list and Using the platforms commands.

   5. If all portal sites are upgraded and the health check indicates that a reboot is needed, then run the following command on all portal VMs:

      systemctl reboot

8. Upgrade the analytics subsystem.
   1. Install the analytics subsystem files:

      apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>

      Note: If you are adding one or more control plane files, specify the file path of each control plane file:

      apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive> <path_to_control_plane_file_1> <path_to_control_plane_file_n>

   2. Verify that the upgrade was successful by running a health check with the following command:

      apicup subsys health-check <subsystem_name>

   3. If upgrading to 10.0.7.0, verify that the health check did not return a false-positive response.
      Note: This step only applies if you upgraded to API Connect 10.0.7.0 and is due to a known issue that will be corrected in a later release.

      Run the following command to get the current version of API Connect on the upgraded subsystem:

      kubectl get apic

      If the health check indicates a successful upgrade but the subsystem is still running the older version of API Connect, skip the remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.

      If the subsystem is running the newer version of API Connect then the upgrade was successful; continue with the remaining steps in this procedure.

   4. If the health check indicates that a reboot is needed, run the following commands on all analytics VMs:

      systemctl reboot

9. Take a backup of your project directory, and database backups of each upgraded API Connect subsystem; see Backing up and restoring on VMware.
10. Optional: Take a virtual machine (VM) snapshot of all your VMs; see Using VM snapshots for infrastructure backup and disaster recovery for details. This action requires a brief outage while the VMs in the subsystem cluster are shut down - do not take snapshots of running VMs, as they might not restore successfully.