Upgrade your management, portal, and analytics subsystems on VMware.
Before you begin
- Review Planning your API Connect upgrade on VMware.
- Complete the Pre-upgrade preparation.
- Enable keep-alive in your SSH client configuration to avoid disconnects during the transfer of
the upgrade images to your VMs. For more information, consult the documentation for the version of
SSH that is installed on your client system (where you run the apicup commands
from).
Procedure
Notes:
- If a backup of any of your subsystems is scheduled to run within a few hours, do not start an
upgrade.
- While the upgrade process is running, do not do any maintenance tasks such as rotating
key-certificates, or restoring from a backup.
- If you encounter an issue during the upgrade, see Troubleshooting upgrades on VMware.
- If some time has passed since you completed the Pre-upgrade preparation, repeat these pre-upgrade steps
to be sure that your subsystems are ready for upgrade.
-
Block all external traffic to the management server.
Upgrading API Connect requires downtime while files are transferred and updated. All external
traffic to the management server from all sources (Cloud Manager UI, API Manager UI, CLI, and REST
API) must be blocked. As a result, the Cloud Manager and API Manager UIs cannot be used during the
upgrade. In addition, the developer portal cannot be used to create, update, or delete any
applications, subscriptions, memberships, or consumer organizations during the upgrade.
- Verify that the new apicup command is in your
operating system's
PATH
variable:
- Upgrade the management
subsystem:
- If you are upgrading to v10.0.7.0 and are not able to configure
S3 object-store backups for your management subsystem, then set
accept-no-backups
to true
to allow upgrade to
proceed:
- Start the upgrade of the management subsystem:
- Install the management subsystem files.
Note:
If you have a two data center disaster recovery
deployment and are upgrading to v10.0.7.0, all data on the warm-standby management subsystem
is deleted during the upgrade, to be replaced by data from the upgraded active management
subsystem.
Include the flag --accept-warm-standby-upgrade-data-deletion
in the
apicup subsys install <mgmt subsystem> <upgrade_tar>
command to acknowledge this temporary data deletion.
If you have a two data center disaster recovery
deployment, and are upgrading to v10.0.6.0, do not follow this step. Instead, follow the steps in
Management subsystem two data center upgrade from v10.0.5.
For more information about two data center disaster recovery upgrade, see Upgrading a two data center deployment.
If you are upgrading to v10.0.7.0, include the
--skip-health-check
argument:
apicup subsys install <subsystem name> <subsystem_upgrade_tar_archive> --skip-health-check
For
all other upgrade paths,
use:apicup subsys install <subsystem name> <subsystem_upgrade_tar_archive>
- Verify that the upgrade was successful by running the
apicup health check:
apicup subsys health-check <subsystem_name>
- If upgrading to 10.0.7.0, verify that the health check did not return a false-positive
response.
Note: This step only applies if you
upgraded to API Connect 10.0.7.0 and is due to a known issue that will be corrected in a later
release.
Run the following command to get the current version of API Connect on the
upgraded subsystem:
- Log in to the VM:
ssh apicadm@<hostname>
- Switch to
root
user:sudo -i
- Get the API Connect version:
kubectl get apic
If the health check indicates a successful upgrade but the subsystem is still running
the older version of API Connect, skip the remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.
If the subsystem is running the newer
version of API Connect then the upgrade was successful; continue with the remaining steps in this
procedure.
- If the health check indicates that a reboot is needed, complete the following steps on all
management VMs:
- Login to the VM:
ssh apicadm@<hostname>
- Switch to
root
user:sudo -i
-
systemctl reboot
- Use
apicops
to validate the
certificates.
For more information about the apicops
tool, see
The API Connect operations tool: apicops.
- Login to the VM:
ssh apicadm@<hostname>
- Switch to
root
user:sudo -i
- Run the following command:
apicops upgrade:stale-certs
- Delete any stale cert-manager certificates.
If a cert-manager certificate failed validation, delete the stale certificate
secret to trigger cert-manager to regenerate it. Run the following
command:
kubectl delete secret <stale-secret>
- Restart the corresponding pod so that it can pick up the new secret.
To determine which pod to
restart, see the following topics:
- If you have multiple management subsystems in the same project, set the portal
subsystem's
platform-api
and consumer-api
certificates to match
the certificates used by their corresponding management subsystems.
This step applies only if you installed more than one management subsystem into a single
project.
A portal subsystem can be associated with only one management subsystem. To associate the
upgraded portal subsystem with the appropriate management subsystem, manually set the
mgmt-platform-api
and mgmt-consumer-api
certificates to match the
ones used by the management subsystem.
- Run the following commands to get the certificates from the management
subsystem:
apicup certs get mgmt-subsystem-name platform-api -t cert > platform-api.crt
apicup certs get mgmt-subsystem-name platform-api -t key > platform-api.key
apicup certs get mgmt-subsystem-name platform-api -t ca > platform-api-ca.crt
apicup certs get mgmt-subsystem-name consumer-api -t cert > consumer-api.crt
apicup certs get mgmt-subsystem-name consumer-api -t key > consumer-api.key
apicup certs get mgmt-subsystem-name consumer-api -t ca > consumer-api-ca.crt
where mgmt-subsystem-name
is the name of the specific
management subsystem that you want to associate the new portal subsystem with.
- Run the following commands to set the portal's certificates to match the certificates
used by the management subsystem:
apicup certs set ptl-subsystem-name platform-api Cert_file_path Key_file_path CA_file_path
apicup certs set ptl-subsystem-name consumer-api Cert_file_path Key_file_path CA_file_path
- Upgrade the portal subsystem.
Attention: If you are upgrading a
two data center disaster recovery deployment, upgrade the
warm-standby data center
first.
For more information about two data center disaster recovery upgrade, see Upgrading a two data center deployment.
- Install the portal subsystem
files:
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>
Note: If you are adding one or more control plane files, specify the file path of each control
plane
file:
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive> <path_to_control_plane_file_1> <path_to_control_plane_file_n>
- Verify that the upgrade was successful by running a health check with the following
command:
apicup subsys health-check <subsystem_name>
Important: If the health check indicates that a reboot is required, do not reboot at this
stage. Reboot after all sites are upgraded, see step
7.
- Verify that the health check did not return a false-positive
response.
Run the following command
to get the current version of API Connect on the upgraded subsystem:
- Log in to the VM:
ssh apicadm@<hostname>
- Switch to
root
user:sudo -i
- Get the API Connect version:
kubectl get apic
If the health check indicates a successful upgrade but the subsystem is still running
the older version of API Connect, skip the remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.
If the subsystem is running the newer
version of API Connect then the upgrade was successful; continue with the remaining steps in this
procedure.
- If you have two data center disaster recovery
deployment and upgraded the portal subsystem in the warm-standby data center, then
start the upgrade on the active data center.
Note: In a two data center disaster recovery deployment, after the warm-standby is upgraded, the
active data center might enter a non-READY state. If this happens, do not be concerned, proceed to
upgrade the portal subsystem in the active data center.
- Verify portal sites are upgraded.
Run the following toolkit commands to ensure that the portal site upgrades are complete:
- Log in as an admin
user:
apic login -s <server_name> --realm admin/default-idp-1 --username admin --password <password>
- Get the portal service ID and
endpoint:
apic portal-services:get -o admin -s <management_server_endpoint> \
--availability-zone availability-zone-default <portal-service-name> \
--output - --format json
- List the
sites:
apic --mode portaladmin sites:list -s <management_server_endpoint> \
--portal_service_name <portal-service-name> \
--format json
Any sites currently upgrading display the
UPGRADING
status; any site that completed its upgrade displays the
INSTALLED
status and the new platform version. Verify that all sites display the
INSTALLED
status before proceeding.
For more information about the
sites
command, see apic
sites:list
and Using the
sites
commands.
- After all sites are in
INSTALLED
state and show the new platform listed,
run:apic --mode portaladmin platforms:list -s <server_name> --portal_service_name <portal_service_name>
Verify that the new version of the platform is the only platform listed.
For
more information about the platforms
command, see apic platforms:list
and Using the
platforms
commands.
- If all portal sites are upgraded and the health check indicates that a reboot is needed, then
run the following command on all portal VMs:
systemctl reboot
- Upgrade the analytics subsystem.
- Install the analytics subsystem
files:
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>
Note: If you are adding one or more control plane files, specify the file path of each control
plane
file:
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive> <path_to_control_plane_file_1> <path_to_control_plane_file_n>
- Verify that the upgrade was successful by running a health check with the following command:
apicup subsys health-check <subsystem_name>
- If upgrading to 10.0.7.0, verify that the health check did not return a false-positive
response.
Note: This step only applies if you
upgraded to API Connect 10.0.7.0 and is due to a known issue that will be corrected in a later
release.
Run the following command to get the current version of API Connect on the
upgraded subsystem:
kubectl get apic
If the health check indicates a
successful upgrade but the subsystem is still running the older version of API Connect, skip the
remaining steps in this procedure and see False positive result from health check after upgrading to 10.0.7.0 for troubleshooting instructions.
If the subsystem is running the newer
version of API Connect then the upgrade was successful; continue with the remaining steps in this
procedure.
- If the health check indicates that a reboot is needed, run the following commands on all
analytics VMs:
systemctl reboot
- Take a backup of your project directory, and database backups of each
upgraded API Connect subsystem; see Backing up and restoring on VMware.
- Optional: Take a virtual machine (VM) snapshot of all your VMs; see Using VM snapshots for infrastructure backup and disaster recovery for details. This action requires a brief outage while
the VMs in the subsystem cluster are shut down - do not take snapshots of running VMs, as they might
not restore successfully.
What to do next
When your management, portal, and analytics subsystem are
upgraded, upgrade your DataPower Gateway Service. See Upgrading DataPower Gateway.