Upgrade paths to 5.0.8.1x and Issues while upgrading and post upgrade 5.0.8.1x

Problem

1. Creating a backup of an API Connect configuration.
   1. Also, it is highly recommended that you take VM Snapshot of API Connect appliances.
2. Upgrade paths to latest v5.0.8.11 or later
   1. If you are on any version prior to 5087ifix4 then upgrade to 5087-ifix4 // to save analytics data 
   2. Apply 50810ifix2 /// Minimum required for OS upgrade
      1. Run config autodiskadd disable on 50810ifix2
         1. run config autodiskadd status // to confirm status shows enable: false
      2. Add a NEW 10 GB Disk.  //If needed this can be removed after upgrade
   3. Apply Latest version
3. Upgrade steps for Portal
   1. If you are on any version prior to 5083 then upgrade to 5083: Debian to Ubuntu16 migration
   2. Upgrade to 5.0.8.8 as minimum version required for next steps. 
   3. 50810ifix: Ubuntu 16 to Ubuntu 18 migration
   4. 50813: General Portal upgrade steps
4. Refresh all Gateway services

1. ih_var filling to 100%
   1.  APAR LI82299-log rotation not working after applying 5.0.8.11 - is fixed from 5.0.8.12-ifix2.
      1. Fix is in ifix4 released in December, 2021 release of 5.0.8.12.
   2. APAR LI82414-disable dateext for log rotate. 
      1. Fix is in 5.0.8.13
2.  Customer not using correct upgrade command.
   If you see below error:

   Filetype is not valid. Please verify that the given file is a valid vcrypt2.
   FAILED: system update failed with result: -255

   Solution: use system update osfirmware not firmware. See Step 5
3. Sample error if 10GB disk is not added

   1. Error customer will see: 

      VM SNAPSHOT/Backup taken, please enter Y/N ?
      
      y
      
      Continuing Upgrade ....
      
      Beginning extraction to new disk....
      
      There is no hard disk available for upgrade. Please add 10GB hard disk.
      
      WARNING: system update completed with result: 256

   2. Solution: Run 'config autodiskadd disable' then add a new disk 10GB size.

4. If disk 2 is modified to anything other than 50GB, post upgrade, some customers will not see the full disk space used. 
   1. In below snippet you can see Disk has total data disk size is 100GB however stat show all command right after shows the total of /wip and /ih_var is less than 100GB.

      1. dipali1/APIConnect> system show platform
         Serial Number: VMWUHUIFLAQxxxxx
         Management MAC: 00:50:56: xx:xx:xx
         Encrypted disk partitions: 4 (APIM, sysrw, WIP, swap)
         Unencrypted disk partitions: 0
         CPU/s: 4
         Total Memory (MB): 7982.27
         Total Swap (MB): 9128.12
         Code Disk(GB):          10
         Data Disk(GB):          100
         Total Data Disk(GB):    100
         dipali1/APIConnect> stat show all
         DISK USAGE:
         File system          Size Used Avail Use% Mounted on
         devtmpfs            3.9G     0  3.9G   0% /dev
         tmpfs               3.9G     0  3.9G   0% /dev/shm
         tmpfs               3.9G   17M  3.9G   1% /run
         tmpfs               3.9G     0  3.9G   0% /sys/fs/cgroup
         /dev/mapper/flash2  9.3G  3.2G  6.1G  35% /
         /dev/sda1           472M   52M  416M  12% /boot
         /dev/mapper/sysrw    21G  401M   20G   2% /ih_var
         /dev/mapper/wip      21G   13G  6.9G  65% /wip
         tmpfs               799M     0  799M   0% /run/user/1001
      2. Solution: Make sure 'config autodiskadd status' shows enable: true. If not, enable it and then reboot. You may need to reboot multiple times for the correct disk space to be allocated. 
5. Post Portal upgrade disk space allocated is not seen
   1. Error: 

      WARNING: The virtual disk size has not been increased from the default of
      30GB. This is only sufficient for PoC and testing purposes. Please refer
      to the Knowledge Center for instructions on increasing the virtual disk size.
      

   2. Solution: APAR LI82459 Fix is released in 5.0.8.13
   3. Workaround: 

      Enter the following command:
      sudo lvextend /dev/ubuntu-vg/ubuntu-lv -l +100%FREE
      sudo resize2fs /dev/ubuntu-vg/ubuntu-lv
      Just do these commands from dev portal command line.

6. Network adaptor issue
   1. Error: After upgrade server does not come up
   2. Issue: 
   3. 
   4. Solution: 
      1. Delete any existing VM snapshot(better if RSS box)
      2. Powered off VM
      3. Add a new Device from edit settings in VM Client for Network adapter
      4. Which will duplicate the existing adapter, in there change the Adapter Type from VMXNET3 to E1000
      5. Power on VM
      6. The system should come up without issues