SevOne NMS Upgrade Process Guide

1. Using ssh, log in to SevOne NMS appliance (Cluster Leader of SevOne NMS cluster) as root.

   $ ssh root@<NMS appliance>

2. Check the version your NMS appliance is running on. For example, the output below indicates that your NMS appliance is on SevOne NMS 6.8.0. To proceed with the upgrade to SevOne NMS 7.0.x, you must be on SevOne NMS 6.7.x or above.
   Example

   
   
   $ SevOne-show-version
   
   SevOne version:     6.8.0
   kernel version:     4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023
   nginx version:      0.0.0
   MySQL version:      10.6.12-MariaDB
   PHP version:        8.1.27
   SSH/SSL version:    OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
   REST API version:   2.1.47, Build time 2024-02-15T08:46:36+0000, Hash 3dd4faa
   
   Hardware Serial Number:  VMware-42
       Intel(R) Xeon(R) CPU
       2 cores @ 2199.998MHz
       8GB RAM
       31GB SWAP
       569 GB / Partition
   

   Important: If Openstack cluster,

   
   $ mkdir /data/upgrade
   
   $ cd /data/upgrade

3. Verify the OS version. Confirm that version is Red Hat Enterprise Linux (RHEL).

   $ cat /etc/redhat-release
   Red Hat Enterprise Linux release 8.9 (Ootpa)

4. Using curl -kO, copy the signature tools checksum file (signature-tools-<latest version>-build.<###>.tgz.sha256.txt) received from SevOne to /opt directory. For example, signature-tools-2.0.3-build.1.tgz.sha256.txt.
5. Using curl -kO, copy the signature tools file (signature-tools-<latest version>-build.<###>.tgz) received from SevOne to /opt directory. For example, signature-tools-2.0.3-build.1.tgz.
6. Verify the signature tools checksum file from /opt directory.

   $ cd /opt

   $ sha256sum --check signature-tools-<latest version>-build.<###>.tgz.sha256.txt

   Example

   $ sha256sum --check signature-tools-v2.0.3-build.1.tgz.sha256.txt

7. Extract the signature tools tar file.

   $ tar -xzvf signature-tools-<latest version>-build.<###>.tgz -C /

   Example

   $ tar -xzvf signature-tools-v2.0.3-build.1.tgz -C /

8. Using curl -O, copy the forward tarball file (<forward SevOne NMS tarball>.tar.gz) received from SevOne to /opt directory but if Openstack, to /data/upgrade. For example, tarball file for SevOne NMS 7.0.1.
9. Using curl -O, copy the checksum file (<SevOne NMS>.sha256.txt) received from SevOne to /opt directory. However, if Openstack, to /data/upgrade directory.
10. Change directory.

    not Openstack

    $ cd /opt

    for Openstack

    $ cd /data/upgrade

11. (optional) Validate the signature for the forward tarball.

    Ensure valid & trusted certificate is used,

    
    
    $ SevOne-validate-image -i v7.0.1-build<enter build number>.tar.gz -s v7.0.1-build<enter build number>.tar.gz.sha256.txt
    
    INFO: Extracting code-signing certificates from image file...
    Image signed by SevOne Release on Sat, 25 May 2024 00:05:30 +0000.
    
    The certificate is trusted.
    Certificate subject=
        commonName                = International Business Machines Corporation
        organizationalUnitName    = IBM CCSS
        organizationName          = International Business Machines Corporation
        localityName              = Armonk
        stateOrProvinceName       = New York
        countryName               = US
    
    Certificate issuer=
        commonName                = DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
        organizationName          = DigiCert, Inc.
        countryName               = US
    
    INFO:  Checking the signature of the image
    
    The image can be installed.

    Please contact SevOne Support Team if the certificate is not trusted or the signature does not match.

12. Before starting the upgrade, it is recommended that you perform the upgrade pre-checks to identify and address the known potential upgrade blocking issues.

    
    SevOne-act check checkout --full-cluster
    
    SevOne-act check listening-ports --full-cluster
    
    SevOne-act stk checkout --full-cluster
    
    SevOne-act stk checkout-full --full-cluster
    
    SevOne-act stk get-peer-time --full-cluster
    
    SevOne-act stk get-free-disk-space --full-cluster
    
    SevOne-act stk partition-read-state --full-cluster
    

13. Once the pre-check completes successfully and you have identified and addressed the known potential upgrade blocking issues, validate the signature and install the forward tarball.
    Important: If you want to upgrade using the self-service upgrade, please refer to section using Self-Service Upgrade. Else, continue.

    
    $ SevOne-validate-image -i v7.0.1-build<enter build number>.tar.gz -s v7.0.1-build<enter build number>.tar.gz.sha256.txt \
    --installer nms

    Important: It can take ~60 - 75 minutes to perform the upgrade - please wait until it completes and you see something similar to the following.

    Example

    ...
    ...
    ... cleanup : Remove nmsUpgradeRepo
    -------------------------------------------------------------------------------------
    ------------------------------------ 0.89s
    cleanup : Check symlink for nmsDowngradeRepo
    -------------------------------------------------------------------------------------
    ----------------------- 0.63s
    cleanup : Remove dereferenced symlink
    -------------------------------------------------------------------------------------
    ------------------------------ 0.03s
    cleanup : Remove nmsDowngradeRepo
    -------------------------------------------------------------------------------------
    ---------------------------------- 0.63s
    cleanup : Copy ansible.cnf to other appliances
    -------------------------------------------------------------------------------------
    --------------------- 0.02s
    cleanup : Upgrade: Remove the flag file for AWS metadata namespaces check if
    upgrading to 6.5.0 or higher ----------------------------------------------- 0.03s
    cleanup : Upgrade: Remove the flag file for AWS metadata namespaces check if
    upgrading to 6.6.0 or higher ----------------------------------------------- 0.02s
    cleanup : Downgrade: Remove aws-nms-collector containers (NMS-80065)
    ------------------------------------------------------------------------------------
    0.01s
    cleanup : Downgrade: Remove AWS-related metadata namespaces and attributes
    (NMS-80065) ------------------------------------------------------------------ 0.01s
    cleanup : Downgrade: Remove AWS-related metadata mappings in 6.6.0
    -------------------------------------------------------------------------------------
    - 0.01s
    cleanup : Downgrade: Remove AWS-related metadata namespaces and attributes in 6.6.0
    --------------------------------------------------------------------- 0.01s
    cleanup : Check if package compat-openssl10 is present and remove it
    ------------------------------------------------------------------------------------
    2.23s
    cleanup : Clean out upgrade/downgrade directories from /opt and the package lists
    ----------------------------------------------------------------------- 5.05s
    cleanup : Clean up mariadb temporary files
    -------------------------------------------------------------------------------------
    ------------------------- 0.03s
    cleanup : Clean up mysql temporary files
    -------------------------------------------------------------------------------------
    --------------------------- 0.03s
    cleanup : Clean up RHEL upgrade temporary files
    -------------------------------------------------------------------------------------
    -------------------- 0.64s
    Turning on SevOne-masterslaved after the upgrade has finished
    -------------------------------------------------------------------------------------
    ------ 5.35s
    Let cluster_state table know upgrade has finished
    -------------------------------------------------------------------------------------
    ------------------ 0.02s
    Remove hosts.ini file
    -------------------------------------------------------------------------------------
    ---------------------------------------------- 0.61s
    Add upgrade entry in nms_upgrade_history table in database
    -------------------------------------------------------------------------------------
    --------- 0.03s
    ===============================================================================
    root@sevone:/opt/ansible [7.0.1] [04:01:45] $

    Note: Option --installer-opts allows flags to be passed to the installer. The following options are case-sensitive parameters. If more than one flag/parameter is passed, you must pass them in single / double-quotes.

    IMPORTANT: For pre-upgrade flags -e and -f,

    Flag -e can be added to option --installer-opts to skip the pre-upgrade errors.

    Flag -f can be added to option --installer-opts to skip the pre-upgrade checks and to force the install.

    However, there are certain pre-checks that do not get skipped even if -e or -f flag is passed. For example, when performing an upgrade, your current SevOne NMS version must be a version that is prior to the SevOne NMS version you are upgrading to. Otherwise, you will get 'Starting version of NMS should be less than the forward version of NMS.' message.

    • -a: Avoid playbook tags to run.
    • -c: Prevents hosts.ini from being automatically regenerated. If this flag is not passed as an option, hosts.ini will be automatically regenerated.
    • -e: Skip pre-upgrade errors if found, applicable only when run WITHOUT -f option.
    • -f: Skip pre-upgrade checks and force install.
    • -n: Don't start in a screen session. Used for automated builds.
    • -s: Run the upgrade without the UI logger.
    • -x: Run pre-upgrade checks with --hub-spoke-network option.
    • -h: Show this help.

    For example,

    To run the upgrade by skipping pre-upgrade checks all together,

    
    $ SevOne-validate-image -i v7.0.1-build<enter build number>.tar.gz -s v7.0.1-build<enter build number>.tar.gz.sha256.txt \
    --installer nms --installer-opts -f

    To run the upgrade with the hub-spoke flag,

    
    $ SevOne-validate-image -i v7.0.1-build<enter build number>.tar.gz -s v7.0.1-build<enter build number>.tar.gz.sha256.txt \
    --installer nms --installer-opts -x

    1. The installer starts a screen session named ansible-{version}.
       Important: The screen session on the terminal must not be detached as the ongoing process is in the memory and packages may no longer be available on the appliance during the upgrade.

       After the ansible-playbook execution completes in the screen, you must exit the screen session.

       Using a separate ssh connection to the boxes while upgrading, may show some PHP / other warnings on the terminal. SevOne recommends you to wait until after the upgrade process completes successfully.

    2. Packages on all peers / hosts are updated at the same time. [tag: prepare_rpm, install_rpm, docker_setup]
    3. Database migrations are run on the clustermaster (mysqlconfig and mysqldata) and active peers (mysqldata only). [tag: database]
    4. System patches are run on all peers at the same time. [tag: systempatches]
    5. Cleanup actions on all hosts is performed last. [tag: cleanup]
    6. SevOne NMS 7.0.1 is installed on your machine.

       Upgrade opens a TUI (Text-based User Interface) or Terminal User Interface window which splits the progress into 3 columns.

       Important: If you are unable to interact with the screen session, you must still be able to view the progress in the Column 1. Allow the upgrade to complete.
       • Column 1: Host panel - shows the progress per host.
       • Column 2: Tasks panel - shows the tasks being executed.
       • Column 3: Logs panel - shows the logs associated with each task.
         Important: FYI

         • Press F1 to show / hide HELP.
         • Ctrl+C - kills ansible execution.

         To navigate between panel / column, press:

         • 1 - to select Column 1 (Host panel)
         • 2 - to select Column 2 (Tasks panel)
         • 3 - to select Column 3 (Logs panel)
         • up arrow - to move cursor / log up
         • down arrow - to move cursor / log down
         • left arrow - to move cursor / log left
         • right arrow - to move cursor / log right

         To detach from screen mode, Ctrl+A followed by the letter d. To attach to the screen mode, enter screen r.

         Upgrade

         
         
         

         Click on F1 to show the HELP menu in the logger

         
         
         
14. After successfully upgrading to SevOne NMS 7.0.x, check the version to ensure that you are on SevOne NMS 7.0.x. i.e., SevOne NMS 7.0.1 as shown in the example below.

    Example

    
    $ podman exec -it nms-nms-nms /bin/bash
    

    
    $ SevOne-show-version
    
    SevOne version:     7.0.1
    kernel version:     4.18.0-553.el8_10.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023
    nginx version:      1.14.1
    MySQL version:      10.6.18-MariaDB
    PHP version:        8.3.7
    SSH/SSL version:    OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
    REST API version:   2.1.47, Build time 2024-05-16T06:53:00+0000, Hash 07f225e
    
    Can't read memory from /dev/mem
    Hardware Serial Number:  
        Intel(R) Xeon(R) CPU
        2 cores @ 2199.998MHz
        8GB RAM
        31GB SWAP
        569 GB / Partition
        569 GB /data partition
    

1. Using ssh, log in to SevOne NMS appliance (Cluster Leader of SevOne NMS cluster) as root.

   $ ssh root@<NMS appliance>

2. Confirm that NMS appliance is running version SevOne NMS 7.0.x. i.e., SevOne NMS 7.0.1 as shown in the example below.

   Example

   
   $ podman exec -it nms-nms-nms /bin/bash
   

   
   $ SevOne-show-version
   
   SevOne version:     7.0.1
   kernel version:     4.18.0-553.el8_10.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023
   nginx version:      1.14.1
   MySQL version:      10.6.18-MariaDB
   PHP version:        8.3.7
   SSH/SSL version:    OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
   REST API version:   2.1.47, Build time 2024-05-16T06:53:00+0000, Hash 07f225e
   
   Can't read memory from /dev/mem
   Hardware Serial Number:  
       Intel(R) Xeon(R) CPU
       2 cores @ 2199.998MHz
       8GB RAM
       31GB SWAP
       569 GB / Partition
       569 GB /data partition
   

3. Verify the OS version. Confirm that version is Red Hat Enterprise Linux.

   $ cat /etc/redhat-release
   Red Hat Enterprise Linux release 8.9 (Ootpa)

4. Go to folder /data.

   $ cd /data

5. Make directory downgrade

   $ mkdir dowgrade

6. Go to folder /data/downgrade>

   $ cd /data/downgrade

7. Using curl -O, copy the signature tools checksum file (signature-tools-<latest version>-build.<###>.tgz.sha256.txt) received from SevOne to /data/downgrade directory. For example, signature-tools-2.0.3-build.1.tgz.sha256.txt.
8. Using curl -O, copy the signature tools file (signature-tools-<latest version>-build.<###>.tgz) received from SevOne to /data/downgrade directory. For example, signature-tools-2.0.3-build.1.tgz.
9. Verify the signature tools checksum file.

   $ sha256sum --check signature-tools-<latest version>-build.<###>.tgz.sha256.txt

   Example

   $ sha256sum --check signature-tools-v2.0.3-build.1.tgz.sha256.txt

10. Extract the signature tools tar file to /data/downgrade directory.

    $ tar -xzvf signature-tools-<latest version>-build.<###>.tgz -C /

    Example

    $ tar -xzvf signature-tools-v2.0.3-build.1.tgz -C /

11. Using curl -O, copy the reverse tarball file (<reverse SevOne NMS tarball>.tar.gz) received from SevOne to /data/downgrade directory. For example, tarball file for SevOne NMS 6.<x>.<y>.
12. Using curl -O, copy the checksum file (<reverse SevOne NMS tarball>.sha256.txt) received from SevOne to /data/downgrade directory.
13. Change directory.

    $ cd /data/downgrade

14. (optional) Validate the signature for the reverse tarball.

    
    $ podman exec -it nms-nms-nms /bin/bash
    

    
    $ SevOne-validate-image -i v7.0.1-to-v6.x.y-build<enter build number>.tar.gz -s v7.0.1-to-v6.x.y-build<enter build number>.tar.gz.sha256.txt
    
    INFO: Extracting code-signing certificates from image file...
    Image signed by SevOne Release on Sat, 25 May 2024 00:04:54 +0000.
    
    The certificate is trusted.
    Certificate subject=
        commonName                = International Business Machines Corporation
        organizationalUnitName    = IBM CCSS
        organizationName          = International Business Machines Corporation
        localityName              = Armonk
        stateOrProvinceName       = New York
        countryName               = US
    
    Certificate issuer=
        commonName                = DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
        organizationName          = DigiCert, Inc.
        countryName               = US
    
    INFO:  Checking the signature of the image
    
    The image can be installed.
    
    

    Important: When downgrading from SevOne NMS 7.0.x, you can only downgrade to SevOne NMS 6.<x>.<y> version you upgraded from.
15. Before starting the downgrade, clean up the /data/downgrade directory, especially the installRPM.tar file and the ansible folder.

    $ rm /data/downgrade/installRPMs.tar
    
    $ rm -rf /data/downgrade/ansible

16. Validate the signature and install the reverse tarball.

    
    $ podman exec -it nms-nms-nms /bin/bash
    
    $ SevOne-validate-image -i v7.0.1-to-v6.x.y-build<enter build number>.tar.gz -s v7.0.1-to-v6.x.y-build<enter build number>.tar.gz.sha256.txt \
    --installer reverse

    Important: It can take ~30 - 35 minutes to perform the downgrade - please wait until it completes and you see something similar to the following.

    Example

    ...
    ...
    ...
    database : Restore previous NMS net.settings ----------------------------------------------------------- 0.38s
    database : Finding the upgrade migration schemas ------------------------------------------------------- 0.06s
    database : set_fact ------------------------------------------------------------------------------------ 0.05s
    database : Finding the downgrade migration schemas ----------------------------------------------------- 0.01s
    database : set_fact ------------------------------------------------------------------------------------ 0.06s
    Applying data database patches ------------------------------------------------------------------------- 0.12s
    Reverting data database patches ------------------------------------------------------------------------ 0.19s
    database : Remove time_enabled column for 6.8.0 only --------------------------------------------------- 0.03s
    cleanup : Check symlink for nmsUpgradeRepo ------------------------------------------------------------- 0.65s
    cleanup : Remove dereferenced symlink ------------------------------------------------------------------ 0.06s
    cleanup : Remove nmsUpgradeRepo ------------------------------------------------------------------------ 0.68s
    cleanup : Check symlink for nmsDowngradeRepo ----------------------------------------------------------- 0.65s
    cleanup : Remove dereferenced symlink ------------------------------------------------------------------ 0.06s
    cleanup : Remove nmsDowngradeRepo ---------------------------------------------------------------------- 0.69s
    cleanup : Clean repo files in /etc/yum.repos.d --------------------------------------------------------- 1.42s
    cleanup : Clean out upgrade/downgrade directories from /opt and the package lists ---------------------- 6.07s
    Let cluster_state table know downgrade has finished ---------------------------------------------------- 0.08s
    Clean out tmp files ------------------------------------------------------------------------------------ 7.42s
    Remove hosts.ini file ---------------------------------------------------------------------------------- 0.62s
    =============================================================================== 
    root@sevone ansible]#

    Note: Option --installer-opts allows flags to be passed to the installer. The following options are case-sensitive parameters. If more than one flag/parameter is passed, you must pass them in single / double-quotes.

    • -a: Avoid playbook tags to run.
    • -e: Skip pre-upgrade errors if found, applicable only when run WITHOUT -f option.
    • -f: Skip pre-upgrade checks and force install.
    • -n: Don't start in a screen session. Used for automated builds.
    • -s: Run the upgrade without the UI logger.
    • -x: Run pre-upgrade checks with --hub-spoke-network option.
    • -h: Show this help.

    For example,

    To run the downgrade by skipping pre-upgrade checks all together,

    
    $ podman exec -it nms-nms-nms /bin/bash
    
    $ SevOne-validate-image -i v7.0.1-to-v6.x.y-build<enter build number>.tar.gz -s v7.0.1-to-v6.x.y-build<enter build number>.tar.gz.sha256.txt \
    --installer reverse --installer-opts -f

    To run the downgrade with the hub-spoke flag,

    
    $ podman exec -it nms-nms-nms /bin/bash
    
    $ SevOne-validate-image -i v7.0.1-to-v6.x.y-build<enter build number>.tar.gz -s v7.0.1-to-v6.x.y-build<enter build number>.tar.gz.sha256.txt \
    --installer reverse --installer-opts -x

    To run the downgrade by skipping pre-upgrade check errors,

    
    $ podman exec -it nms-nms-nms /bin/bash
    
    $ SevOne-validate-image -i v7.0.1-to-v6.x.y-build<enter build number>.tar.gz -s v7.0.1-to-v6.x.y-build<enter build number>.tar.gz.sha256.txt \
    --installer reverse --installer-opts -e

    1. The installer starts a screen session named ansible-{version}.
       Important: The screen session on the terminal must not be detached as the ongoing process is in the memory and packages may no longer be available on the appliance during the downgrade.

       After the ansible-playbook execution completes in the screen, you must exit the screen session.

       Using a separate ssh connection to the boxes while downgrading, may show some PHP / other warnings on the terminal. SevOne recommends you to wait until after the downgrade process completes successfully.

    2. Packages on all peers / hosts are updated at the same time. [tag: prepare_rpm, install_rpm, docker_setup]
    3. Reverse database migrations are run on the clustermaster (mysqlconfig and mysqldata) and active peers (mysqldata only). [tag: database]
    4. System patches are run on all peers at the same time. [tag: systempatches]
    5. Cleanup actions on all hosts is performed last. [tag: cleanup]
    6. If you were on SevOne NMS 7.0.1, SevOne NMS 6.8.0 is now installed on your machine.

       Downgrade opens a TUI (Text-based User Interface) or Terminal User Interface window which splits the progress into 3 columns.

       Important: If you are unable to interact with the screen session, you must still be able to view the progress in the Column 1. Allow the downgrade to complete.
       • Column 1: Host panel - shows the progress per host.
       • Column 2: Tasks panel - shows the tasks being executed.
       • Column 3: Logs panel - shows the logs associated with each task.
         Important: FYI

         • Press F1 to show / hide HELP.
         • Ctrl+C - kills ansible execution.

         To navigate between panel / column, press:

         • 1 - to select Column 1 (Host panel)
         • 2 - to select Column 2 (Tasks panel)
         • 3 - to select Column 3 (Logs panel)
         • up arrow - to move cursor / log up
         • down arrow - to move cursor / log down
         • left arrow - to move cursor / log left
         • right arrow - to move cursor / log right

         To detach from screen mode, Ctrl+A followed by the letter d. To attach to the screen mode, enter screen r.

         Downgrade

         
         
         

         Click on F1 to show the HELP menu in the logger

         
         
         
17. Once the downgrade is complete, check the kernel packages installed.

    Check kernel version

    
    $ SevOne-show-version
    OR
    $ rpm -qa | grep kernel
    OR
    $ uname -r

    Important: Kernel automatically gets updated as part of the downgrade and not every NMS release has a new kernel.

    Depending on the NMS release, kernel versions must be:

    SevOne NMS Version	Kernel Version
    NMS 7.0.1	4.18.0-553.el8_10
    NMS 6.8.0	4.18.0-513.11.1.el8_9.x86_64
    NMS 6.7.0	4.18.0-499.el8.x86_64

    Kernel version is based on the NMS release.

    Kernel version must be based on the NMS release. If not, you must reboot the entire cluster by executing the step below to apply the new kernel otherwise, reboot is not required.

    
    $ SevOne-shutdown reboot

18. Confirm that NMS appliance is running version SevOne NMS 6.8.0.

    
    $ SevOne-show-version
    
    SevOne version:     6.8.0
    kernel version:     4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023
    nginx version:      0.0.0
    MySQL version:      10.6.12-MariaDB
    PHP version:        8.1.27
    SSH/SSL version:    OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
    REST API version:   2.1.47, Build time 2024-02-15T08:46:36+0000, Hash 3dd4faa
    
    Hardware Serial Number:  VMware-42
        Intel(R) Xeon(R) CPU
        2 cores @ 2199.998MHz
        8GB RAM
        31GB SWAP
        569 GB / Partition
    

19. Clean up the /data/downgrade directory after the downgrade, especially the installRPM.tar file and the ansible folder.

    $ rm /data/downgrade/installRPMs.tar
    
    $ rm -rf /data/downgrade/ansible

20. Execute the following command to identify errors, if any.

    $ SevOne-act check checkout --full-cluster --verbose

Obtain the URL by using the GUI Installer or Command Line Interface. Enter the URL obtained in the browser of your choice to perform the steps in section Upgrade Stages.

1. Using the browser of your choice, enter the URL that the installer has returned in the step above. For example, https://10.49.14.220:9443/.
2. Enter the login credentials to launch the upgrade stages.
   
   
   

After all the upgrade stages have completed successfully, please refer to section post-Upgrade Steps below.

The default port is 9443. You may change the port from SevOne NMS > Administration > Cluster Manager > Cluster Settings tab > Ports subtab > field SevOne-gui-installer Port. After changing the port, you must regenerate the installer URL from Administration > Cluster Manager > Cluster Upgrades tab > click Run Installer.

SevOne recommends you to use any free port (i.e., port that is not used by any other service) in the range 1024 - 65535.

Execute the following command to check port availability.

$ netstat -anp |grep tcp |grep LISTEN |grep <port_number_desired>

Self-Service Upgrade running on gunicorn server uses the self-signed certificates of the nginx server by default.

"SSL_CERT_PATH": "/secrets/nginx/nginx.crt",
"SSL_KEY_PATH": "/secrets/nginx/nginx.key"

Execute the following command to restart gunicorn.

$ systemctl restart sevone-installer-gunicorn

If the custom certificate is based on hostname, execute the following steps.

1. Using a text editor of your choice, edit /opt/sevone_installer/SevOne-django-settings.sh file.

   $ vi /opt/sevone_installer/SevOne-django-settings.sh

2. Update the variables.

   # Change IP, HOSTNAME (variables value to hostname or ip address
   # whichever set in valid SSL cert), SSL_CERT_PATH & SSL_KEY_PATH
   IP='<hostname_of server>'
   HOSTNAME='<hostname_of_server>'

   Note: /opt/sevone_install/SevOne-django-settings.sh file contains the following by default.

   # Use nginx cert and key, generate open-ssl if (nginx cert/key) doesn't exist.
   NGINX_CERT='/secrets/nginx/nginx.crt'
   NGINX_KEY='/secrets/nginx/nginx.key'
   
   if [[ -f "$NGINX_CERT" && -f "$NGINX_KEY" ]];
   then
   SSL_CERT_PATH="$NGINX_CERT";
   SSL_KEY_PATH="$NGINX_KEY";

3. Run SevOne-django-settings.sh script.

   $ ./opt/sevone_installer/SevOne-django-settings.sh

4. Restart gunicorn.

   $ systemctl restart sevone-installer-gunicorn

The upgrade script, SevOne-validate-image, creates the following files.

• /opt/sevone_installer/config.json
• /opt/sevone_installer/build/static/env.json

Variable SEVONE_INSTALLER_BIND_IP in config.json file contains the IPv6 address or the IP address / FQDN of the Cluster Leader stored in the peers table.

In case where proxy is set, the config.json and env.json files must be updated manually and the installer service must be restarted. Execute the following steps.

• Using a text editor of your choice, edit /opt/sevone_installer/config.json to change the IP address to the proxy IP address / FQDN of the Cluster Leader in variable SEVONE_INSTALLER_BIND_IP. If you are using certificates, then FQDN of the Cluster Leader must be assigned to variable SEVONE_INSTALLER_BIND_IP.
• Using a text editor of your choice, edit /opt/sevone_installer/build/static/env.json to change the IP address to the proxy IP address / FQDN of the Cluster Leader in variable url. If you are using certificates, then FQDN of the Cluster Leader must be assigned to variable url.
• Change directory to /opt/sevone_installer.

$ cd /opt/sevone_installer

$ python3 manage.py collectstatic

Restart the installer service

$ systemctl restart sevone-installer-gunicorn.service

The upgrade artifact must be placed in/opt directory or /data/upgrade directory (for Openstack) of the active Cluster Leader for the installer to detect an upgrade. If the SevOne NMS cluster is already on the target version, the installer will not detect an available upgrade path and return a message that no update is available.

If pre-checks fail, you will not be allowed to continue with the upgrade. This is intentional. SevOne does not support forcing an upgrade on corrupted environments. However, you can force an upgrade from the CLI, just like before, and then monitor the progress from the User Interface.

If upgrade fails, you may download the cluster and/or individual peer logs and contact SevOne Support from the link in error message. For debugging, old steps of checking the upgrade in screen and logs/retry attempts all remain the same.

No. Two upgrades cannot be run at the same time. If an upgrade is running and a new installer instance is launched, it will show the status of the ongoing upgrade only.

No. All peers must be reachable from the Cluster Leader to run an upgrade. The requirements are the same as before. However, if after starting an upgrade, a peer becomes unreachable, you can see its status on the installer's User Interface and the upgrade will stop, like before.

Nothing has changed here. Like before, the installer does not handle HSA failover. If there is a failover after setting up the installer, the setup will have to be done separately on the now active Cluster Leader.

• Make sure you are accessing the URL with https and, accept the certificate warning.
• Check the status of the installer service systemctl status sevone-installer-gunicorn.service.
• If systemctl status sevone-installer-gunicorn.service is running and you are still unable to access the URL then, check the gunicorn log file for more details.
  • /var/log/SevOne-gui-installer/gunicorn.log OR
  • /var/log/SevOne-gui-installer/access.log
• Check request status codes 500 or 404 in /var/log/SevOne-gui-installer/access.log.

No. The installer has to be run from the active Cluster Leader only.

• Check log files in /var/log/SevOne-gui-installer/.
• Stop installer service systemctl stop sevone-installer-gunicorn.service if it is running.
• Using the text editor of your choice, edit /opt/sevone_installer/sevone_installer/settings.py file. Change variable DEBUG to TRUE.
• Run development web server like python3 manage.py runserver 0.0.0.0:1234.

1. Using ssh, log in to SevOne NMS appliance (Cluster Leader of SevOne NMS cluster) as root.

   $ ssh root@<NMS appliance>

2. Check the current SOA version.

   Example

   $ rpm -qa |grep soa
   SevOne-soa-6.8.0-1.28.el8.x86_64

   Note: If you need to upgrade the SOA version, then continue with the steps below. Else, you are all set.
3. Using curl -O, copy the SOA .rpm file to /opt directory but if Openstack, to /data/upgrade to the Cluster Leader and its peers.
   Important: The latest SOA .rpm file can be downloaded from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact SevOne Support Team for the file.
4. Upgrade SOA version. This step will perform the upgrade and restart SOA.

   Example

   not Openstack

   $ SevOne-peer-do 'rpm -Uvh /opt/SevOne-soa-7.0.0-1.19.el8.x86_64'

   for Openstack

   $ SevOne-peer-do 'rpm -Uvh /data/upgrade/SevOne-soa-7.0.0-1.19.el8.x86_64'

5. Check the SOA version to make sure that it is on the version you have upgraded SOA to.

   Example

   $ rpm -qa |grep soa
   SevOne-soa-7.0.0-1.19.el8.x86_64

Please refer to section Expand Logical Volume below.

Please refer to AWS Quick Start Guide for details.

Latest versions of Chrome, Firefox, and Safari are supported. Internet Explorer (IE) is not supported.

Prior to rebooting the appliances in the cluster using one of the options mentioned above, you must make note of the appliance that is the Cluster Leader in the cluster. From Administration > Cluster Manager > Cluster Overview tab > field Cluster Leader provides the name of the appliance that is the leader. For example, pandora-01 as shown in the screenshot below, is the Cluster Leader.

It is not necessary to perform failover / failback while performing reboot of appliances, but to minimize the polling downtime due to a reboot, this option can be used to perform failover, reboot, and failback between the primary / secondary appliances.

The failover operation must be done manually on a per peer basis, and it is not recommended to perform failover on more than one peer at the same time. However, the reboot of multiple appliances can be done in batches of 4 to 5 appliances at the same time. It is important to note that the failover steps for Cluster Leader pair must be done last once all other appliances have been rebooted.

If the upgrade is in a complete maintenance window and an outage on polling is acceptable for the duration of the restart of the appliances, then you can choose to perform the restart of the appliances in any order. However, SevOne always recommends restart of the passive appliances first and then, restart the active appliances for all the peers.

Execute the following script from the active Cluster Leader.

$ for IP in $(SevOne-peer-list); do echo -en "IP: $IP \t"; ssh $IP 'echo -e "Hostname: $(hostname) \t
System Uptime: $(uptime)" '; done

1. Perform some cleanup.

   $ rm /opt/installRPMs.tar
   
   $ rm -rf /opt/ansible

2. After the reboot completes successfully, SSH back to your active Cluster Leader of SevOne NMS cluster to check the installed kernel packages. Please see the table below to confirm that you have the correct kernel version.

   Check kernel version

   
   $ podman exec -it nms-nms-nms /bin/bash
   
   $ SevOne-show-version
   OR
   $ rpm -qa | grep kernel
   OR
   $ uname -r

   Important: Kernel automatically gets updated as part of the upgrade and not every NMS release has a new kernel.
   Depending on the NMS release, kernel versions must be:

   SevOne NMS Version	Kernel Version
   NMS 7.0.1	4.18.0-553.el8_10
   NMS 6.8.0	4.18.0-513.11.1.el8_9.x86_64
   NMS 6.7.0	4.18.0-499.el8.x86_64

   Kernel version is based on the NMS release.

   Kernel version must be based on the NMS release. If not, you must reboot the entire cluster by executing the step below to apply the new kernel otherwise, reboot is not required.

   
   $ podman exec -it nms-nms-nms /bin/bash
   
   $ SevOne-shutdown reboot

3. Verify the Operating System version. Confirm that version is Red Hat Enterprise Linux after the upgrade.

   $ cat /etc/redhat-release
   Red Hat Enterprise Linux release 8.9 (Ootpa)

4. Execute the following command to identify errors, if any.

   $ SevOne-act check checkout --full-cluster --verbose

Post upgrade, ensure that docker services running/enabled prior to the upgrade are still running/enabled.

• Using ssh, log in to SevOne NMS appliance as root.

  $ ssh root@<NMS appliance>

• Check docker service is running / enabled.

  Check for actual containers. For example, soa

  $ docker ps
  
  CONTAINER ID IMAGE COMMAND
  CREATED STATUS PORTS NAMES
  675c0f79c08f docker.s1artrtp1.s1.devit.ibm.com/soa:v6.8.0-build.2 "/entrypoint.sh" 31
  minutes ago Up 31 minutes soa

  Check whether docker service is active

  $ SevOne-peer-do "systemctl is-active docker.service"
  --- Gathering peer list...
  --- Running command on 127.0.0.1...
  active
  [ OKAY ]
  

  
  $ SevOne-peer-do "systemctl is-enabled docker.service"

  Check whether docker service is enabled

  $ SevOne-peer-do "systemctl is-enabled docker.service"
  --- Gathering peer list...
  --- Running command on 127.0.0.1...
  enabled
  [ OKAY ]

• Restart SOA.

  $ supervisorctl restart soa

Log File can be found in /var/SevOne/ansible-upgrade/<fromVersion-toVersion>/<timestamp>/<peerIP>.log.

For example, /var/SevOne/ansible-upgrade/v6.8.0-v7.0.1/<timestamp>/<peerIP>.log

• Each peer will have its own log file located on the Cluster Leader.
• A new log file will be created for each run of the upgrade and is split by the timestamp folder.

1. Open your VMware vSphere Client.
   Note: Your pages may vary from the screenshots in the steps below.
   
   
   
   
2. Search the virtual machine for the hardware component you want to upgrade. For example, vDNC100_RHEL.
3. Right-click on the virtual machine and select Edit Settings.
   

   Example

   
   
   
   

   where,

   • Hard disk 1 is your root partition (/) and is set to 50GB.
   • Hard disk 2 is your data partition (/data) and is set to 1024GB.

     You will see the current settings of your virtual machine.

1. SSH into your SevOne NMS virtual machine and log in as root.

   $ ssh root@<virtual machine IP address or hostname>

   Example

   $ ssh root@10.168.117.98

2. Execute the following commands in the order shown below.
   1. lsblk command reads the sysfs filesystem and udev db to gather information.

      $ lsblk
      
      NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
      fd0 2:0 1 4K 0 disk
      sda 8:0 0 50G 0 disk
      ├─sda1 8:1 0 2M 0 part
      ├─sda2 8:2 0 500M 0 part /boot
      └─sda3 8:3 0 49.5G 0 part
      ├─cl-root 253:0 0 45.5G 0 lvm /
      └─cl-swap 253:1 0 4G 0 lvm [SWAP]
      sdb 8:16 0 1T 0 disk
      └─sdb1 8:17 0 1024G 0 part
      └─data_vg-data_lv 253:2 0 1024G 0 lvm /data
      sr0 11:0 1 1024M 0 rom

      Important: You will see that sda3 is 49.5G.
   2. View the existing partitions and storage devices.

      $ parted -l | head -12
      
      Model: VMware Virtual disk (scsi)
      Disk /dev/sda: 53.7GB
      Sector size (logical/physical): 512B/512B
      Partition Table: msdos
      Disk Flags:
      
      Number Start End Size Type File system Flags
      1 1049kB 3146kB 2097kB primary
      2 3146kB 527MB 524MB primary xfs boot
      3 527MB 53.7GB 53.2GB primary lvm

      Important: Disk /dev/sda currently has 53.7GB.
   3. The pvs command provides physical volume information in a configurable form, displaying one line per physical volume.

      $ pvs
      
      PV VG Fmt Attr PSize PFree
      /dev/sda3 cl lvm2 a-- <49.51g 0
      /dev/sdb1 data_vg lvm2 a-- <1024.00g 0

   4. The pvdisplay allows you to see the attributes of one or more physical volumes like size, physical extent size, space used for the volume group descriptor area and so on. In the command below, physical volume is passed to obtain its. For example, /dev/sda3.

      $ pvdisplay /dev/sda3
      
      --- Physical volume ---
      PV Name /dev/sda3
      VG Name cl
      PV Size <49.51 GiB / not usable 0
      Allocatable yes (but full)
      PE Size 4.00 MiB
      Total PE 12674
      Free PE 0
      Allocated PE 12674
      PV UUID tJa0zz-UQEc-nKHh-mvoo-RfM6-NMM9-fyeaZC

      Important: Free PE is 0 for /dev/sda3.
   5. Based on target version's hardware requirements mentioned in SevOne NMS Installation Guide - Virtual Appliance, if you need to increase your hard disk based on the specifications, go to your VMware vSphere Client > choose your virtual machine for example, vDNC100_RHEL and its IP address. Right-click on the virtual machine and select Edit Settings. Increase your Hard disk 1 from 50GB to 150GB and click OK in the lower-right corner.
      
      
      
      Important: Hard disk 1 change from 50GB to 150GB will only take effect after the logical volume is extended by executing the lvextend command as shown below.
   6. The vgs command provides volume group information in a configurable form, displaying one line per volume group.

      $ vgs
      
      VG #PV #LV #SN Attr VSize VFree
      cl 1 2 0 wz--n- <49.51g 0
      data_vg 1 1 0 wz--n- <1024.00g 0

   7. The vgdisplay commanddisplays volume group properties (such as size, extents, number of physical volumes, etc.) in a fixed form. In the command below, VG Name, cl, is passed to obtain the attributes of this physical volume.

      $ vgdisplay cl
      
      --- Volume group ---
      VG Name cl
      System ID
      Format lvm2
      Metadata Areas 1
      Metadata Sequence No 3
      VG Access read/write
      VG Status resizable
      MAX LV 0
      Cur LV 2
      Open LV 2
      Max PV 0
      Cur PV 1
      Act PV 1
      VG Size <49.51 GiB
      PE Size 4.00 MiB
      Total PE 12674
      Alloc PE / Size 12674 / <49.51 GiB
      Free PE / Size 0 / 0
      VG UUID 7cMstg-INr3-loF0-vT2N-ghpb-IBJx-bzXuv7

      Important: VG Size is 49.51 GiB.
   8. Rescan to see that diskhas resized from 53687091200 (50GB) to 161061273600 (150GB).

      $ echo 1>/sys/class/block/sda/device/rescan; dmesg -T | tail -2
      
      [Thu Dec 8 16:44:33 2022] sd 0:0:0:0: [sda] 314572800 512-byte logical blocks: (161 GB
      /150 GiB)
      [Thu Dec 8 16:44:33 2022] sda: detected capacity change from 53687091200 to
      161061273600

   9. View the existing partitions and storage devices again.

      $ parted -l | head -12
      
      Model: VMware Virtual disk (scsi)
      Disk /dev/sda: 161GB
      Sector size (logical/physical): 512B/512B
      Partition Table: msdos
      Disk Flags:
      
      Number Start End Size Type File system Flags
      1 1049kB 3146kB 2097kB primary
      2 3146kB 527MB 524MB primary xfs boot
      3 527MB 53.7GB 53.2GB primary lvm

      Important: Disk, /dev/sda, has been resized to 161GB.
   10. Now, resize the disk partition from 50GB to 150GB.

       Example

       For this example, at the prompts below, you will enter the following in the order listed.

       • print - will show you the disk size.
       • print free - shows the free space available. In the example below, you will see the following.

         Number Start End Size Type File system Flags
         32.3kB 1049kB 1016kB Free Space
         1 1049kB 3146kB 2097kB primary
         2 3146kB 527MB 524MB primary xfs boot
         3 527MB 53.7GB 53.2GB primary lvm
         53.7GB 161GB 107GB Free Space

       • resizepart - resizes the partition.
         • at prompt Partition number?, enter 3 (this is obtainted from the Number column in the example above).
         • at prompt End? [53.7GB]?, enter 161GB (this is obtained from the End column in the example above).
       • print - verifies the disk size change to 161GB.

       • quit - allows you to quit parted utility.

       $ parted /dev/sda
       
       GNU Parted 3.1
       Using /dev/sda
       Welcome to GNU Parted! Type 'help' to view a list of commands.
       
       (parted) print
       Model: VMware Virtual disk (scsi)
       Disk /dev/sda: 161GB
       Sector size (logical/physical): 512B/512B
       Partition Table: msdos
       Disk Flags:
       
       Number Start End Size Type File system Flags
       1 1049kB 3146kB 2097kB primary
       2 3146kB 527MB 524MB primary xfs boot
       3 527MB 53.7GB 53.2GB primary lvm
       
       (parted) print free
       Model: VMware Virtual disk (scsi)
       Disk /dev/sda: 161GB
       Sector size (logical/physical): 512B/512B
       Partition Table: msdos
       Disk Flags:
       
       Number Start End Size Type File system Flags
       32.3kB 1049kB 1016kB Free Space
       1 1049kB 3146kB 2097kB primary
       2 3146kB 527MB 524MB primary xfs boot
       3 527MB 53.7GB 53.2GB primary lvm
       53.7GB 161GB 107GB Free Space
       
       (parted) resizepart
       Partition number? 3
       End? [53.7GB]? 161GB
       
       (parted) print
       Model: VMware Virtual disk (scsi)
       Disk /dev/sda: 161GB
       Sector size (logical/physical): 512B/512B
       Partition Table: msdos
       Disk Flags:
       
       Number Start End Size Type File system Flags
       1 1049kB 3146kB 2097kB primary
       2 3146kB 527MB 524MB primary xfs boot
       3 527MB 161GB 160GB primary lvm
       
       (parted) quit
       Information: You may need to update /etc/fstab.
       $

   11. Check to see if the disk is resized.

       $ parted -l | head -12
       
       Model: VMware Virtual disk (scsi)
       Disk /dev/sda: 161GB
       Sector size (logical/physical): 512B/512B
       Partition Table: msdos
       Disk Flags:
       
       Number Start End Size Type File system Flags
       1 1049kB 3146kB 2097kB primary
       2 3146kB 527MB 524MB primary xfs boot
       3 527MB 161GB 160GB primary lvm

       For partition 3, you will see that the End column now has 161GB.

   12. Now, you need to resize the physical volume in Logical Volume Manager. The pvs command provides physical volume information in a configurable form, displaying one line per physical volume.

       $ pvs
       
       PV VG Fmt Attr PSize PFree
       /dev/sda3 cl lvm2 a-- <49.51g 0
       /dev/sdb1 data_vg lvm2 a-- <1024.00g 0

   13. Display the physical volume is /dev/sda3, for example.

       $ pvdisplay /dev/sda3
       
       --- Physical volume ---
       PV Name /dev/sda3
       VG Name cl
       PV Size <49.51 GiB / not usable 0
       Allocatable yes (but full)
       PE Size 4.00 MiB
       Total PE 12674
       Free PE 0
       Allocated PE 12674
       PV UUID tJa0zz-UQEc-nKHh-mvoo-RfM6-NMM9-fyeaZC

       Important: Free PE is still 0 for /dev/sda3.
   14. Resize the Physical Volume.

       $ pvresize /dev/sda3
       
       Physical volume "/dev/sda3" changed
       1 physical volume(s) resized or updated / 0 physical volume(s) not resized

   15. After resizing the physical volume, display the physical volume is /dev/sda3, for example, again.

       $ pvdisplay /dev/sda3
       
       --- Physical volume ---
       PV Name /dev/sda3
       VG Name cl
       PV Size 149.45 GiB / not usable <1.57 MiB
       Allocatable yes
       PE Size 4.00 MiB
       Total PE 38259
       Free PE 25585
       Allocated PE 12674
       PV UUID tJa0zz-UQEc-nKHh-mvoo-RfM6-NMM9-fyeaZC

       Free PE is now 25585 for /dev/sda3.

   16. You are now ready to extend the logical volume and filesystem using lvextend command with all, 100%, of available free space.
   17. Display information related to file systems about total and available space for root partition.

       $ df -hT /
       
       Filesystem Type Size Used Avail Use% Mounted on
       /dev/mapper/cl-root xfs 46G 17G 29G 37% /

       Size is 46G for 50GB hard disk.

   18. lvs command provides logical volume information in a configurable form, displaying one line per logical volume.

       $ lvs
       LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
       Convert
       root cl -wi-ao---- <45.51g
       swap cl -wi-ao---- 4.00g
       data_lv data_vg -wi-ao---- <1024.00g

   19. lvdisplay commanddisplays the properties of LVM logical volumes. For example, logical volume, cl, in the example below.

       $ lvdisplay cl
       --- Logical volume ---
       LV Path /dev/cl/swap
       LV Name swap
       VG Name cl
       LV UUID jj8N64-6UPp-WWQz-UKGj-TAxf-MwXW-HJCu7n
       LV Write Access read/write
       LV Creation host, time localhost, 2017-12-07 22:10:11 +0000
       LV Status available
       # open 2
       LV Size 4.00 GiB
       Current LE 1024
       Segments 1
       Allocation inherit
       Read ahead sectors auto
       - currently set to 8192
       Block device 253:1
       --- Logical volume ---
       LV Path /dev/cl/root
       LV Name root
       VG Name cl
       LV UUID Jqecl2-YErD-8zwd-JHOG-MPZB-lmdI-HdcT8u
       LV Write Access read/write
       LV Creation host, time localhost, 2017-12-07 22:10:11 +0000
       LV Status available
       # open 1
       LV Size <45.51 GiB
       Current LE 11650
       Segments 1
       Allocation inherit
       Read ahead sectors auto
       - currently set to 8192
       Block device 253:0

   20. You are now ready to extend the logical volume using lvextend command.

       $ lvextend -l+100%FREE -r /dev/cl/root
       
       Size of logical volume cl/root changed from <45.51 GiB (11650 extents) to <145.45 GiB
       (37235 extents).
       Logical volume cl/root successfully resized.
       meta-data=/dev/mapper/cl-root isize=512 agcount=4, agsize=2982400 blks
       = sectsz=512 attr=2, projid32bit=1
       = crc=1 finobt=0 spinodes=0
       data = bsize=4096 blocks=11929600, imaxpct=25
       = sunit=0 swidth=0 blks
       naming =version 2 bsize=4096 ascii-ci=0 ftype=1
       log =internal bsize=4096 blocks=5825, version=2
       = sectsz=512 sunit=0 blks, lazy-count=1
       realtime =none extsz=4096 blocks=0, rtextents=0
       data blocks changed from 11929600 to 38128640

   21. Check the root partition size for /dev/cl/root.

       $ lvdisplay /dev/cl/root
       
       --- Logical volume ---
       LV Path /dev/cl/root
       LV Name root
       VG Name cl
       LV UUID Jqecl2-YErD-8zwd-JHOG-MPZB-lmdI-HdcT8u
       LV Write Access read/write
       LV Creation host, time localhost, 2017-12-07 22:10:11 +0000
       LV Status available
       # open 1
       LV Size <145.45 GiB
       Current LE 37235
       Segments 1
       Allocation inherit
       Read ahead sectors auto
       - currently set to 8192
       Block device 253:0

   22. Display information related to file systems about total and available space for root partition.

       $ df -hT /
       
       Filesystem Type Size Used Avail Use% Mounted on
       /dev/mapper/cl-root xfs 146G 17G 129G 12% /

       You will see that the root partition has been updated and resized. It is now 146G.

   23. Exit from Command Line Interface.

       $ exit

   24. Your virtual machine resource requirements now align with what is required by SevOne NMS' target version. You may now proceed with the steps to perform the upgrade.