Running smarter concurrent fix pack updates for Db2 pureScale instances

You can update multiple hosts in a Db2 pureScale instance while the remaining members and cluster caching facilities continue to process transactions.

Before you begin

Before attempting to run the concurrent fix pack update on your Db2® pureScale® cluster, ensure that the following conditions are met:

All prerequisites are met. For more information, see Preparing to install a fix pack.
You have root user authority and instance owner authority.

Online fix pack updates are supported between the Db2 version that is installed on your Db2 pureScale instance and the Db2 version of the fix pack or special build that you want to install. Run the installFixPack command with the -show_level_info option to determine whether the update is supported. Run the command from the location of the new fix pack or special build image.
The following example shows sample output of running

installFixPack
-show_level_info

    Code level         =  Version:11 Release:5 Modification:8 Fixpack:0 
    Architecture level =  Version:11 Release:5 Modification:8 Fixpack:0 
    Section level      =  Version:11 Release:5 Modification:8 Fixpack:0 
     
    Supports online update = Yes 
     
    Minimum committed code level required for online install = 
       Version:11 Release:5 Modification:6 Fixpack:0 
     
    The execution completed successfully. 
     
    For more information see the DB2 installation log at "/tmp/installFixPack.log.13369572". 
    DBI1070I  Program installFixPack completed successfully.

About this task

Options are available to the installFixPack command that can update up to half of the hosts in a Db2 pureScale cluster concurrently, with no impact on running transactions:

-concurrency: Automatically updates all hosts in a cluster with a specified degree of concurrency.
-autoupdate: Automatically updates all hosts in a cluster with an optimum degree of concurrency.

With either option, you run the command from any host in your cluster and the Db2 installer automatically updates all hosts.

Restriction: The following restrictions apply when you run the command with either option:

All hosts must be updated before you can commit changes and update the Db2 pureScale instance.
The value of the -concurrency parameter cannot be more than half of the members.
The secondary CF must be in peer state after it is updated. This state change happens automatically if any connection is made to a database after the secondary CF is updated.

Procedure

Extract the fix pack or special build image to a directory that is accessible to all hosts.

Note: Do not use a General Parallel File System, like IBM GPFS, to store the image. A GPFS might go into maintenance mode during the update and make the image inaccessible. Unexpected failures can result that can be difficult from which to recover. Instead, set up a Network File System (NFS) server to share the image between the hosts, or create a directory, such as /db2_images/db2_update001, on each host and extract the image to these directories, separately.
Run the following command to determine whether all members are online, and that the secondary CF is in PEER state:
```
db2instance –list
```

If the output shows that the secondary CF is in CATCHUP state, you can check the catch-up progress percent by querying the DB2_CF administrative view:

db2 "SELECT ID as CF_ID, varchar(CURRENT_HOST,21) AS HOST, varchar(STATE,14) AS CF_STATE FROM SYSIBMADM.DB2_CF"

The following example output shows that cfserver56 is 79% complete to being in a peer state with the primary:

CF_ID  HOST             CF_STATE 
------ ---------------  -------------- 
   128 cfserver56       CATCHUP(79%) 
   129 cfserver54       PRIMARY 

  2 record(s) selected.

Run the installFixPack command with the –show_update_order option to print the order that the installer uses to update the target hosts. While the command does not run any update operation on these hosts, it does run some preliminary checks to ensure that the cluster is ready to be updated.
The following example shows the syntax for running the command with –show_update_order option and a concurrency setting of 2:
```
<media-dir>/installFixPack –p <FP-install-path> -I <instance-name> -concurrency 2 –show_update_order
```
where
- <media-dir> is the location of the fix pack package.
- <FP-install-path> is the path to the target member or CF to be updated.
- <instance-name> is the name of your Db2 pureScale instance.
The following example shows the syntax for the command with –show_update_order option and the autoupdate option included:
```
<media-dir>/installFixPack –p <FP-install-path> -I <instance-name> -autoupdate –show_update_order 
```
Run the installFixPack command again, but without the –show_update_order, to update all the hosts in your Db2 pureScale cluster.
The following example shows the syntax for running the installFixPack command with a concurrency setting of 2:
```
:<media-dir>/installFixPack –p <FP-install-path> -I <instance-name> -concurrency 2 
```
The following example shows the syntax for the command with the autoupdate option included:
```
Example with -autoupdate: <media-dir>/installFixPack –p <FP-install-path> -I <instance-name> -autoupdate 
```
Note: The value of FP-install-path must be the same on all hosts, and the path must be different from the path to the currently installed Db2 instance.
If a tiebreaker host exists for the cluster, complete the following steps:
1. Enter cm maintenance mode from the old code level:
```
export DB2INSTANE=<instance_name>; <OLD-FP-install-path>/bin/db2cluster -cm -enter -maintenance
```
2. Enter cfs maintenance mode from the old code level:
```
export DB2INSTANCE=<instance_name>; <OLD-FP-install-path>/bin/db2cluster -cfs -enter -maintenance 
```
3. Update the tiebreaker host by running the installFixPack command from the target fix pack image:
```
<media-dir>/installFixPack -b <OLD-FP-install-path> -p <FP-install-path> -L
```
  Note: The value of FP-install-path must be the same on all hosts:
```
<media-dir>/installFixPack -b /opt/ibm/db2/V11.5 -p /opt/ibm/db2/V11.5M5FP0 -L 
```
4. Exit cm maintenance mode from the new code level:
```
export DB2INSTANCE=<instance_name>; <FP-install-path>/bin/db2cluster -cm -exit -maintenance 
```
5. Exit cfs maintenance mode from the new code level:
```
export DB2INSTANCE=<instance_name>; <FP-install-path>/bin/db2cluster -cfs -exit -maintenance 
```
6. Update the IBM Global Security Kit (GSKit) path to the newer fix pack location:
```
/var/db2/db2ssh/db2locssh set_gskit_path <FP-install-path>/lib64/gskit/Copy code
```
7. Verify that db2locssh is functioning properly. For example, if your system's hostname is "hostT", run the hostname command via db2locssh:
```
 root@hostT>/var/db2/db2ssh/db2locssh root@hostT hostname hostT
```
If all the steps are successful, then you successfully applied the fix pack to the tie-breaker host.
Note: You can run your workloads for the new fix pack before committing to it, in order to validate its functionality. If this validation is not satisfactory, you can cancel the new fix pack update, and move back to the previous fix pack. For more information, see Canceling online fix pack updates.
Commit the online fix pack update so that your Db2 pureScale instance is updated to the new fix pack level:
```
<media-dir>/installFixPack -commit_level -I <instance-name> -l <log-file-name> -t <trace-file-name>
```
As an instance user, verify that your instance and databases show the new committed fix pack level:
```
db2pd -ruStatus
```
If you want to use capabilities specific to the fix pack, update the system catalog objects in your databases:
1. Log on as the instance owner.
2. For each database in the instance, issue the db2updv115 command:
```
db2updv115 -d db-name
```

Example

Example 1: Updating with the -autoupdate option for maximum concurrency.

Run the installFixPack command with the -autoupdate option, but also include the -show_update_order option. The following example shows how the installer plans to update the hosts:

# installFixPack -p /opt/ibm/db2/V11.5FP -I db2inst1 -autoupdate -show_update_order -y
DBI1961I  The installFixPack command was run with -show_update_order. This option is informational and no update operation will occur.

The installFixPack command will update the hosts with concurrency "3".

The installFixPack command will update the hosts in the following order:
host1.ibm.com,host2.ibm.com,host3.ibm.com,
host4.ibm.com,
host5.ibm.com,
host6.ibm.com

When the -show_update_order command runs successfully, run the real update. The following example shows the command syntax and output from running the installFixPack command with the -autoupdate option:

# installFixPack -p /opt/ibm/db2/V11.5FP -I db2inst1 -autoupdate -y
Performing online fixpack update on host: "host1.ibm.com".
 
Performing online fixpack update on host: "host2.ibm.com".
 
Performing online fixpack update on host: "host3.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host1.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host1.ibm.com" for details.
 
Performing online fixpack update on host: "host4.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host2.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host2.ibm.com" for details.
 
Performing online fixpack update on host: "host5.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host3.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host3.ibm.com" for details.
 
Performing online fixpack update on host: "host6.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host4.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host4.ibm.com" for details.
 
DBI1964I  The online fixpack update operation succeeded on "host5.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host5.ibm.com" for details.
 
DBI1964I  The online fixpack update operation succeeded on "host6.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host6.ibm.com" for details.

Example 2: Keeping more hosts online throughout the update by using the -concurrency <# of hosts> option.

Run the installFixPack command with the -concurrency value set to 2, but also include the -show_update_order option. The following example shows how the installer plans to update the hosts:

# installFixPack -p /opt/ibm/db2/V11.5FP -I db2inst1 -concurrency 2 -show_update_order -y
DBI1961I  The installFixPack command was run with -show_update_order. This option is informational and no update operation will occur.

The installFixPack command will update the hosts with concurrency "2".

The installFixPack command will update the hosts in the following order:
host1.ibm.com,host2.ibm.com,
host3.ibm.com,
host4.ibm.com,
host5.ibm.com,
host6.ibm.com

If the -show_update_order command runs successfully, run the real update. The following example shows the command syntax and output from running the installFixPack command with the -concurrency option set to 2:

# installFixPack -p /opt/ibm/db2/V11.5FP -I db2inst1 -concurrency 2 -y
Performing online fixpack update on host: "host1.ibm.com".
 
Performing online fixpack update on host: "host2.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host1.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host1.ibm.com" for details.
 
Performing online fixpack update on host: "host3.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host2.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host2.ibm.com" for details.
 
Performing online fixpack update on host: "host4.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host3.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host3.ibm.com" for details.
 
Performing online fixpack update on host: "host5.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host4.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host4.ibm.com" for details.

Performing online fixpack update on host: "host6.ibm.com".
 
DBI1964I  The online fixpack update operation succeeded on "host5.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host5.ibm.com" for details.
 
DBI1964I  The online fixpack update operation succeeded on "host6.ibm.com". Refer to /tmp/installFixPack_Concurrent.log.841135 on host "host6.ibm.com" for details.

What to do next

You can monitor the update on an individual host by opening a separate session on that host. Once logged in, check the contents of /tmp/installFixPack_Concurrent.log.<pid> to monitor the status of the update.

If there is a failure during the concurrent fix pack update, refer to the logs for details. When the issue is resolved, you can update the rest of the cluster manually or follow the instructions from step 2 onward. The installer keeps track of the hosts have been updated, so updates are applied only to those hosts that need them.