In the system console Events pane, when you select a time interval value of Last 24 Hours or Last Hour there are no events displayed.

Workaround:

-----------

1. Select the time interval value of All to display all events.
2. If the events are not sorted in descending order, click the Updated on field in the table to sort the events in descending order. The most recent events are displayed first in the table.

Fixed:

-----------

Addressed in V1.0.0.2-V1.0.0.5 fixpacks and All  V1.1  Systems

V1.0.0.6/V1.1.0.2 Fixpacks remove the appliance console from the appliance.

KI003332

I_V1.0.0.0

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

I_V1.0.0.4

I_V1.0.0.5

I_V1.1.0.0

I_V1.1.0.1

The system console GUI fails to load the login page or provide any corrective instructions when viewed through the latest version of the Mozilla Firefox browser (20.0) released on April 2, 2013.

Install and enable the Firefox ESR release from the following page: http://www.mozilla.org/en-US/firefox/organizations/all.html

Alternatively, use another supported browser such as Internet Explorer V8 or V9.  

KI002637

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

I_V1.0.0.4

I_V1.0.0.5

I_V1.1.0.1

A resume option has changed for the miupdate command
 

The -e alternative to the --resume option for the miupdate command has been changed to -resume.

Do not use the -e option to resume the update process. Use the -resume option instead. For example, to resume the update process after the system console suspends the update to reboot one or more nodes in the system, the following miupdate command options can be used:

miupdate --resume | -resume [management | prepare | apply | commit]

KI002702

During fix pack registration, there is at least one command to SSH to localhost (127.0.0.1) to perform some tasks. However, SSH does not recognize the localhost host key under the IP address 127.0.0.1, and waits for you to enter Yes to accept the host key.

Before you install the fix pack, enter the following command as root on the management host to accept the host key:

When prompted to accept the new host key, enter Yes.

KI002696

During the management phase of a fix pack installation, if you attempt to log in to the system console GUI or to use it for tasks other than installing the fix pack, the system console GUI becomes inaccessible at the URL https://management_host_name, where management_host_name represents the host name or IP address of the management host.

The system console GUI might also become inaccessible after the management phase completes.

Restart the system console GUI:
1. Log in to the management host as the root user.
2. Run the following command:

KI003581

During the apply phase of a fix pack installation the progress bar incorrectly indicates the stage is 100% complete
 

During the apply phase of a fix pack installation, the progress bar in the system console GUI might indicate that the stage is 100% complete but the Apply Fix Pack window indicates that stage 4 (Apply to non-management hosts) is in a Running state.

1. Ignore the percentage complete value displayed in the progress bar.
2. Verify that the apply stage has completed successfully. The Apply Fix Pack window displays a Completed state for stage 4 (Apply to non-management hosts) when the apply stage has completed successfully.

KI007645

After the commit phase of a fix pack installation completes successfully, the progress bar in the system console GUI might indicate that the stage is less than 100% complete but the Apply Fix Pack window indicates that stage 5 (Commit) is in a Completed state.

1. Ignore the percentage complete value displayed in the progress bar.
2. Verify that the commit stage has completed successfully. The Apply Fix Pack window displays a Completed state for stage 5 (Commit) to indicate the commit stage has completed successfully.

KI004047

Paging space on the management node is set to the incorrect value of 48 GB. The paging space needs to be set to a value of 128 GB.

Use the mgmt_ps script to set the value of the paging space to 128 GB on the management host.

Note: You do not need to change the value of the paging space on the standby management host. The paging space is set to the correct value of 128 GB on the standby management host

1. Obtain the mgmt_ps.zip file from IBM Support.

2. Log in to the management host as the root user.

3. Copy the mgmt_ps.zip file to a temporary directory on the management host.

4. Navigate to the temporary directory and extract the mgmt_ps script.

5. Grant execute permission on the script:

KI003962

The fix pack installation hangs when only a directory is specified as the location of the fix pack file
 

When specifying the location of the fix pack file on the management host, if you specify only a directory and do not include the fix pack file name, the fix pack installation hangs. No warning message or error message is displayed.

Refresh the system console and specify the full path and the file name of the fix pack file in the Add Fix Pack window.

KI004076

The miinfo -d -c command incorrectly indicates that the versions of some components are higher than expected
 

When you run the miinfo -d -c command, the command incorrectly indicates that the versions of the following software components are higher than expected:

Ignore the warning about the higher than expected versions of these software components. The version returned by the command is correct and is the expected version.

• miinfo was part of the appliance console which has been removed. appl_ls_sw can be used instead for this purpose.
• WebSphere Application Server was part of Warehouse Tools which is removed as part of these fixpacks.
• InfoSphere Warehouse components are part of Warehouse Tools which is removed as part of these fixpacks.
• Updates to the catalog for later fixpacks should correctly show the levels for Optim Performance Manager and OQWT.

KI004088

I_V1.0.0.0

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

I_V1.0.0.4

I_V1.1.0.0

If you do not stop the system console before you use the miauth command to change the password for the restuser user, as documented in the password change procedure Changing the passwords for system console component users, the system console cannot access Systems Director.

1. Log in to the management host as root.

2. Stop the system console:

IMPORTANT: For future password changes for the restuser user, use the password change procedure that is documented in Changing the passwords for system console component users.

• IBM Systems Director is disabled and no longer used as part of the Appliance Console. The smcli command is no longer used and the restuser user is no longer managed.

• The appliance console is removed. mistop and mistart are no longer valid commands.

KI004070

I_V1.0.0.2

The system console becomes unresponsive after the fix pack file is uploaded
 

After you upload the fix pack file to the system, after approximately one hour the Add Fix Pack window in the system console remains unresponsive and greyed out.

.Workaround:

Verify that the system console is unresponsive by attempting to open the Welcome page in the system console.

Wait approximately 10 minutes for the system console modules to restart, and then continue with the fix pack installation.

KI004082

I_V1.0.0.2

The miupdate -resume current command hangs when resuming the fix pack installation after the M_Failed_Apply_Nonimpact substage.

.Workaround:

1. Log in as root on the management host and issue the following command:

where <pid> represents the process ID of the miupdate process or the process ID of the MIUpdateCLI process.

3. Click the Resume button in the system console.

KI004991

The fix pack installation fails in the Apply to management hosts stage. After you address the issue and click Resume current phase in the system console, the system console does not respond and the fix pack installation does not continue.

.Workaround:

1. Log in to the management host as the root user.

2. Verify that the fix pack installation is in the M_Applied_Nonimpact substage:

KI004991

The fix pack installation fails in the Apply to management hosts stage. After you address the issue and click Resume current phase in the system console, the system console does not respond and the fix pack installation does not continue .

.Workaround:

1. Log in to the management host as the root user.

2. Verify that the fix pack installation is in the M_Failed_Apply_Nonimpact substage:
appl_ls_cat

3. Resume the fix pack installation:
miupdate -u management

KI004080

If the Apply to non-management hosts stage of the fix pack installation fails, the fix pack installation stops but the system console incorrectly displays the status of the stage as Waiting in the Apply Fix Pack window. The status of the stage is Failed to apply without impact to non-management hosts and is displayed correctly in the Fix Pack Detail panel. The fix pack installation cannot be resumed by clicking the Start current stage button.

.Workaround:

1. Log in to the management host as root.

2. Review the fix pack installation log /BCU_share/aixapply/pflayer/pl_update.log and correct the problem identified in the log.

3. Click the Resume button in the system console to resume the fix pack installation.

KI004080

In the Apply to non-management hosts stage of the fix pack installation, when you click the Resume button in the system console to resume the installation from the status Failed to apply without impact to non-management hosts, the fix pack installation stops before the end of the Apply to non-management hosts stage. The status of the fix pack installation in the Fix Pack Details panel is Applied without impact to non-management hosts.

.Workaround:

1. Log in to the management host as root.

2. Verify that the fix pack installation is in the Applied non impact to non management hosts status:

KI003892

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

I_V1.0.0.4

I_V1.0.0.5

I_V1.1.0.1

After the fix pack installation completes you might not be able to log into the system console GUI because the profiles files on the management host were not correctly updated.

.Workaround:

During a fix pack installation, profiles are automatically backed up on the management host.

You can restore the profiles using the following steps:
1. Log in to the management host as the root user.
2. Identify the most recent profile backups in the /.profile, /etc/profile, and /etc/security/profile directories. The file names include the date the back up was created.
3. Issue the the following commands:

KI004754

Fix pack installation stops with user validation failure at preview stage
 

If the bcuaix core warehouse instance owner password was previously changed by using a method other than the miauth command method, a user validation failure occurs when you attempt to run the preview stage of the fix pack installation procedure. This failure stops the preview stage from running.

Workaround:

Change the password for the bcuaix core warehouse instance owner by using the following supported miauth command method:

1. Log in to the management host as the root user and issue the following command:

2. Run the preview stage of the fix pack installation procedure.

KI004971

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

If the preview stage fails, the system console Fix Pack panel shows an error message. After fixing the environment and resuming the preview stage of the fix pack installation process, the status of the preview stage in the system console is shown as Waiting instead of Running.

Workaround:

If the preview stage of the installation process is in the Waiting state, start the preview stage again by clicking the Resume button in the system console.

KI004907

The fix pack installation fails during the preview stage because the HMC cannot connect to an endpoint. The following error message is displayed:

ENV_VALIDATION::PREVIEW::FAILED::CEC connectivity validation with HMC failed

Workaround:

Identify and replace the invalid IP addresses stored in the HMC.

1. Log in to the HMC as the hscroot user and run the following command:

where invalid_IP_address is the invalid IP address you identified in step 2.

KI005059

Fixpack

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

During the management stage firmware upgrade of the HMC, the HMC fails to restart and results in a failure to upgrade the HMC firmware.

A symptom of the issue can be seen in the /BCU_share/aixappl/pflayer/log/pl_update.log file, which shows the following example log entries:

[22 May 2014 07:56:03,249] <28311760 UPDT PREP TRACE host01> For message id::631
[22 May 2014 07:56:03,251] <28311760 UPDT PREP ERROR host01> Error on nodes (172.23.1.245).
[22 May 2014 07:56:03,268] <28311760 UPDT PREP INFO host01> STEP_END::3::HMC_UPD::FAILED

When the HMC fails to restart, the HMC local console displays the following error message after the HMC is manually restarted:

Critical Error 1901
A critical error has prevented normal HMC startup. Please reboot the HMC and try again. If the problem persists, contact your support personnel. 1901: HMC Startup aborted due to a malfunction of a required module.

Workaround:

After encountering the HMC restart failure, complete the following steps to resume the firmware upgrade process:

1. Verify that the HMC is offline by running the ping command. If the HMC is confirmed to be offline, manually start the HMC.

2. After the HMC comes up, wait 10 minutes to be certain that all of the HMC services are started.

3. Resume the management stage update process by running the following command on the management host as root:

KI005062

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

The V7000 storage drive firmware upgrade process fails due to a failure of the utilitydriveupgrade command of the drive upgrade utility to be successfully installed, even after the successful installation of the installation package. Investigate the pl_update.trace log to determine if the utilitydriveupgrade command has failed.

Sample information from the PL log in /BCU_share/aixappl/pflayer/log/pl_update.trace:

[20 May 2014 03:06:54,797] <40566876 CTRL  TRACE  stgkf301> command: ssh admin@172.23.3.201 LANG=en_US svcservicetask applysoftware -file  IBM_INSTALL_driveUpgrade_130610
[20 May 2014 03:06:54,798] <40566876 CTRL  TRACE  stgkf301>  CMMVC6227I The package installed successfully.
[20 May 2014 03:06:54,798] <40566876 CTRL  TRACE  stgkf301> Rc = 1
[20 May 2014 03:06:54,799] <40566876 CTRL  DEBUG  stgkf301> Error String: CMMVC6227I The package installed successfully.
[20 May 2014 03:06:54,799] <40566876 CTRL  DEBUG  stgkf301> command succeeded - drive upgrade utility installed successfully
[20 May 2014 03:06:54,800] <40566876 CTRL  DEBUG  stgkf301> verifying installation of utilitydriveupgrade command
[20 May 2014 03:06:55,014] <40566876 CTRL  TRACE  stgkf301> command: ssh admin@172.23.3.201 utilitydriveupgrade
[20 May 2014 03:06:55,015] <40566876 CTRL  TRACE  stgkf301>  rbash: utilitydriveupgrade: command not found
[20 May 2014 03:06:55,015] <40566876 CTRL  TRACE  stgkf301> Rc = 127
[20 May 2014 03:06:55,016] <40566876 CTRL  DEBUG  stgkf301> utilitydriveupgrade did not get installed properly
[20 May 2014 03:06:55,016] <40566876 CTRL  TRACE  stgkf301> workaround: driveupgradeutility is having some issues, applying workaround by installing softwareupgrade test utility and retrying
[20 May 2014 03:06:55,017] <40566876 CTRL  DEBUG  stgkf301> apply: retrying install of drive upgrade utility
[20 May 2014 03:06:55,038] <40566876 CTRL  DEBUG  stgkf301> apply: Uploading update file /BCU_share/bwr2/firmware/storage/2076/image/imports/testupdate/IBM2076_INSTALL_upgradetest_11.15
[20 May 2014 03:06:55,235] <40566876 CTRL  TRACE  stgkf301> command: LANG=en_US scp /BCU_share/bwr2/firmware/storage/2076/image/imports/testupdate/IBM2076_INSTALL_upgradetest_11.15 admin@172.23.3.201:/home/admin/upgrade
[20 May 2014 03:06:55,236] <40566876 CTRL  TRACE  stgkf301> Rc = 0
[20 May 2014 03:06:55,237] <40566876 CTRL  DEBUG  stgkf301> apply: uploaded test update file IBM2076_INSTALL_upgradetest_11.15 to storwize
[20 May 2014 03:06:55,638] <40566876 CTRL  TRACE  stgkf301> command: ssh admin@172.23.3.201 LANG=en_US svctask applysoftware -file  IBM2076_INSTALL_upgradetest_11.15
[20 May 2014 03:06:55,638] <40566876 CTRL  TRACE  stgkf301>  CMMVC6227I The package installed successfully.
[20 May 2014 03:06:55,639] <40566876 CTRL  TRACE  stgkf301> Rc = 1
[20 May 2014 03:06:55,639] <40566876 CTRL  DEBUG  stgkf301> Error String: CMMVC6227I The package installed successfully.
[20 May 2014 03:06:55,640] <40566876 CTRL  DEBUG  stgkf301> apply: upgrade test utility installed successfully
[20 May 2014 03:06:55,641] <40566876 CTRL  DEBUG  stgkf301> Successfully applied workaround, retrying utilitydriveupgrade installation
[20 May 2014 03:06:55,702] <40566876 CTRL  TRACE  stgkf301> { Entering Ctrl::Updates::Storwize::get_imagefile_name (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storwize.pm line 1104)
[20 May 2014 03:06:55,703] <40566876 CTRL  TRACE  stgkf301> Args:["/BCU_share/bwr2/firmware/storage/2076/image/imports","driveutility"]
[20 May 2014 03:06:55,755] <40566876 CTRL  TRACE  stgkf301> command: ls /BCU_share/bwr2/firmware/storage/2076/image/imports/driveutility
[20 May 2014 03:06:55,755] <40566876 CTRL  TRACE  stgkf301>  IBM_INSTALL_driveUpgrade_130610
[20 May 2014 03:06:55,756] <40566876 CTRL  TRACE  stgkf301> Rc = 0
[20 May 2014 03:06:55,756] <40566876 CTRL  DEBUG  stgkf301> prepare: type driveutility image file: IBM_INSTALL_driveUpgrade_130610
[20 May 2014 03:06:55,757] <40566876 CTRL  TRACE  stgkf301> Return: 0
[20 May 2014 03:06:55,757] <40566876 CTRL  TRACE  stgkf301> Exiting Ctrl::Updates::Storwize::get_imagefile_name }
[20 May 2014 03:06:55,758] <40566876 CTRL  DEBUG  stgkf301> prepare: 172.23.3.201: installing drive upgrade utility
[20 May 2014 03:06:56,060] <40566876 CTRL  TRACE  stgkf301> command: ssh admin@172.23.3.201 LANG=en_US svcservicetask applysoftware -file  IBM_INSTALL_driveUpgrade_130610
[20 May 2014 03:06:56,061] <40566876 CTRL  TRACE  stgkf301>  CMMVC5993E The specified upgrade package does not exist.
[20 May 2014 03:06:56,061] <40566876 CTRL  TRACE  stgkf301> Rc = 1
[20 May 2014 03:06:56,062] <40566876 CTRL  DEBUG  stgkf301> Error String: CMMVC5993E The specified upgrade package does not exist.
[20 May 2014 03:06:56,063] <40566876 CTRL  DEBUG  stgkf301> drive upgrade utility failed to install:172.23.3.201
[20 May 2014 03:06:56,063] <40566876 CTRL  DEBUG  stgkf301> utilitydriveupgrade installation failed even after re-installing software test utility
[20 May 2014 03:06:56,064] <40566876 CTRL  DEBUG  stgkf301> Unable to fix problems in utilitydriveupgrade, drive update failed

Workaround:

Resume the firmware upgrade procedure by running the following command from the management host as root:

Note: If the problem persists, contact IBM Support with the following logs at hand:

/BCU_share/aixappl/pflayer/log -> All of the files within this directory

/log/pfmgt.trace

ssh admin@<storwize_ip> svc_snap

KI005064

I_V1.0.0.1

I_V1.0.0.2

I_V1.0.0.3

V7000 storage drive goes offline after firmware upgrade succeeds
 

The V7000 storage drive might go offline after a successful firmware upgrade. Investigate the pl_update.trace log to determine if a drive has gone offline.

Sample information from the PL log in /BCU_share/aixappl/pflayer/log/pl_update.trace:

[14 May 2014 07:34:36,828] <30867468 CTRL  TRACE  stgkf201> command: ssh admin@172.23.2.206 LANG=en_US utilitydriveupgrade -drivemodel ALL -filename  IBM2076_DRIVE_20130314
[14 May 2014 07:34:36,829] <30867468 CTRL  TRACE  stgkf201>  Upgrading drive id 47, ( 1 / 48 )
[14 May 2014 07:34:36,829] <30867468 CTRL  TRACE  stgkf201>  Upgrading drive id 23, ( 2 / 48 )
[14 May 2014 07:34:36,829] <30867468 CTRL  TRACE  stgkf201>  Upgrading drive id 35, ( 3 / 48 )
[14 May 2014 07:34:36,830] <30867468 CTRL  TRACE  stgkf201>  Upgrading drive id 11, ( 4 / 48 )
[14 May 2014 07:34:36,830] <30867468 CTRL  TRACE  stgkf201>  Upgrading drive id 4, ( 5 / 48 )
[14 May 2014 07:34:36,830] <30867468 CTRL  TRACE  stgkf201>  ERROR: Drive 4 is no longer online after being upgraded.
[14 May 2014 07:34:36,831] <30867468 CTRL  TRACE  stgkf201>     The current drive status is offline
[14 May 2014 07:34:36,831] <30867468 CTRL  TRACE  stgkf201> Rc = 1
[14 May 2014 07:34:36,832] <30867468 CTRL  DEBUG  stgkf201> Extracted msg from NLS: apply: 172.23.2.206 ssh admin@172.23.2.206 LANG=en_US utilitydriveupgrade -drivemodel ALL -filename  IBM2076_DRIVE_20130314 command failed.

Workaround:

1. Bring the offline drives back online by running the following command on the management host as root:

Note: If the problem persists, contact IBM Support with the following logs at hand:

/BCU_share/aixappl/pflayer/log -> All of the files within this directory

/log/pfmgt.trace

ssh admin@<storwize_ip> svc_snap

KI004969

The 1 Gbps network switches might lose their connections to the corporate network because ports are configured with bpdu-guard, usually on ports 43-48. After the firmware upgrade and reboot as part of the Fix Pack 1.0.0.3 installation, the network switch ports might go into errdisable mode and be disabled if bridge protocol data units (BPDU) frames are detected.

Workaround:

The following steps should only be performed on V1.0 Appliances and not on V1.1 Appliances.

To prevent this issue from occurring, disable bpdu-guard on the uplink ports of both of the 1 Gbps network switches.

1. Log in to the 1 Gbps network switches as the admin user.

2. Verify that the 1 Gbps network switches have bpdu-guard enabled on its ports.

KI005272

I_V1.0.0.3

I_V1.0.0.4
I_V1.0.0.5

These issues might occur as a result of the fix pack installation. The paging00 device is missing when a check is done by running the lsps -a command on a host. The result of this missing paging device is that the paging space is cut in half from the standard 128 GB.

An indication that your system is experiencing the missing paging device issue is shown in the output that results from running the following command on any host as root:

$ dsh -n $ALL "lsps -a" | dshbak -c

In the following example output, the paging00 device is missing on all of the hosts except the management host (host01):

HOSTS -------------------------------------------------------------------------
host01
-------------------------------------------------------------------------------
Page Space      Physical Volume   Volume Group    Size %Used Active  Auto  Type Chksum
paging00        hdisk0            rootvg       65536MB     3   yes   yes    lv     0
hd6             hdisk0            rootvg       65536MB     3   yes   yes    lv     0

HOSTS -------------------------------------------------------------------------
host02, host03, host04, host05, host06
-------------------------------------------------------------------------------
Page Space      Physical Volume   Volume Group    Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg       64512MB     1   yes   yes    lv     0

A related issue is that the paging space on the management host might not be set to auto.

-----------

KI005121

Ethernet network switch firmware upgrade fails during apply stage of fix pack installation
 

During the apply stage of the fix pack installation process to upgrade the firmware of the Ethernet network switches, a comparison with the expected firmware image version might fail. If this happens, the firmware for the switch images has not been successfully uploaded.

You can confirm that this issue has occurred by checking the /var/aixappl/pflayer/log/pl_update.trace log. Run the following command to examine the log file:

tail -f /var/aixappl/pflayer/log/pl_update.trace

Look for the corresponding messages (highlighted in bold here) in the following example log output:

[05 Jun 2014 19:48:16,849] <26542124 UPDT APPI DEBUG  host01mgmt> Command Status:BNT:net3:172.23.1.251:1:Compare of firmware failed for switch after copy.
[05 Jun 2014 19:48:16,850] <26542124 UPDT APPI DEBUG  host01mgmt>  BNT:net1:172.23.1.253:0:Compare of firmware success for switch after copy.
[05 Jun 2014 19:48:16,850] <26542124 UPDT APPI DEBUG  host01mgmt>  BNT:net0:172.23.1.254:0:Compare of firmware success for switch after copy.
[05 Jun 2014 19:48:16,850] <26542124 UPDT APPI DEBUG  host01mgmt>  BNT:net2:172.23.1.252:0:Compare of firmware success for switch after copy.
[05 Jun 2014 19:48:16,850] <26542124 UPDT APPI DEBUG  host01mgmt>  , Command Status->1
[05 Jun 2014 19:48:16,856] <26542124 UPDT  ERROR  host01mgmt> TASK_END::13::1 of 1::NetFWUPD::172.23.1.251::::RC=1::The NET FW update failed on the node 172.23.1.251.
[05 Jun 2014 19:48:16,858] <26542124 UPDT  INFO   host01mgmt> TASK_END::13::1 of 1::NetFWUPD::172.23.1.253::::RC=0::The NET FW update is successful on the node 172.23.1.253.
[05 Jun 2014 19:48:16,860] <26542124 UPDT  INFO   host01mgmt> TASK_END::13::1 of 1::NetFWUPD::172.23.1.254::::RC=0::The NET FW update is successful on the node 172.23.1.254.
[05 Jun 2014 19:48:16,862] <26542124 UPDT  INFO   host01mgmt> TASK_END::13::1 of 1::NetFWUPD::172.23.1.252::::RC=0::The NET FW update is successful on the node 172.23.1.252.
[05 Jun 2014 19:48:17,011] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=Infrastructure AND Solution_version=3.0.3.1, to update status of Product
[05 Jun 2014 19:48:17,108] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Sub_module_type=Infrastructure AND Solution_version=3.0.3.1, to update status of sub module
[05 Jun 2014 19:48:17,537] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=Infrastructure AND Solution_version=3.0.3.1, to update status of Product
[05 Jun 2014 19:48:17,602] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Sub_module_type=Infrastructure AND Solution_version=3.0.3.1, to update status of sub module
[05 Jun 2014 19:48:17,931] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=Infrastructure AND Solution_version=3.0.3.1, to update status of Product
[05 Jun 2014 19:48:17,998] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Sub_module_type=Infrastructure AND Solution_version=3.0.3.1, to update status of sub module
[05 Jun 2014 19:48:18,385] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=Infrastructure AND Solution_version=3.0.3.1, to update status of Product
[05 Jun 2014 19:48:18,454] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Sub_module_type=Infrastructure AND Solution_version=3.0.3.1, to update status of sub module
[05 Jun 2014 19:48:18,600] <26542124 UPDT APPI ERROR  host01mgmt> Error on nodes (172.23.1.251).
[05 Jun 2014 19:48:18,632] <26542124 UPDT APPI INFO   host01mgmt> STEP_END::13::NetFW_UPD::FAILED
[05 Jun 2014 19:48:18,642] <26542124 UPDT APPI DEBUG  host01mgmt> Error occured in apply for product netfw1
[05 Jun 2014 19:48:18,719] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=netfw1 AND Solution_version=3.0.3.1, to update status of Product
[05 Jun 2014 19:48:18,864] <26542124 UPDT APPI ERROR  host01mgmt> Apply (impact) phase for solution has failed.
[05 Jun 2014 19:48:18,935] <26542124 UPDT APPI DEBUG  host01mgmt> Executing query Logical_name=bwr3 AND Solution_version=3.0.3.1, to update status of Solution
[05 Jun 2014 19:48:19,088] <26542124 UPDT APPI INFO   host01mgmt> PHASE_END APPLY IMPACT
[05 Jun 2014 19:48:19,090] <26542124 UPDT APPI ERROR  host01mgmt> The apply phase for the release 'bwr3' failed.

Workaround:

Resume the firmware upgrade procedure by running the following command on the management host as root:

KI004393

I_V1.0

I_V1.1

.

Tuning database queries returns an error message that the database is not configured for tuning
 

After you have tuned some queries successfully, you might see an error message that states the database is not configured for tuning when you attempt to tune additional queries.

This problem can occur when you tune DDL statements using the web console, for example by selecting "Tune All with This Web Console" from the Execution Summary tab in the SQL dashboard. When DDL statements are tuned, they are also executed under the user ID that deployed the database, and some of the tables that are created under the schema of that user ID can cause problems with query tuning.

Workaround:

If the error occurs, connect to the database and look for tables that use the schema of the user ID that was used to deploy the database.

For those objects, delete the following tables, which can cause the error:

ADVISE_*
EXPLAIN_*
OBJECT_METRICS

You can also remove the following tables, which are not needed:

QT_*
OPT_PROFILE

To prevent the error from happening again, ensure that you do not tune DDL statements. You can use dashboard filters in the database performance monitor web console to remove DDL statements from the grid so that you do not tune them as part of a workload.

KI004721

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4

I_V1.1.0.0

When changing the password for the dirinst1 IBM Systems Director instance owner user, certain characters contained within the password causes the password change procedure to fail.

Workaround:

Do not include any of the following characters in the password for the dirinst1 IBM Systems Director instance owner user:

! % ˆ & ( ) | " ' ? , < > * $ @ = + [ ] \ / ; : . { } ` ˜ # - ´ ¨

In addition, do not begin the password for the dirinst1 IBM Systems Director instance owner user with any of the following characters:

_ 0 1 2 3 4 5 6 7 8 9

KI005291

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

Logging out from storage web console results in error message
 

This issue occurs when you access the storage web console from the system console by clicking System > Service Level Access, selecting one of the links in the IBM Storwize V7000 section, and logging out by clicking the Log out link on the storage web console. When you log out of the storage web console, a 404 File not found error is displayed.

Workaround:

When this issue occurs, click the Back button of your browser to return to the IBM Storwize V7000 login page.

• The appliance console is removed which removes the availability of this Service Level Access feature.

KI004678

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

As a result of selecting a number of events to delete from the Events page of the system console by using the Delete selected events button (red cross) that is located in the upper-right corner of the events table above the Actions column, the Delete the number selected events that are currently showing option deletes only 1 or 2 of the selected events, not all.

Workaround:

If you want to delete a small number of events, you can use the Delete button located in the Action column for each individual event.

If you want to delete a large number of events, you can filter events by choosing event text and attribute settings to list events and then delete the list of filtered events. It is important to note that this procedure deletes all of the events that match the current filter, even if the listed events are not individually selected for deletion. To delete a large number of filtered events from the Events page of the system console, do the following steps:

1. Navigate to the Events page by clicking System > Events found either in the System menu, or by expanding the Welcome page Working with the system section and clicking the link in Review system events.

2. Set the event filters located across the top of the events table to specifically list the events that you want to delete. Refresh the events list to see the list of filtered events.

3. To make the Delete selected events button visible above the Action column of the events table, select at least one of the filtered events.

4. Click the Delete selected events button.

5. To delete all of the events in the filtered list, select the Delete all events that match the current filter option. Click OK.
 

• The appliance console is removed which removes this feature.

KI004952

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

On the system console Hardware > Network Devices page, it is not possible to monitor the actual status of the Ethernet network switches because they are always shown as Available and Powered On.

Workaround:

To reliably verify the availability of Ethernet network switches by using the system console, do the following steps:

1. Navigate to the Events page by clicking System > Events found either in the System menu, or by expanding the Welcome page Working with the system section and clicking the link in Review system events.

2. Search for the keyword 'switch' on the Events page and filter the events with severity Critical or Informational.

The following example output, showing only three relevant columns, is the result of a filtered search:

Event text                            Type              Severity

Eth_network_switch_name is offline.   Network switch    Critical

Eth_network_switch_name is online.    Network switch    Informational

KI004434

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2

When attempting to use the miauth -pw command to reset a password for a node or hardware device, the following output is displayed in the system console:

Changing the password for user 'username' on resource 'device'.
The password change on the 'device' resource failed.
The script failed with the following message:
-------------
spawn /usr/bin/passwd username
Changing password for "username"
username's Old password:
3004-604 Your entry does not match the old password.

where username is the name of the user for the password that you are attempting to change, and device is the name of the node or hardware device.

The error message is displayed because a command other than miauth was previously used to change the node or hardware device password. Using a method other than the miauth command method results in password synchronization issues.

Workaround:

1. Run the following command:

miauth -u user_name -p device_type -pw -oldpw

where user_name is the user name of the node or device, and device_type is the type of node or hardware device. The device_type options are os, hmc, net, san, and storage.

2. When you are prompted for a new password, enter the password that you want to use for the node or hardware device you have specified.

3. When you are prompted to enter the old password, enter the password that was originally provided when it was changed with a command other than miauth.

After the password change is complete, a message is displayed indicating that the operation was successful.

KI004752

After several HA failures, IBM Tivoli System Automation for Multiplatforms (Tivoli SA MP) might not be able to restart the warehouse tools administration console because the warehouse tools application server (WASAPP) is stuck in an Online state.

-----------

Kill the warehouse tools application server profile process by completing the following steps:

1. Log in to the management node as the root user.

2. Delete the warehouse tools application server profile pid file by running the following command:

• rm /usr/IBM/dwe/appserver_001/appServer_10/profiles/AppSrv01/logs/server1/server1.pid

Deletion of the server pid file causes an inconsistency that eventually kills the pid process. The killed pid process removes the Stuck Online status on the application server and the monitor returns an Offline status. Tivoli SA MP can then restart the warehouse tools administration console.

KI005210

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.1.0.0

-----------

1. To restore the cluster, stop any DB2 resources by running the following command as root on the administration host:

• hastopdb2

2. Manually add the missing database partition entries to the db2nodes.cfg file.

3. Restart the DB2 resources by running the following command as root on the administration host:

• hastartdb2

KI004994

I_V1.0

I_V1.1

On occasion, running the hafailover command can result in the display of the following message:

Failed to start all resources. Check state with hals

This message is erroneous because sometimes the failover process takes longer to complete than the configured timeout period. All of the resources did not start because the failover process is still underway.

-----------

Periodically check the status of the failover by running the following command on the management host as root:

hals

If the resources are in a Pending online state, wait a few minutes before repeating the previous command. If all of the resources are Online and accessible after a reasonable period of time, no further action is required.

If all of the resources are not Online and accessible after a reasonable period of time, contact IBM Support for assistance.

KI004767

Upgrade of database performance monitor to version 5.3 results in warning that performance monitoring is not fully enabled
 

Some, rather than all, of the tables with an IBMPDQ schema are dropped from the opmdb performance monitor database during the database performance monitor upgrade process from version 5.2 to version 5.3. As a result, the following warning message is displayed in the database performance monitor user interface:

Performance monitoring is not fully enabled

-----------

Drop the remaining tables with an IBMPDQ schema in the opmdb performance monitor database by completing the following steps:

1. Bring the database performance monitor user interface offline by running the following command on the management host as the root user:

• chrg -o offline -s "Name = 'db2_perf-rg'"

2. Before proceeding to the next step, verify that the DPM COMPONENT has gone from a Pending to an Offline OPSTATE by running the following command:

• hals
  
  The following output is an example of running the previous command:
  
  MANAGEMENT DOMAIN
  +============+=========+=========+=========+=================+=================+=============+
  | COMPONENT  | PRIMARY | STANDBY | CURRENT | OPSTATE         | HA STATUS       | RG REQUESTS |
  +============+=========+=========+=========+=================+=================+=============+
  | WASAPP     | host01  | host03  | host01  | Online          | Normal          | -           |
  | DB2APP     | host01  | host03  | host01  | Online          | Normal          | -           |
  | DPM        | host01  | host03  | host01  | Offline         | Normal          | -           |
  | DB2DPM     | host01  | host03  | host01  | Online          | Normal          | -           |
  +============+=========+=========+=========+=================+=================+=============+

3. Connect to the opmdb performance monitor database by running the following command on the management host as the db2opm user:

• db2 connect to opmdb

4. Drop the IBMPDQ schema and all of the objects contained within it by running the following command:

• db2 "CALL SYSPROC.ADMIN_DROP_SCHEMA('IBMPDQ', NULL, 'ERRORSCHEMA', 'ERRORTABLE')"

5. Stop and restart the database performance monitor by running the following commands on the management host as the root user:

• hastopdpm
  hastartdpm

After the IBMPDQ schema is dropped, stopping and restarting the database performance monitor in step 5 results in a re-creation of the IBMPDQ schema and all of its objects, and the monitored database configuration is synchronized with the repository server.

Note: A result of stopping the database performance monitor in step 5 is a short period of uncollected database performance monitoring data.

KI005415

I_V1.0

I_V1.1

Powering up a node or nodes after a power shut down might result in the system log daemon (syslogd) not automatically starting on the host or hosts of the system. An inoperative syslogd results in the system log not being updated and error messages shown after running some commands.

To verify that syslogd is inoperative on any of the hosts, run the following command on any host as root:

dsh -n ${ALL} 'lssrc -s syslogd | grep inoperative' | dshbak -c

The following example output is a result of running the previous command and shows on which hosts the syslogd is inoperative:

HOSTS -------------------------------------------------------------------------
host02, host03, host04, host05, host06, host08
-------------------------------------------------------------------------------
 syslogd ras inoperative  3:23:38 PM 

To verify that an active syslogd is not logging events in the /var/log/syslog.out log file on each of the hosts, check for a zero file size. Run the following command on any host as root:

dsh -n ${ALL} 'v=/var/log/syslog.out;([ -f ${v} ] && wc -l ${v}) || (touch ${v} && wc -l ${v})'

The following example output is a result of running the previous command and shows that syslogd is not logging events on host04 because
the system log does not contain any entries as indicated by its zero
file size:

host01:    28735 /var/log/syslog.out
host03:    18750 /var/log/syslog.out
host08:     1059 /var/log/syslog.out
host06:    16870 /var/log/syslog.out
host02:     3971 /var/log/syslog.out
host05:    24312 /var/log/syslog.out
host07:    18656 /var/log/syslog.out
host04:    0 /var/log/syslog.out

-----------

If the system log daemon is inoperative or not logging events on any of the hosts, complete the following steps:

1. Verify that the /var/log/syslog.out log file exists, and if the log file does not exist, create it. Both of these tasks can be completed by running the following command on any host as root:

• dsh -n ${ALL} 'v=/var/log/syslog.out;[ ! -f ${v} ] && echo "Node does not have a valid $v file." && touch ${v}' | dshbak -c

2. Start the syslogd process by running the following command on any host as root:

• dsh -n ${ALL} 'refresh -s syslogd' | dshbak -c

3. Verify that an active system log daemon is logging events in the /var/log/syslog.out log file on each of the hosts by checking for a non-zero file size. Run the following command on any host as root:

• dsh -n ${ALL} 'v=/var/log/syslog.out;([ -f ${v} ] && wc -l ${v}) || (touch ${v} && wc -l ${v})'

KI006086

After installing PDOA fix pack 3 you may find that call-home settings on the primary and secondary HMC have been reset. This will occur if the HMC was at any firmware level older than V7.7.3 prior to installing PDOA fix pack 3.

-----------

KI004960

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4

Older versions of CAS have an unstable Communication State with IBM Systems Director. This unstable Communication State results in the appl_ls_hw command sometimes showing the server with a Communication State that is Off even though the server is reachable at times.

During the fix pack installation when the server reboots after the AIX update, there is a chance that the Communication State of the server is Off, which causes the AIX reboot operation to fail.

When a server fails to reboot the AIX OS, an error message is displayed in the system console. You can also check the pl_update.log file for a message similar to the following example message:

[18 Mar 2014 12:06:27,464] <29819052 UPDT APPI DEBUG  host01mgmt> AIX update Reboot Error on node server5
[18 Mar 2014 12:06:27,514] <29819052 UPDT  INFO   host01mgmt> TASK_END::1::3 of 3::AIXUPD_REBOOT::172.23.1.17::::RC=1::The AIX reboot operation failed on 172.23.1.17.

-----------

1. Log in to the host and verify that the server was actually restarted by running the following command:

• uptime
  
  The uptime command shows how long the server has been running.

2. Identify the logical name of the servers where the AIX reboot operation failed by running the following command from the management node as root:

• appl_ls_hw -M
  
  The following example output is a result of running the previous command:
  
  NAME       HOSTNAME      IP             MODULE        STATUS      DESCRIPTION 
  server0    host03mgt     172.21.1.14    Data          Stopping    IBM Power 730 Express Operating System
  server1    host02mgt     172.21.1.15    Foundation    Online      IBM Power 740 Express Operating System
  server2    host04mgt     172.21.1.16    Data          Online      IBM Power 730 Express Operating System
  server3    host01mgt     172.21.1.17    Foundation    Online      IBM Power 740 Express Operating System
  server4    host02mgt     172.21.1.18    Foundation    Online      IBM Power 740 Express Operating System
  server5    host06mgt     172.21.1.19    Data          Online      IBM Power 730 Express Operating System
  server6    host01mgmt    172.21.1.10    Foundation    Online      IBM Power 740 Express Operating System
  
  In the previous example output, servers that failed to reboot their AIX OS show a Stopping status. In this example output, server0 is in the Stopping state.

3. To bring online the server that failed to reboot the AIX OS, run the following command from the management node as root:

• appl_start -l <Server_logical_name>
  
  The following example output is a result of running the previous command to bring server0 online:
  
  (0) root @ stgkf201: 7.1.0.0: /
  $ appl_start -l server0
  Checking the status for the resource 'server0'.
  The resource 'server0' is online. Skipping the power on operation.
  The SSHD daemon started successfully on '172.21.1.14'.
  Retrieving and updating the status in the database for 'server0'.
  Retrieving and updating the child node(s) status in the database for 'server0'.
  NAME        HOSTNAME         IP            MODULE     STATUS         DESCRIPTION
  server0     host03corp       9.3.22.27     Data       Online         IBM Power 730 Express Operating System
  (0) root @ stgkf201: 7.1.0.0: /
  $

4. After the server shows an Online status, the fix pack installation process can be resumed by running the following command:

• miupdate -resume

KI004991

After resuming from a failed stage, the fix pack installation hangs. In the Fix Pack panel of the system console, the level 3.0.3.1 status indicates that the current stage was completed by displaying the status as Applied to management hosts or Applied to non-management hosts. However, clicking the View Progress button in the Fix Pack Details panel of the system console shows that the management or apply stage is in the Running state.

-----------

1. Log in as the root user on the management host and run the following command to determine if any miupdate -resume or MIUpdateCLI -resume processes are still running:

• ps -eaf | grep resume
  
  In the output returned by the command, identify the process IDs for the miupdate process and the MIUpdateCLI process. In the following sample output, the miupdate process ID is 11599914 and the MIUpdateCLI process ID is 12976198.
  
  root 11599914 1 0 12:25:11 - 0:00 /bin/sh /opt/IBM/mi/bin/miupdate -resume current -n
  root 12976198 11599914 0 12:25:11 - 0:02 /usr/java6_64/bin/java -classpath .:/opt/IBM/mi/lib/*:/opt/IBM/mi/lib/log4j/* -Dlog4j.configuration=file:/opt/IBM/mi/configuration/log4j.properties -Djavax.net.ssl.trustStore=/opt/ibm/director/lwi/security/keystore/ibmjsse2.jks -DUSERDIR=/usr/IBM/applmgmt/isas.server com.ibm.isas.cli.command.application.MIUpdateCLI -resume current -n
  root 21037306 35848408 0 12:39:58 pts/8 0:00 grep resume                

2. Terminate any running miupdate -resume or MIUpdateCLI -resume processes by running the following command:

• kill -9 <pid>
  
  where <pid> represents the process ID of the running miupdate or MIUpdateCLI process.

3. Click the View Progress button in the Fix Pack Details panel of the system console to check if the status of the stage has changed.

• If the status changed to Completed, start the next stage of the fix pack installation process by clicking the OK button in the system console.
• If the status is still Running, log in to the management node as the root user and run the following example command to start the next stage of the fix pack installation:
















































































































































 
• miupdate -u <next_stage>
  
  Examples:
  
  If you are currently in the management stage, run the following command to start the prepare stage:
  
  miupdate -u prepare
  
  If you are currently in the apply stage, run the following command to start the commit stage:
  
  miupdate -u commit

KI005115

The fix pack installation fails in the apply stage due to a problem with the SSD drawer.  You will see error messages in the /BCU_share/aixappl/pflayer/log/pl_update.log file that are similar to the following output:

• Running command -- LANG=en_US /usr/sbin/lsmcode -c -r -d ses1 -- on 172.23.1.13
  command output-> ??????
  return code -> 1
  A query of the firmware level of ses1 172.23.1.13 failed.
  Command Output->A query of the firmware level of ses1 172.23.1.13 failed.
  Return code->1

After you address the problem with the SSD drawer, the device name of the SSD drawer might change when you reboot the node. If the device name is changed, you will not be able to resume the fix pack installation. 

-----------

Address the problem with the SSD drawer and, if necessary, change the device name of the SSD drawer back to the original name before you resume the fix pack installation.

1.  Shut down the node connected to the SSD drawer:

• shutdown -F

2. Power off the BlueHawk SSD adapter and then power it back on.  

3. Start the node.  You can do this manually or through the HMC.

4.  Run the configuration manager: 

• cfgmgr

5. Verify if the name of the SSD drawer is reset to the original name.  The original name is contained in the error message.

• /usr/sbin/lsmcode -c -r -d <original_device_name>
  
  where <original_device_name> represents the original name of the SSD drawer in the error message.

6.  If the name of the SSD drawer was not reset to the original name, reset it.

• a.  Issue the following command:
  
  
  • rendev -l <original_device_name> -n <new_device_name>
  
  b. Verify that the name of the SSD drawer has been changed to the original name: 
  
  
  • /usr/sbin/lsmcode -c -r -d <original_device_name>

7.  Resume the fix pack installation:

• miupdate -resume

KI004080

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5

I_V1.1.0.1

The fix pack installation fails in the Apply to management hosts stage. After you address the issue and click Resume current phase in the system console, the system console does not respond and the fix pack installation does not continue.

-----------

1. Log in to the management host as the root user.

2. Verify that the fix pack installation is in the M_Applied_Nonimpact substage:

• appl_ls_cat

3. Resume the fix pack installation:

• miupdate -u management

KI004080

The fix pack installation fails in the Apply to management hosts stage. After you address the issue and click Resume current phase in the system console, the system console does not respond and the fix pack installation does not continue.

-----------

1. Log in to the management host as the root user.

2. Verify that the fix pack installation is in the M_Failed_Apply_Nonimpact substage:

• appl_ls_cat

3. Resume the fix pack installation:

• miupdate -u management

KI004080

If a failure occurs during the early management substage that validates the environments of the management and the standby management hosts, the status of the fix pack installation hangs at the Prepared management hosts state, as shown on the left side of the Fix Packs panel in the system console or you can use the appl_ls_cat command to view states. You can also verify that miupdate processes are not running by issuing the ps -ef | grep -i miupdate command. The Previewed state, which preceded the Prepared management hosts state, signalled the successful completion of the Preview stage of the fix pack installation procedure.

After the management environments are fixed and the management stage of the fix pack installation procedure is resumed, the installation stops in the M_Prepared state, which means the fix pack installation procedure successfully completed the management prepare substage, but does not continue further. Attempts to normally restart the fix pack installation process do not work.

-----------

To restart the fix pack installation procedure at the management stage, run the following commands in sequence from the management host as root:

miupdate -u management
miupdate -u prepare
miupdate -resume current
miupdate -u management

KI005346

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3

Fix pack installation fails with a GPFS error after updating the V7000 storage during the apply stage. GPFS on the management host displays arbitrating state during multiple attempts to start and finally ends with a start error as shown in the following portion of example entries in the pl_update.log:

[11 Jun 2014 17:37:47,787] <9699536 GPFS  DEBUG  plsapd01> Executing command on 172.23.1.1-> ssh root@172.23.1.1 /usr/lpp/mmfs/bin/mmgetstate -Y
[11 Jun 2014 17:37:49,274] <9699536 GPFS  DEBUG  plsapd01> Return code-> 0
[11 Jun 2014 17:37:49,275] <9699536 GPFS  DEBUG  plsapd01>  Output-> , mmgetstate::HEADER:version:reserved:reserved:nodeName:nodeNumber:state:quorum:nodesUp:totalNodes:remarks:cnfsState
[11 Jun 2014 17:37:49,275] <9699536 GPFS  DEBUG  plsapd01>  mmgetstate::0:1:::plsapd01:1:arbitrating:1*:0:8::(undefined):
[11 Jun 2014 17:37:49,276] <9699536 GPFS  DEBUG  plsapd01> GPFS state on node 172.23.1.1 : arbitrating
[11 Jun 2014 17:37:49,277] <9699536 GPFS  DEBUG  plsapd01> GPFS is not yet started completly on 172.23.1.1. waiting for 30 sec more
[11 Jun 2014 17:38:16,007] <23199746 GPFS  DEBUG  plsapd01> Timeout:GPFS is not active on node 172.23.1.3
[11 Jun 2014 17:38:19,277] <9699536 GPFS  DEBUG  plsapd01> Timeout:GPFS is not active on node 172.23.1.1
[11 Jun 2014 17:38:19,300] <21954626 STRT  ERROR  plsapd01> Could not start the product 'GPFS' on 'server7 server2'
[11 Jun 2014 17:38:19,331] <20840644 UPDT APPI WARN   plsapd01> Failed to start solution.
[11 Jun 2014 17:38:19,381] <20840644 UPDT APPI DEBUG  plsapd01> Executing query Logical_name=bwr1 AND Solution_version=3.0.3.1, to update status of Solution
[11 Jun 2014 17:38:19,489] <20840644 UPDT APPI INFO   plsapd01> PHASE_END APPLY IMPACT
[11 Jun 2014 17:38:19,490] <20840644 UPDT APPI DEBUG  plsapd01> Apply phase done for release 'bwr1' successfully
[11 Jun 2014 17:38:19,491] <20840644 UPDT RESU INFO   plsapd01> PHASE_END RESUME
[11 Jun 2014 17:38:19,493] <20840644 UPDT RESU ERROR  plsapd01> The resume phase for the release 'bwr1' failed.

-----------

Fix the GPFS error and then continue the fix pack installation by completing the following steps.

1. Run the following commands on the management host as the root user:

• appl_start -g -s Admin
  appl_start -q -s Management 
  appl_start

2. Check the fix pack installation status by running the following command:

• appl_ls_cat
  • If the fix pack installation status is in the Applied state, proceed to the Commit phase.
  • If the fix pack installation status is not in the Applied state, resume the fix pack installation by running the following command:
  
  
  • miupdate -resume

• The versions above introduced a GPFS design change that separates the management hosts into their own GPFS cluster.

KI005022

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.1

he message display does not update when resuming the fix pack installation after an error during the management stage. The system console continues to show the following error message:

Error: Error on nodes (172.23.1.10).. Refer to the platform layer log file for details. To ...

For example, when the management host reboots, the warning message that the management host is rebooting is not shown. This warning message is normally shown when the management host reboots during the management stage of a successful fix pack installation process that does not require to be resumed after an error or a failure.

-----------

KI004283

I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5

I_V1.0.0.6

I_V1.1.0.1

I_V1.1.0.2

During fix pack installation, the Ethernet switches might lose their configurations when the switch firmware is upgraded in two stages, first to an interim level, then to the target level.

-----------

Contact IBM Support and reference this technote. Do no proceed without working with a support engineer.

As a precautionary measure, automated backups of the switch configurations are taken before the preview phase and during the apply phase of the fix pack installation in V1.0.0.3 and above. The backup files are located on the management host in the /BCU_share/net_switch_backup folder.

Restore the Ethernet switch configurations from their configuration backup files by completing the following steps:

1. Copy the configuration files to a folder on a USB drive or on a local computer after the preview phase. As an example, kf3 is the USB drive folder name used here when the usbcopy command is run at a later step.

Explanation: Instruct the customer to make a copy of the switch configuration backup files to a USB drive or their local computer after the preview stage because the switch usually loses its configurations during the apply phase. It will be impossible to remotely access the management host to make a copy of the configuration backup files after the apply phase if the switch has lost its configurations.

2. Connect a serial cable to the serial port on the switch and start a hyperterminal session.

3. Enter the following settings for the hyperterminal:

• Bits per second: 9600
  Data bits: 8
  Parity: None
  Stop bits: 1
  Flow controls: None
  
  Click OK.

4. Click Enter on the hyperterminal session.

5. Enter the following password:

• admin

6. At the prompt, run the following command:

• > enable

• The following message is displayed:
  Enable privilege granted.

7. Run the following command:

• # config t
  
  The following message is displayed:
  Enter configuration commands, one per line. End with Ctrl /Z.

8. Install the USB drive with both the OS and boot image firmware loaded.

9. Run the following commands:

• > enable
  # config t
  # clear flash-config
  Enter Y to continue.

10. Run the following command:

• # reload
  Enter Y to continue.

11. Log in to switch and enter the following password:

• admin

12. Enter the following commands:

• > enable
  # config t

Now run the usbcopy cmd by using the following general format:

• usbcopy fromusb USB_folder_name/Backup_file_name active

For top 1G switch

• # usbcopy fromusb kf3/172.23.3.11_backup.txt active

For bottom 1G switch

• # usbcopy fromusb kf3/172.23.3.12_backup.txt active

For top 10G switch

• # usbcopy fromusb kf3/172.23.3.21_backup.txt active

For bottom 10G switch

• # usbcopy fromusb kf3/172.23.3.22_backup.txt active

13. Enter the following command:

• # dir

Look for the file that contains the loaded configuration information by checking the time stamp. The file with the latest time stamp is the one most recently loaded.

• Conf1 is an active configuration.
  Conf2 is a backup configuration.
• If the configuration is active (if conf1 has latest time stamp), run the following commands:

• # copy active-config running-config
  # copy running-config startup-config
  
  Enter Y to continue.

• If the configuration is backup (if conf2 has latest time stamp), run the following commands:

• # copy backup-config running-config
  # copy running-config startup-config
  
  Enter Y to continue.

14. Reload the switch configurations by running the following command:

• # reload

• Enter Y to continue.

KI005412

This issue can occur for either of the following reasons:

• Adding a new node on which fix pack 3 was installed.
• Customer installs fix pack 3 themselves.

The fix pack updates the console code and introduces a bug that prevents the SAN switch status in the Hardware > Network Devices panel from getting updated monitoring information. For example, if the SAN switch temperature changes, this does not get registered in the console. Another sign of this problem is seen in an add-node scenario. When adding a new node on which fix pack 3 was installed, all of the listed network devices, including the SAN switches, are shown as a Network Switch type. When you click on any switch listed in the Hardware > Network Devices panel to view the details, the Model field is not set.

-----------

To fix this issue, a console code patch must be performed after the installation of fix pack 3.

1. Download the patched query_switches.groovy file from IBM Support.

2. With the patched file, replace the broken query_switches.groovy file in the following directory:

• /usr/IBM/applmgmt/isas.async/private/expanded/ibm/isas.hardware-1.0.0.3/app/scripts/switches

3. After replacing the broken query_switches.groovy file with the patched file, verify that the patched file has the following permissions:

• -rw-r--r--
  
  If the permissions on the file are not as previously shown, run the following command:
  
  chmod 644 /usr/IBM/applmgmt/isas.async/private/expanded/ibm/isas.hardware-1.0.0.3/app/scripts/switches/query_switches.groovy

KI004996

I_V1.0.0.3

While upgrading the 6.4.0.3 level firmware of the V7000 storage controller to 6.4.1.6 level, the upgrade process hangs in the upgrading state. The following example message is shown:

apply: 172.23.1.237: broke out of timed wait after 36 iterations of maximum 36. update status is <upgrading>

If the previous example message was not seen, you can verify the upgrade status by running the following command:

svcinfo lssoftwareupgradestatus -nohdr

The following example output is a result of running the previous command:

upgrading

-----------

Contact IBM Support before proceeding with the workaround. Only a qualified SSR or CE should be allowed to manipulate the hardware which can only be engaged through IBM Support as this minimizes the risk of loss of storage configuration or loss of data in the worst case.

1. Reseat the canister of the offline storage controller that is stuck in the upgrading state. Usually only one of the two storage canisters is stuck. The stuck canister is the one that does not respond to ping commands.

• a. After reseating the canister, wait 15-30 minutes to give it time to rejoin the cluster in an online state. You can run the following command to verify that the canister (nodex) is now in an online state before proceeding to the next step:
  
  
  • svcinfo lsnodecanister
    
    Example output:
    
    id name  UPS_serial_number WWNN             status IO_group_id IO_group_name config_node UPS_unique_id    hardware iscsi_name                            iscsi_alias panel_name enclosure_id canister_id enclosure_serial_number
    1  node1                   5005076802006DBC online 0           io_grp0       no          5005076802006DBC 100      iqn.1986-03.com.ibm:2145.v70001.node1             01-1       1            1           78N17BH
    2  node2                   5005076802006DBD online 0           io_grp0       yes         5005076802006DBD 100      iqn.1986-03.com.ibm:2145.v70001.node2             01-2       1            2           78N17BH
  
  b. Verify the upgrade status by running the following command:
  
  
  • svcinfo lssoftwareupgradestatus
    
    Example output:
    
    status
    upgrading

 

• • If the upgrade status is upgrading, then let the upgrade process continue until the upgrade status is inactive, which indicates the upgrade was successful.

• • If the upgrade status is stalled, then continue to step 2.

2.  The upgrade status changes to the stalled state. Verify by running the following command as the superuser on the V7000 that is having the issue:

• svcinfo lssoftwareupgradestatus -nohdr
  
  Example output:
  
  stalled

3. To abort the firmware upgrade process and revert the firmware back to its previous level, run the following command as the superuser on the V7000 that is having the issue:

• svcservicetask applysoftware -abort

4. To verify that the firmware upgrade process is downgrading the firmware back to its previous level, run the following command as the superuser on the V7000 that is having the issue:

• svcinfo lssoftwareupgradestatus -nohdr

      

• Example output:
  
  downgrading

5. The storage controller becomes inaccessible and does not respond to pings for a few minutes. This is expected. Wait until it becomes accessible before continuing to the next step.

6. After logging in to the cluster, the firmware is back at the 6.4.0.3 level. Verify that the firmware upgrade process is in an inactive state by running the following command as the superuser on the V7000 that is having the issue:

• svcinfo lssoftwareupgradestatus -nohdr

     

• Example output:
  
  inactive

7.  Copy the firmware upgrade image to the /home/admin/upgrade Storwize directory by running the following example command on the management host as root:

• scp <image_dir>/<image_name> superuser@<v7k_ip>:/home/admin/upgrade
  
  Example:
  
  scp /BCU_share/bwr3/firmware/storage/2076/image/imports/controller/IBM2076_INSTALL_6.4.1.6 superuser@172.23.1.185:/home/admin/upgrade

8. To resume the V7000 storage controller firmware upgrade, run the following command on the management host as root:

• miupdate -resume
  
  Note: You can also resume the firmware upgrade by using the system console. With the fix pack installation in a failed state, press the OK button with the following option selected: The error condition has been addressed. Resume this stage.

KI005345

Firmware upgrades do not succeed for V7000 storage drives that have a different product ID that is not part of the upgrade bundle or have a higher firmware level than the upgrade target level.

-----------

V1.0.0.3 Fixpacks ONLY!

After the preview stage (stage 1) completes and before you start the management stage (stage 2), install a manual fix for the V7000s.

1. Download in binary mode the Storwize.pm_Ctrl file and the Storwize.pm_UPD file from the IBM Support.

• Important: If the files are not downloaded in binary mode, the ^M characters might be appended to each file, which can cause code compilation to fail. Check each downloaded file for the ^M characters and remove them, if found. Otherwise, you can use checksum (cksum command) to confirm that the downloaded files on the AIX machine were not altered.

2. Copy the files to the /BCU_share directory on the management host.

3. Determine the logical name assigned to the fix pack by running the following command:

• appl_ls_cat
  
  The name is listed in the row that contains the version 3.0.3.1 (FP3). In the sample output, the logical name is bwr2.
  
  Sample output:
  
  NAME VERSION STATUS    DESCRIPTION
  bwr0 3.0.0.0 Committed Initial images for IBM PureData System for Operational Analytics
  bwr1 3.0.1.0 Committed Updates for IBM_PureData_System_for_Operational_Analytics
  bwr2 3.0.3.1 Applied   Updates for IBM_PureData_System_for_Operational_Analytics

4. Copy files to the appropriate locations by running the following commands as root on the management host:

• cd /BCU_share
  
  cp $PL_ROOT/lib/Ctrl/Updates/Storwize.pm $PL_ROOT/lib/Ctrl/Updates/Storwize.pm_org
  
  cp /BCU_share/<name>/code/ISAS/Update/Foundation/Infrastructure/Storwize/Storwize.pm /BCU_share/<name>/code/ISAS/Update/Foundation/Infrastructure/Storwize/Storwize.pm_org
  
  cp Storwize.pm_UPD /BCU_share/<name>/code/ISAS/Update/Foundation/Infrastructure/Storwize/Storwize.pm
  
  cp Storwize.pm_Ctrl $PL_ROOT/lib/Ctrl/Updates/Storwize.pm
  
  Note: In the following commands, replace <name> with the logical name that you identified in step 3.

KI005222

The appl_stop command successfully stops and reboots the Hardware Management Console (HMC), but a failure return code of 1 is falsely reported. The appl_stop command calls the hmcshutdown command that succeeds to call the ForceShutdown script. The erroneous return code is caused by the script not completing before the exit of the hmcshutdown command.

The following list contains the commands that are affected by this false status reporting issue:

appl_stop -l logical_name_of_hmc
appl_stop -r resource_type_of_hmc
appl_stop -l logical_name_of_hmc -R
appl_stop -r resource_type_of_hmc -R

Example output of the appl_stop command to stop the HMC:

appl_stop -l hmc0

Checking the status for the resource 'hmc0'.
Stopping the resource, 'hmc0'. This may take long time.
The SSHD daemon started successfully on '172.23.4.241'.
The power off operation failed for the resource hmc0.
Error stopping the resource, hmc0.

The HMC status shows that it is Stopping and eventually does stop. Example shows only the HMC part of the output:

appl_ls_hw

NAME   HOSTNAME   IP         MODULE     STATUS       DESCRIPTION
hmc0              9.3.2.16              Stopping     IBM Hardware Management Console
hmc1              9.3.2.17              Online       IBM Hardware Management Console

-----------

1. Ignore the false return code 1 that is reported after running the appl_stop command.

2. Run the appl_start command to restart the HMC after it was stopped or rebooted.

3. Ping the HMC after completion of the restart or reboot to verify that it is online.

KI004775

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

After a prolonged idle time while logged in to the system console, no contents are displayed and you might see the error message Sorry, an error occurred when you refresh the system console display or click on the UI. You might also be prompted to log in to the system console again, but are unable to log in when you provide the correct user ID and password.

The system console fails to display the correct content because the isas.server module is stopped and the isas.console.system is stopped or in an unknown state. You might also see the log in prompt due to the GUI timing out, but because the isas.server module is down, the user ID and password cannot be authenticated and the Invalid user name and password error message is displayed.

-----------

1. Verify that the isas.server module is stopped and the isas.console.system is stopped or in an unknown state by running the following command:

• mistatus -v

2. Start the isas.server module by running the following command:

• mistart

KI004810

When the SAS RAID adapter in an SSD enclosure encounters an error that the adapter does not understand, the adapter assumes a Degraded state and disables the adapter cache as a precaution.

Note: There is no significant impact with the degraded adapter state and a disabled adapter cache in a RAID 10 array. However, customer perception is impacted when they see a Degraded array state without an explanation or a hint about the nature of the problem.


Diagnostic prerequisite:

Verify the names of your SSD RAID adapters on the system. On the administration and standby administration hosts, the adapters are sissas1 and sissas2. On the data and standby hosts, the adapters are sissas2 and sissas3.

Run the following command on the administration host and a data host:

lsdev|grep "RAID SAS"

The following output is a result of running the previous command on the administration host:

(130) root @ paradise02: 7.1.0.0: /
$ lsdev|grep "RAID SAS"
sissas1        Available 06-00       PCIe2 3.1GB Cache RAID SAS Enclosure 6Gb x8
sissas2        Available 07-00       PCIe2 3.1GB Cache RAID SAS Enclosure 6Gb x8

The following output is a result of running the previous command on a data host:

(130) root @ paradise05: 7.1.0.0: /
$ lsdev|grep "RAID SAS"
sissas2        Available 09-00       PCIe2 3.1GB Cache RAID SAS Enclosure 6Gb x8
sissas3        Available 0A-00       PCIe2 3.1GB Cache RAID SAS Enclosure 6Gb x8

Detection of the problem:

There is no indication in the errpt that there is a problem. However, you can obtain an indication that there is a problem by either of the following two methods:

1. Run the following command to check for a Degraded state of the RAID 10 Array on each of the SSD RAID adapters:

• sissasraidmgr -L -l sissas2 -j 3

-----------

Contact IBM Support before proceeding with any workaround.

KI005061

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4

IBM Systems Director storage control update fails during inventory collection
 

The fix pack update fails during the management stage while updating the IBM Systems Director storage control component.

There is a known issue with IBM Systems Director where the ssh server drops the ssh connection during an update due to its high workload, many ciphers, or slow responses from the target, any of which can cause a timeout and result in collect inventory failure.

Investigation of the PL trace file finds that the error is due to a connection issue. The following message is found in the PL trace file:

Inventory collection failed for system "sysNode". Verify the connection to the system and collect inventory.

The following example entries were found in the PL trace file:

[22 May 2014 12:48:21,886] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301> ATKUPD764I Update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" was installed on system "sysNode" successfully.
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301> ATKUPD795I You must manually restart the IBM Systems Director management server after this install completes for the updates to take effect.
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301> ATKUPD739I Collecting inventory on system "sysNode".
[22 May 2014 12:48:21,887] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,888] <11403344 UPDT APPI TRACE stgkf301> ATKUPD711I Still collecting inventory for system "sysNode".
[22 May 2014 12:48:21,888] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,888] <11403344 UPDT APPI TRACE stgkf301> ATKUPD706E Inventory collection failed for system "sysNode". Verify the connection to the system and collect inventory.
[22 May 2014 12:48:21,888] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,888] <11403344 UPDT APPI TRACE stgkf301> ATKUPD572I Running compliance on system "sysNode".
[22 May 2014 12:48:21,889] <11403344 UPDT APPI TRACE stgkf301>
[22 May 2014 12:48:21,889] <11403344 UPDT APPI TRACE stgkf301> ATKUPD734E An error was encountered during the "Install Updates" task. Search above for previous related errors, fix each error, and then retry the operation. 

-----------

If the cause of the update failure was a connection issue due to high workloads or delays in the connection, the problem can be resolved by resuming the fix pack installation. Run the following command on the management host as root:

• miupdate -resume

If the previous command fails to resume the fix pack installation, contact IBM Support.

KI005116

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4

After updating the IBM Systems Director (ISD), the processes might get locked and hit a resource crunch. The ISD local CAS agent fails to start and the server communication state is stuck at 3 (expected state value: 2) leading to a Storage Control (SC) update failure.

To confirm that this issue has occurred, check for the corresponding messages in the /var/aixappl/pflayer/log/pl_update.trace file by running the following command on the management host as root:

tail -f /var/aixappl/pflayer/log/pl_update.trace

The following example output is a result of running the previous command:

[03 Jun 2014 23:45:54,341] <13631536 UPDT APPI TRACE  hostname>
[03 Jun 2014 23:45:54,341] <13631536 UPDT APPI TRACE  hostname>  ATKUPD275E The managed system "sysNode" is either offline or locked. Ensure that the system is available and attempt to perform the operation again.

[03 Jun 2014 23:45:54,341] <13631536 UPDT APPI TRACE  hostname>
[03 Jun 2014 23:45:54,341] <13631536 UPDT APPI TRACE  hostname>  ATKUPD287E The install needed updates task has completed with errors. Read the above messages for details on the error.
[03 Jun 2014 23:45:54,341] <13631536 UPDT APPI TRACE  hostname>
[03 Jun 2014 23:45:54,342] <13631536 UPDT APPI TRACE  hostname>  ATKUSC307E Command installneeded completed with errors. For more information, see the job log for job InstallNeededTask.
[03 Jun 2014 23:45:54,342] <13631536 UPDT APPI TRACE  hostname>
[03 Jun 2014 23:45:54,342] <13631536 UPDT APPI TRACE  hostname> command /opt/ibm/director/bin/smcli installneeded -v -F /BCU_share/bwr3/software/DirectorServer/imports/SCupdates4241 returned status =  32

-----------

1. Reboot the management host by running the following command on the management host as root:

• reboot

2. After the management host comes up, verify that the ISD is Active, that the ISD version is 6.3.3.1, and that the local CAS agent is running by running the following commands on the management host as root:

• smstatus
  
  smcli lsver
  
  /opt/ibm/director/agent/runtime/agent/bin/endpoint.sh status
  
  If the status of the local CAS agent is Stopped, contact the support team.

3. Collect server inventory by running the following command on the management host as root:

• smcli collectinv -p “All Inventory” -i <IP_mgmt_server>
  
  where <IP_mgmt_server> represents the IP address of the management server.

4. Install the SC update by running the following command on the management host as root:

• smcli installneeded -v -F <path_of_SC_update_package>
  
  The following example output is a result of running the previous command:
  
  smcli installneeded -v -F /BCU_share/bwr3/software/DirectorServer/imports/SCupdates4241
  
  ATKUPD489I Collecting inventory for one or more systems.
  ATKUSC206I Generating SDDs for path: "/BCU_share/bwr3/software/DirectorServer/imports/SCupdates4241".
  ATKUPD293I Update "com.ibm.director.storage.storagecontrol.member.AIX_4.2.4.fp1-build-00009" was successfully imported to the library.
  ATKUPD293I Update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" was successfully imported to the library.
  ATKUPD573I Running compliance for all new updates that were found.
  ATKUPD286I The import updates task has completed successfully.
  ATKUPD725I The update install task has started.
  ATKUPD487I The download task has finished successfully.
  ATKUPD629I Installation staging will be performed to 1 systems.
  ATKUPD632I The Installation Staging task is starting to process system "sysNode".
  ATKUPD633I The Installation Staging task has finished processing system "sysNode".
  ATKUPD630I The update installation staging has completed.
  ATKUPD760I Start processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD770I Still processing update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" and system "sysNode".
  ATKUPD764I Update "com.ibm.director.storage.storagecontrol.mgr.AIX_4.2.4.fp1-build-00009" was installed on system "sysNode" successfully.
  ATKUPD795I You must manually restart the IBM Systems Director management server after this install completes for the updates to take effect.
  ATKUPD739I Collecting inventory on system "sysNode".
  ATKUPD572I Running compliance on system "sysNode".
  ATKUPD727I The update install task has finished successfully.
  ATKUPD288I The install needed updates task has completed successfully.

5. After the completion of the SC update installation, stop and restart the ISD by running the following commands on the management host as root:

• smstop
  
  smstart
  
  Verify that the ISD server is Active before proceeding to the next step by running the following command on the management host as root:
  
  smstatus -r

6. Verify that the ISD version was updated to 6.3.3.1 by running the following command on the management host as root:

• smcli lsver

7. Verify that the SC version was updated to 4.2.4.1 by running the following command on the management host as root:

• cat /opt/ibm/director/StorageControl/SCVersion.properties | grep StorageControlVersion
  
  The following example output is a result of running the previous command:
  
  StorageControlVersionWithBuildPadded=4.2.4.fp1-build-00010-20140421-iFix
  StorageControlVersion=4.2.4.1

8. Resume the fix pack installation.

KI005982

I_V1.0.0.3

If your system locale is set to a language other than English, you will find that the "+" icon is missing when you attempt to add a new fix pack file to the management host on your 1.0.0.3 (Fix Pack 3) system by using the Fix Packs panel of the system console. Not being able to add and register the new fix pack prevents you from installing the new fix pack by using the system console.

-----------

To restore the missing "+" icon in the Fix Packs panel of the system console so that you can install the fix pack by using the system console, complete the following steps:

1. Set the system locale to English.
2. Restart the system console.
3. Install the fix pack by using the system console.
4. Commit the fix pack installation.
5. Set the system locale to the desired language.

KI006755

After installing PDOA fix pack 4 you may find that single sign-on from the console to the DPM and/or Warehouse Admin console(s) fails with an Error 500 message.

-----------

You must download the Unrestricted JDK JCE policy files and patch them into /usr/java6_64/jre/lib/security directory on the management host. The exact download locations and steps are listed below:

1. Download the Unrestricted JDK JCE policy files from https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=jcesdk . Select the option "Java 5.0 SR16, Java 6 SR13, Java 6 SR5 (J9 VM2.6), Java 7 SR4, Java 8 GA, and all later releases " for download.

2. Unzip the unrestrictedpolicyfiles.zip which will give you 2 JAR files: local_policy.jar and US_export_policy.jar

3. On the management node, as root, create a backup dir /usr/java6_64/jre/lib/security/backup and move the files local_policy.jar and US_export_policy.jar to this directory.

4. Place the new JARs of the same name in the management node's /usr/java6_64/jre/lib/security/ directory.

5. Run this command on the management node as root:
miresolve -restart

These 5 steps will fix the problem with single sign-on and restart the console. You can now login normally to the DPM and Warehouse Admin consoles.

KI004026

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

The maximum number of simultaneous SSH sessions that can be connected to an Ethernet network switch is four.

The following message is displayed after attempting to create a fifth SSH session from the system console:
java.net.ConnectException: A remote host refused an attempted connect operation.

The following example message is displayed after attempting to create a fifth SSH session from a terminal application on a remote computer:
Server unexpectedly closed network connection

-----------

Do not attempt to connect more than the maximum number of four simultaneous SSH sessions to an Ethernet network switch.

Close one or more of the four existing SSH sessions to be able to connect one or more new SSH sessions to a maximum of four.

KI004712

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.0
I_V1.1.0.1

The port number of any link that is selected from a storage web console page is lost and results in a 404 File not found error. This issue occurs when you access the storage web console from the system console by clicking System > Service Level Access, selecting one of the links in the IBM Storwize V7000 section, and after logging in as the superuser for the first time.

-----------

When this issue occurs, do the following steps:

1. Click the Back button of your browser to return to the IBM Storwize V7000 login page. Save the port number that you find in the URL in the address bar of your browser.

2. Click the Forward button of your browser to return to the error page. Insert the missing port number in the URL that you find in the address bar of your browser. Load this web page by pressing the Enter key or clicking the browser Load button.

KI007646

I_V1.0

I_V1.1

When the screen resolution is lower than the 1280x1024 minimum browser resolution requirement for the Firefox browser, a scroll bar is not present to access the bottom portion of the database performance monitor system console panel. This issue prevents being able to configure some workload management variables.

-----------

To resolve this issue, you can take one of the following actions:

• Use a system that can support the 1280x1024 screen resolution requirement.
• Use a different browser, such as Chrome or Internet Explorer.

KI006085

Accesssys command might fail and cause a locked state after changing the network switch password
 

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.1.0.0

Accesssys command might fail and cause a locked state after changing the network switch password
 

fter changing the password for a network switch, the process involves discovering the network switch and then running the accesssys command that requests secure access to the switch by using the new password. On occasion, the accesssys command fails to get access to the switch and the switch goes into a locked state.

To confirm that this issue occurred, look for return code 66 in the pflayer log file. The following pflayer log file example shows the accesssys command failed:

[09 Jan 2014 12:07:20,367] <24248418 CTRL  DEBUG  host01> Executing command->/opt/ibm/director/bin/smcli discover -i 172.23.1.251 -t "Switch"
[09 Jan 2014 12:07:31,595] <24248418 CTRL  DEBUG  host01> Ret Code-> 0
[09 Jan 2014 12:07:31,596] <24248418 CTRL  DEBUG  host01>  Command output-> Discovery completion percentage 50%
[09 Jan 2014 12:07:31,596] <24248418 CTRL  DEBUG  host01>  Discovery completion percentage  100%
[09 Jan 2014 12:07:31,596] <24248418 CTRL  DEBUG  host01>  Discovery completed:
[09 Jan 2014 12:07:31,596] <24248418 CTRL  DEBUG  host01>  100%
[09 Jan 2014 12:07:31,597] <24248418 CTRL  DEBUG  host01> Waiting for 20 sec between discovery and accesssys
[09 Jan 2014 12:07:51,598] <24248418 CTRL  DEBUG  host01> Executing command-> /opt/ibm/director/bin/smcli lssys -i 172.23.1.251 -t "Switch"
[09 Jan 2014 12:07:52,011] <24248418 CTRL  DEBUG  host01> Ret Code-> 0
[09 Jan 2014 12:07:52,011] <24248418 CTRL  DEBUG  host01>  Command output-> fcm_switch2
[09 Jan 2014 12:07:52,328] <24248418 CTRL  DEBUG  host01> Executing command ->/opt/ibm/director/bin/smcli accesssys -u admin -p ***** -i 172.23.1.251
[09 Jan 2014 12:07:52,606] <24248418 CTRL  DEBUG  host01> Ret Code-> 66
[09 Jan 2014 12:07:52,606] <24248418 CTRL  DEBUG  host01>  Command output-> DNZCLI0727I : Waiting for request access to complete on... fcm_switch2
[09 Jan 2014 12:07:52,606] <24248418 CTRL  DEBUG  host01>  Result Value: DNZCLI0736I : The system is not available. : fcm_switch2
[09 Jan 2014 12:07:52,649] <24248418 CTRL  DEBUG  host01> For net switch accessys waiting 20 sec more in case of failure
[09 Jan 2014 12:08:12,650] <24248418 CTRL  DEBUG  host01> For net switch executing accessys again in case of failure
[09 Jan 2014 12:08:12,970] <24248418 CTRL  DEBUG  host01> Executing command ->/opt/ibm/director/bin/smcli accesssys -u admin -p ***** -i 172.23.1.251
[09 Jan 2014 12:08:13,209] <24248418 CTRL  DEBUG  host01> Ret Code-> 66
[09 Jan 2014 12:08:13,209] <24248418 CTRL  DEBUG  host01>  Command output-> DNZCLI0727I : Waiting for request access to complete on... fcm_switch2
[09 Jan 2014 12:08:13,209] <24248418 CTRL  DEBUG  host01>  Result Value: DNZCLI0736I : The system is not available. : fcm_switch2
[09 Jan 2014 12:08:13,219] <24248418 CTRL  ERROR  host01> Discovery failed for net3.

-----------

To unlock and access the network switch, run the following command:

smcli accesssys -i <switch_ip> -u admin -p <password>

KI006330

I_V1.0.0.3

I_V1.0.0.4

I_V1.0.0.5

I_V1.1.0.1

-----------

You must locate the fix pack files on the management host in an absolute path that starts with only the following file systems:

• /backups
• /BCU_share
• /opmfs
• /opt
• /optimdatatools
• /tmp
• /usr
• /var

The following examples are valid absolute paths on the management host:

• /BCU_share/
• /tmp/<dir_name1>/
• /usr/<dir_name1>/<dir_name2>/

KI006280

SAN switch password change fails
 

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.1.0.0

-----------

To resolve the issue, complete the following steps:

1. Rediscover the SAN switch by running the following command:

• /opt/ibm/aixappl/pflayer/bin/icmds/appl_ctrl_san discover -l <resource_logical_name>

2. Retry changing the SAN switch password by running the following command:

• appl_conf chpw -l <resource_logical_name> -u <user_name> -p <new_password>

KI006280

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.1.0.0

After the successful change of the SAN switch passwords, the lssys command does not return a list of the SAN switches.

Example command line output to change the password:

$ time appl_conf chpw -r san -u admin -p -o
Enter the new password
Enter the old  password
Changing the password for user 'admin' on resource 'san0'.
The password change was successful for the user 'admin' on resource 'san0'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Password change notification to Systems Director is successful.
Changing the password for user 'admin' on resource 'san1'.
The password change was successful for the user 'admin' on resource 'san1'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Password change notification to Systems Director is successful.
Changing the password for user 'admin' on resource 'san2'.
The password change was successful for the user 'admin' on resource 'san2'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Password change notification to Systems Director is successful.
Changing the password for user 'admin' on resource 'san3'.
The password change was successful for the user 'admin' on resource 'san3'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Password change notification to Systems Director is successful.
The script was executed with following status:
----------------------------------------------------------
SCHEMA::CHPW::LOGICAL_NAME::STATUS(PASS/FAIL)::DESCRIPTION
CHPW::san0::PASS::The password change for 172.23.3.31 was successful.
CHPW::san1::PASS::The password change for 172.23.3.32 was successful.
CHPW::san2::PASS::The password change for 172.23.3.33 was successful.
CHPW::san3::PASS::The password change for 172.23.3.34 was successful.
----------------------------------------------------------

Example lssys command output:

$ smcli lssys -l -i 172.23.3.32
DNZCLI0241E : (Run-time error) The system with IP address or host name 172.23.3.32 was not found.
Use the smcli lssys -A IPv4Address,HostName command to list IP addresses and host names for all systems.

-----------

To resolve the issue, complete the following steps:

1. Rediscover the SAN switch by running the following command:

• /opt/ibm/aixappl/pflayer/bin/icmds/appl_ctrl_san discover -l <resource_logical_name>

2. Retry listing the SAN switch by running the following command:

• smcli lssys -l -i <san_ip_address>

KI006280

Ethernet network switch goes into partially locked state during password change
 

I_V1.0.0.0
I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.1.0.0

After changing the password for the Ethernet network switches, the switch status is listed as PartiallyLocked.

Example lssys command output before the password change:

$ smcli lssys -A "CommunicationState,AccessState"
 adminnode_1: Unlocked, 2
 DATA-STANDBY-SN101F54R: Unlocked, 2
 datanode_4: Unlocked, 2
 datanode_5: Unlocked, 2
 ETHERNET0-IBM*8205-E6D*101F52R: Unlocked, 2
 ETHERNET0-IBM*8205-E6D*101F53R: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F0CR: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F0DR: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F54R: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F55R: Unlocked, 2
 fcm_switch1: Unlocked, 2
 fcm_switch2: Unlocked, 2
 mgt_switch1: Unlocked, 2
 mgt_switch2: Unlocked, 2

Example output during the password change:

/opt/ibm/aixappl/pflayer/bin/appl_conf chpw -r net -u admin -p <new_password>
spawn time /opt/ibm/aixappl/pflayer/bin/appl_conf chpw -r net -u admin -p
Enter the new password
Changing the password for user 'admin' on resource 'net0'.
The password change was successful for the user 'admin' on resource 'net0'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Discovery failed for 172.23.1.11
Password change notification to Systems Director failed for nodes 172.23.1.11.
Changing the password for user 'admin' on resource 'net1'.
The password change was successful for the user 'admin' on resource 'net1'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Discovery failed for 172.23.1.12
Password change notification to Systems Director failed for nodes 172.23.1.12.
Changing the password for user 'admin' on resource 'net2'.
The password change was successful for the user 'admin' on resource 'net2'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.
Discovery failed for 172.23.1.21
Password change notification to Systems Director failed for nodes 172.23.1.21.
Changing the password for user 'admin' on resource 'net3'.
The password change was successful for the user 'admin' on resource 'net3'.
Updating the password in the database for user admin.
The password was successfully updated in the database.
Notifying Systems Director of a password change for user admin.

Example lssys command output after the password change:

$  smcli lssys -A "CommunicationState,AccessState"
 adminnode_1: Unlocked, 2
 DATA-STANDBY-SN101F54R: Unlocked, 2
 datanode_4: Unlocked, 2
 datanode_5: Unlocked, 2
 ETHERNET0-IBM*8205-E6D*101F52R: Unlocked, 2
 ETHERNET0-IBM*8205-E6D*101F53R: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F0CR: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F0DR: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F54R: Unlocked, 2
 ETHERNET0-IBM*8231-E2D*101F55R: Unlocked, 2
 fcm_switch1: PartiallyLocked, 2
 fcm_switch2: PartiallyLocked, 2
 mgt_switch1: PartiallyLocked, 2
 mgt_switch2: PartiallyLocked, 2

-----------

To resolve the issue, complete the following steps:

1. Remove the Ethernet network switch from the system by running the following command:

• smcli rmsys -i <net_switch_ip_address>

2. Rediscover the Ethernet network switch on the system by running the following command:

• /opt/ibm/aixappl/pflayer/bin/icmds/appl_ctrl_net discover -l <resource_logical_name>

3. Retry changing the Ethernet network switch password by running the following command:

• appl_conf chpw -l <resource_logical_name> -u <user_name> -p <new_password>

KI006204

Fix pack installation phases show only highest version of Ethernet network switch firmware
 

Fix pack installation phases show only highest version of Ethernet network switch firmware
 

During the preview, prepare, and apply phases of fix pack installation, both the log output and the appl_ls_cat command output show only the highest version of the firmware for the G8264 Ethernet network switch (7.9.12.0), even though the G8052 Ethernet network switch has a different firmware version (7.9.11.0).

Example log output after the preview phase:

Note: This example output can also be viewed in MI preview.

PRODUCT_UPDATES::SSDDrawerFW_PROD::SSDDrawerFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::67G5::67E5::SSDDrawerFW Update
PRODUCT_UPDATES::PFW_PROD::PFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::AL740_156::AL770_048::PFW Update
PRODUCT_UPDATES::PFW_PROD::PFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::AL770_098::AL770_048::PFW Update
PRODUCT_UPDATES::FCAdapterFW_PROD::FCAdapterFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::202307::0315050680,0315050680,0315050680,0315050680,::FCAdapterFW 5273 Update
PRODUCT_UPDATES::FCAdapterFW_PROD::FCAdapterFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::0320051000::0315050680,0315050680,0315050680,0315050680,::FCAdapterFW EN0Y Update
PRODUCT_UPDATES::NetAdapterFW_PROD::NetAdapterFW::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::0400401800007::0310303970033::NetAdapterFW 1648 Update
PRODUCT_UPDATES::GPFS_PROD::GPFS::Management,Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::4.1.0.6::3.4.0.14::GPFS
PRODUCT_UPDATES::CAS_PROD::CAS::Mgmt_Standby,Admin,Admin_Standby,User,Data,Standby::6.3.3.0::6.3.0.3::CAS Update
PRODUCT_UPDATES::StorageFW_PROD::StorageFW::Infrastructure::7.3.0.9::6.4.1.6::StorageFW Update
PRODUCT_UPDATES::SANFW_PROD::SANFW::Infrastructure::v7.2.1d::v7.0.2c::SANFW Update
PRODUCT_UPDATES::NetFW_PROD::NetFW::Infrastructure::7.9.12.0::7.7.3.0::NetFW Update
END::PRODUCT_UPDATES

Example appl_ls_cat command output after the preview phase:

NAME                     VERSION                       STATUS                   OPERATION      DESCRIPTION
netfw1                   7.9.12.0                      Previewed                manage         NetFW Update

However, after the commit phase, both firmware versions are displayed.

Example appl_ls_cat command output after the commit phase:

NAME                     VERSION                       STATUS                   OPERATION      DESCRIPTION
netfw2                   7.9.12.0                      Committed                manage         NetFW Update
netfw3                   7.9.11.0                      Committed                manage         NetFW Update

-----------

A resolution to this restriction is not available at this time.

KI007486

I_V1.1.0.1

I_V1.1.0.2

Flash storage upgrade gets stalled during apply phase which results in the apply failure

Sample output of the failure from the PL log

[06 Mar 2017 07:45:40,711] <4653948 CTRL  DEBUG  ibis01> apply: 172.23.1.182: waiting for upgrade to complete, iteration <1>: update_status=<upgrading 2>
[06 Mar 2017 07:50:41,284] <4653948 CTRL  DEBUG  ibis01> get_update_status: status is <upgrading 2
[06 Mar 2017 07:50:41,284] <4653948 CTRL  DEBUG  ibis01> >
[06 Mar 2017 07:50:41,286] <4653948 CTRL  DEBUG  ibis01> apply: 172.23.1.182: waiting for upgrade to complete, iteration <2>: update_status=<upgrading 2>
[06 Mar 2017 07:55:42,812] <4653948 CTRL  DEBUG  ibis01> get_update_status: status is <stalled 23
[06 Mar 2017 07:55:42,812] <4653948 CTRL  DEBUG  ibis01> >
[06 Mar 2017 07:55:42,813] <4653948 CTRL  DEBUG  ibis01> apply: 172.23.1.182: waiting for upgrade to complete, iteration <3>: update_status=<stalled 23>
[06 Mar 2017 08:00:42,814] <4653948 CTRL  DEBUG  ibis01> apply: 172.23.1.182: broke out of timed wait after 4 iterations of maximum 48. update status is <stalled 23>
[06 Mar 2017 08:00:42,815] <4653948 CTRL  DEBUG  ibis01> Extracted msg from NLS: apply: 172.23.1.182 Error: The update status of the end point is <stalled 23>.
[06 Mar 2017 08:00:42,815] <4653948 CTRL  DEBUG  ibis01> apply: 172.23.1.182: error: error state, update status is <stalled 23>
[06 Mar 2017 08:00:42,853] <7013026 CTRL  DEBUG  ibis01> apply: storage1: apply failed

Verifying the status on the failed node by executing lssoftwareupgradestatus command

$ ssh admin@172.23.1.182
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status  percent_complete
stalled 23

-----------

1. Abort the upgrade using applysoftware -abort command. Wait until the status becomes inactive

IBM_FlashSystem:ibisFlash_00:admin>applysoftware -abort
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status      percent_complete
downgrading 23
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status      percent_complete
downgrading 23
...
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status      percent_complete
downgrading 23
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status      percent_complete
downgrading 23
IBM_FlashSystem:ibisFlash_00:admin>lssoftwareupgradestatus
status   percent_complete
inactive 0 


2. Verify if there are any events for internal errors "Node warmstarted due to an internal error"
This is a known issue with flash storage. One can clear this using cheventlog

IBM_FlashSystem:ibisFlash_00:superuser>lseventlog
sequence_number last_timestamp object_type object_id object_name  copy_id status  fixed event_id error_code description                          secondary_object_type secondary_object_id
103             170209221539   node        2         node2                message no    980349              Node added
114             170301053034   drive       0                              message no    988024              Flash module format complete
115             170301053034   drive       6                              message no    988024              Flash module format complete
...
129             170306074033   cluster               ibisFlash_00         message no    980506              Update prepared
130             170306075323   node        1         node1                alert   no    074002   2030       Internal error                       canister              1
131             170306075401   enclosure   1                              alert   no    085048   2060       Reconditioning of batteries required
...
135             170306075411   cluster               ibisFlash_00         message no    980509              Update stalled
136             170306075411   node        1         node1                alert   no    009100   2010       Update process failed
IBM_FlashSystem:ibisFlash_00:superuser>

Here event 130 has internal error on node with error code 2030.

Detail listing of event:-
IBM_FlashSystem:ibisFlash_00:superuser>lseventlog 130
sequence_number 130
first_timestamp 170306075323
first_timestamp_epoch 1488815603
last_timestamp 170306075323
last_timestamp_epoch 1488815603
object_type node
object_id 1
object_name node1
copy_id
reporting_node_id
reporting_node_name
root_sequence_number
event_count 1
status alert
fixed no
auto_fixed no
notification_type error
event_id 074002
event_id_text Node warmstarted due to an internal error
error_code 2030
error_code_text Internal error
machine_type 9840AE2
serial_number 1351351
FRU None
fixed_timestamp
fixed_timestamp_epoch
callhome_type software
sense1 41 73 73 65 72 74 20 46 69 6C 65 20 2F 62 75 69
sense2 6C 64 2F 74 6D 73 2F 53 56 43 5F 4F 44 45 5F 52
sense3 32 2F 32 30 31 35 2D 30 33 2D 30 39 5F 31 32 2D
sense4 31 34 2D 30 34 2F 72 32 2F 73 72 63 2F 75 73 65
sense5 72 2F 64 72 76 2F 70 61 2F 70 6C 70 61 2E 63 20
sense6 4C 69 6E 65 20 31 35 39 33 00 00 00 00 00 00 00
sense7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
sense8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
secondary_object_type canister
secondary_object_id 1
IBM_FlashSystem:ibisFlash_00:superuser>

Event_id_text shows Node warmstarted due to an internal error

We need to clear the event on the node using cheventlog fix command.

IBM_FlashSystem:ibisFlash_00:admin>cheventlog -fix 130
IBM_FlashSystem:ibisFlash_00:admin>lseventlog
sequence_number last_timestamp object_type object_id object_name  copy_id status  fixed event_id error_code description                          secondary_object_type secondary_object_id
103             170209221539   node        2         node2                message no    980349              Node added
114             170301053034   drive       0                              message no    988024              Flash module format complete
115             170301053034   drive       6                              message no    988024              Flash module format complete
117             170301053039   drive       2                              message no    988024              Flash module format complete
118             170301053039   drive       7                              message no    988024              Flash module format complete
119             170301053039   drive       8                              message no    988024              Flash module format complete
120             170301053044   drive       1                              message no    988024              Flash module format complete
121             170301053044   drive       3                              message no    988024              Flash module format complete
122             170301053044   drive       4                              message no    988024              Flash module format complete
123             170301053044   drive       5                              message no    988024              Flash module format complete
124             170301053044   drive       9                              message no    988024              Flash module format complete
129             170306074033   cluster               ibisFlash_00         message no    980506              Update prepared
131             170306075401   enclosure   1                              alert   no    085048   2060       Reconditioning of batteries required
132             170306075401   enclosure   1                              alert   no    085048   2060       Reconditioning of batteries required
133             170306075406   enclosure   1                              message no    988030              External data link degraded          canister              2
134             170306075406   enclosure   1                              message no    988030              External data link degraded          canister              1
135             170306075411   cluster               ibisFlash_00         message no    980509              Update stalled
137             170306223519   cluster               ibisFlash_00         message no    980510              Update aborted
138             170306224949   node        2         node2                message no    980349              Node added
139             170306224949   cluster               ibisFlash_00         message no    980508              Update Failed
IBM_FlashSystem:ibisFlash_00:admin> 

3. Once we have aborted the update and cleared the event we resume the upgrade

KI007488

I_V1.0.0.5

I_V1.1.0.1

When the non-management Apply is running, during the PFW update, the cec reboots. This brings down the management node and stops the fix pack update.

-----------

To resume the update:

Verify the console is started using the mistatus command run as root on the management host. and then execute miupdate -resume

(0) root @ ibis01: 7.1.0.0: /
$ mistatus
CDTFS000063I The system console is started.
(0) root @ ibis01: 7.1.0.0: /
$ 


The command 'miupdate -resume' will not work. Instead:

Determine the 'bwr#' associated with the fixpack.  Use the appl_ls_cat command, run as root on the management host.

$ appl_ls_cat
NAME                     VERSION                       STATUS                   DESCRIPTION
bwr0                     3.0.3.0                       Committed                Initial images for IBM PureData System for Operational Analytics
bwr1                     4.0.5.0                       Applied                  Updates for IBM_PureData_System_for_Operational_Analytics


Substitute the 'bwr#' found above, in this example it is 4.0.5.0 and run the appl_install_sw commmand as root on the management host.

echo "appl_install_sw -l bwr1 -resume > /tmp/appl_install_sw_$(date +"%Y%m%d_%H%M%S").out 2>&1" | at now

This will run the fixpack application outside of the session (these can be long running commands susceptible to terminal loss). Tail the /tmp/appl_install_sw_<date>.out file to view the progress.

KI007489

I_V1.0.0.5

I_V1.1.0.1

The output excerpt from mi command line or through console GUI:
=====================================================
    Log file:
    Infrastructure Infrastructure SAN switch firmware: SANFW apply started 1 of 1 task completed

    Log file:
    Infrastructure Infrastructure Network switch firmware: NetFW apply started 1 of 1 task completed

    Log file:
    Infrastructure Infrastructure Storage firmware: StorageFW apply started 3 of 3 task completed

    Log file:
    Infrastructure Infrastructure Storage firmware: StorageFW apply started 3 of 3 task completed

========================
"The operation failed during the apply stage. The resume phase for the release 'bwr1' failed..
Refer to the platform layer log file for details."
========================

-----------

Verify that update was completed by running appl_ls_cat command.

In case of resume scenarios during mgmt update status should be M_Applied

(0) root @ ibis01: 7.1.0.0: /BCU_share/aixappl/pflayer/log
$ appl_ls_cat
NAME                     VERSION                       STATUS                   DESCRIPTION
bwr0                     4.0.4.2                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics
bwr1                     4.0.5.0                       M_Applied                  Updates for IBM_PureData_System_for_Operational_Analytics_DB2105

(0) root @ ibis01: 7.1.0.0: /BCU_share/aixappl/pflayer/log

In case of resume scenarios during non management/core update status should be  Applied

(0) root @ ibis01: 7.1.0.0: /BCU_share/aixappl/pflayer/log
$ appl_ls_cat
NAME                     VERSION                       STATUS                   DESCRIPTION
bwr0                     4.0.4.2                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics
bwr1                     4.0.5.0                       Applied                  Updates for IBM_PureData_System_for_Operational_Analytics_DB2105

(0) root @ ibis01: 7.1.0.0: /BCU_share/aixappl/pflayer/log

If the status shows either Applied or M_Applied then proceed for the next step ignoring the failure message

V1.0.0.6/V1.1.0.2 The console is removed as part of these fixpacks and the fixpack methodology has changed.

KI007423

I_V1.0.0.5

I_V1.1.0.1

During the PDOA V1.1 FP5/FP1 management apply phase the fix pack may encounter an error. The pflayer log file may show the following error:

...
[10 Nov 2016 20:07:24,829] <8913318 UPDT  DEBUG  ibis01> Node: 172.23.1.1 Return: 256
[10 Nov 2016 20:07:24,844] <8913318 UPDT  ERROR  ibis01> TASK_END::10::5 of 6::ISW_APPLY::172.23.1.1:: ::RC=1::CDTFS000048E An error occurred while updating InfoSphere Data Warehouse.\n\nDetails:\nThe command \"/BCU_share/bwr1/software/ISW/isw/install.bin -DDS_HA_MODE=TRUE -i silent -f /BCU_share/update_105_tFFF.rsp -Dprofile=BCU_share/bwr1/software/ISW/PDS -Dlog=/tmp/isw_full.log\" failed with the error:\n\n\"\"\n\nUser Response:\nContact IBM Support for assistance.

-----------

There is a known issue with the ISW installer returning a status of 256 back to the caller of the install.bin command line even though the installation was a success. To verify:

1. Login to the management node as root in an ssh session.


2. Look for one of the following directories:

/usr/IBM/dwe/appserver_001/iswapp_10.5/logs
or
/usr/IBM/dwe/appserver_001/iswapp_10/logs


3. Look for a file call 'ISWinstall_summary_<date>.log with a recent date.


4. Run the following:

grep -i status logs/ISWinstall_summary_1701121953.log

This should return a large number of lines with 'Status: SUCCESSFUL'.

If this is the case, it is safe to resume the fix pack as the update was successful.

KI007457

I_V1.0.0.5

I_V1.1.0.1

During the management apply phase, the apply phase may fail with the following symptoms in the logs.

RC=1::Can't locate DBI.pm in @INC (@INC contains: /usr/opt/perl5/lib/5.10.1/aix-thread-multi /usr/opt/perl5/lib/5.10.1 /usr/opt/perl5/lib/site_perl/5.10.1/aix-thread-multi /usr/opt/perl5/lib/site_perl/5.10.1 /usr/opt/perl5/lib/site_perl .) at /BCU_share/bwr1/code/ISAS/Update/Common/OPM/scripts/wait_for_start.pl line 149.\n

or

DBI connect('OPMDB','',...) failed: [IBM][CLI Driver] SQL1031N  The database directory cannot be found on the indicated file system.  SQLSTATE=58031
at /BCU_share/bwr1/code/ISAS/Update/Common/OPM/scripts/wait_for_start.pl line 155

and hals shows that the DPM components are failed over to the standby management host.

-----------

These messages indicate that the during the management apply phase, the DB2 Performance Monitor (DPM) component failed over to the management stand-by node. There are some known issues with DPM on startup that can lead to failures. If the above symptoms are seen then the next steps are to:

1. Use hals to determine if the DPM resources are indeed failed over.

2. Use lssam on the management host to determine if there are any failed states.

3. Use 'resetrsrc' on any DPM resources that are in a failed state.

4. Verify with lssam that the resources are no longer in a failed state.

5. Use 'hafailover <managementstandby> DPM' to move the DPM resources to the management host.

6. Verify that the DPM resources successfully moved to the management host.

7. Resume the Fix Pack

KI007495

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.1

Storage update fails during the apply phase because of drive update failures; one or more storage drives might be in the offline state.

Console output during failure:
========================
The operation failed during the apply stage. Storage update failed on 172.23.1.186.
Refer to the platform layer log file for details.
========================

Sample output of the failure from the PL log:

[08 Mar 2017 04:13:43,315] <28704888 CTRL  DEBUG  finch01> Extracted msg from NLS: apply: 172.23.1.186 ssh admin@172.23.1.186 ssh admin@172.23.1.186 LANG=en_US svctask applydrivesoftware -file  IBM2076_DRIVE_20160923 -type firmware -drive 0:1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47 command failed.
[08 Mar 2017 04:13:43,315] <28704888 CTRL  DEBUG  finch01> apply: 172.23.1.186: error: ssh admin@172.23.1.186 ssh admin@​172.23.1.186 LANG=en_US svctask applydrivesoftware -file  IBM2076_DRIVE_20160923 -type firmware -drive 0:1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47 command failed , rc=127


By using executing the lsdrive command, we can verify the drive statuses on the failed storage box. As an example:

$ ssh superuser@172.23.1.186 "lsdrive"
id status  error_sequence_number use    tech_type capacity mdisk_id mdisk_name member_id enclosure_id slot_id node_id node_name auto_manage
0  online                        member sas_hdd   837.9GB  0        ARRAY3     11        2            1                         inactive  
1  online                        member sas_hdd   837.9GB  0        ARRAY3     10        1            1                         inactive  
2  online                        member sas_hdd   837.9GB  0        ARRAY3     9         2            2                         inactive  
3  online                        member sas_hdd   837.9GB  0        ARRAY3     8         1            10                        inactive  
4  online                        member sas_hdd   837.9GB  0        ARRAY3     7         1            2                         inactive  
5  offline 273                   failed sas_hdd   837.9GB                                2            10                        inactive  
6  online                        member sas_hdd   837.9GB  0        ARRAY3     5         2            9                         inactive  
7  online                        member sas_hdd   837.9GB  0        ARRAY3     4         1            9                         inactive  
8  online                        member sas_hdd   837.9GB  0        ARRAY3     3         2            11                        inactive  
9  online                        member sas_hdd   837.9GB  0        ARRAY3     2         2            8                         inactive  
10 online                        member sas_hdd   837.9GB  0        ARRAY3     1         1            8                         inactive  
11 online                        member sas_hdd   837.9GB  0        ARRAY3     0         1            11                        inactive  
12 online                        member sas_hdd   837.9GB  1        ARRAY4     11        2            7                         inactive  
13 offline 268                   spare  sas_hdd   837.9GB                                1            7                         inactive  
14 online                        member sas_hdd   837.9GB  1        ARRAY4     9         2            6                         inactive  
15 online                        member sas_hdd   837.9GB  1        ARRAY4     8         1            6                         inactive  
16 online                        member sas_hdd   837.9GB  1        ARRAY4     7         1            5                         inactive  
17 online                        member sas_hdd   837.9GB  1        ARRAY4     6         1            12                        inactive

Here we see that 2 drives (drive 5 and drive 13) are in the offline state which is the reason for the failure.

-----------

Run the lsdrive to see the list of failed drives and then do the following:

1. Fix the drives.

2. Ensure the statuses of the drives are all online.

3. Resume the fix pack update.

KI007496

I_V1.0.0.5

I_V1.1.0.1

The upgrade of the CEC that hosts the management and admin nodes is done and the CEC gets rebooted and the run is halted due to this reboot.
When the CEC gets back online and the run is resumed, the BCU_share gets mounted on the management and admin nodes.
Subsequently, the other CECs gets upgraded and rebooted.
But when the respective nodes come back up, the BCU_share does not get mounted again and the upgrade proceeds to the point where it tries to access BCU_share and fails.

Symptoms:
1. The output excerpt from the log
=====================================================
[16 Nov 2016 08:32:35,041] <2884066 ADPT ERROR ibis01> Failed to unpack the adapter firmware file /BCU_share/bwr1/firmware/fc_adapter/df1000f114100104/image/df1000f114100104.
203305.aix.rpm on 172.23.1.4.
=====================================================

2. The /BCU_share NFS mount shared from the management host is not mounted on all hosts.

-----------

After the failure is identified, simply resume. The resume code will verify that /BCU_share is mounted across the hosts.

V1.0.0.6/V1.1.0.2 The fixpack mechanism has completely changed rendering this KI moot.

KI007479

I_V1.0.0.5

I_V1.1.0.1

Running miinfo compliance command shows some levels are not correct. 'miinfo -d -c'

You may see the following under some servers:

IBM Systems Director Common Agent           6.3.3.1                                                                     Higher
IBM InfoSphere Warehouse                    10.5.0.20151104_10.5.0.8..0                                                 Lower
IBM InfoSphere Optim Query Workload Tuner   The version of the product cannot be determined or it is not installed.     NA

-----------

1. IBM Systems Director Common Agent          6.3.3.1                                                                     Higher
The common agent should no longer be tracked by the compliance software. This is a defect in the compliance check program and will not impact the operation of the appliance.

2. IBM InfoSphere Warehouse                   10.5.0.20151104_10.5.0.8..0                                                 Lower
The InfoSphere Warehouse software compliance check uses 20151117 instead of 20151104. This is a defect in the compliance check code and will not impact the operation of the appliance.

3. IBM InfoSphere Optim Query Workload Tuner  The version of the product cannot be determined or it is not installed.     NA
This is normal for nodes that are currently running as standby hosts. The compliance checker has a limitation in that it cannot check the level when a core host is currently a designated standby host.

KI007503

The IBM PureData System for Operational Analytics V1.0 FP5 package cannot be directly applied to FP3 environments. There are three distinct issues with FP5 on FP3 environments.

1. It is possible to register the fixpack, however after registration the console will no longer start. During the preview step similar messages to the following will appear in the log.

• 03 Apr 2017 03:54:54,222] <30474324 UPDT PREV INFO   paradise01> PRODUCT_UPDATES::OPM_PROD::OPM::Management::5.3.1.0.8440::5.3.0.0.7336::OPM
  [03 Apr 2017 03:54:54,481] <30474324 UPDT PREV DEBUG  paradise01> server_logical_names = server6
  [03 Apr 2017 03:55:01,876] <30474324 UPDT  DEBUG  paradise01> Stage getlevel failed:
  [03 Apr 2017 03:55:01,877] <30474324 UPDT  DEBUG  paradise01> Use of uninitialized value in split at /opt/ibm/aixappl/pflayer/lib/ISAS/PlatformLayer/TSA/Topology.pm line 138, <> line 10.
  [03 Apr 2017 03:55:01,878] <30474324 UPDT PREV ERROR  paradise01>  Use of uninitialized value in split at /opt/ibm/aixappl/pflayer/lib/ISAS/PlatformLayer/TSA/Topology.pm line 138, <> line 10.
  [03 Apr 2017 03:55:01,927] <30474324 UPDT PREV DEBUG  paradise01> Executing query Logical_name=bwr1 AND Solution_version=4.0.5.0, to update status of Solution
  [03 Apr 2017 03:55:02,056] <30474324 UPDT PREV INFO   paradise01> PHASE_END PREVIEW
  [03 Apr 2017 03:55:02,058] <30474324 UPDT PREV ERROR  paradise01> The preview phase for the release 'bwr1' failed.

2. It is possible to apply the fixpack via command line, however the fixpack will fail validation as the firmware levels on the V7000 is too low for FP5 to update.


3. It is possible to apply the fixpack via command line, however the fixpack will fail validation as the firmware levels on the SAN switches is too low for FP5 to update.

-----------

See the document 'How to apply the IBM PureData System for Operational Analytics V1.0 FP5 on a FP3 environment?' for more information about the FP3 to FP5 scenario.

KI007523

I_V1.0.0.5

I_V1.1.0.1

The AIX hosts may have failed paths to the external storage. The run the following command as root on the management host:

dsh -n $ALL "lspath | grep hdisk | grep -v Enabled | wc -l"  | dshbak -c

Will return output if there are failed paths to the storage.

-----------

There are two remedy options.

1. Reboot the host with the failed paths. This will effectively bounce the port. This may require an outage.

2. Follow the instructions below to bounce only the port. All access to the storage is fully redundant with multiple paths which is why the system can start even with failed paths. This method avoids an outage and effectively bounces the port.

The following should be done one port at a time and should be performed either in an outage window, or a time when the system will have very low I/O activity.

a. For each host login, determine the ports with failed paths using the 'lspath | grep hdisk | grep -v Enabled  | while read stat disk dev rest;do echo "${dev}";done | sort | uniq' command. This command will return the uniq set of ports connected to hdisk devices that are Missing or Failed.

Example output:
fscsi10

b. For each port, create a script and update the 'export id=1' to match the # in the fscsi# id of the failed path. This script will then remove all paths to that port, set the port to the defined state, and will then rediscover the paths. This effectively bounces the port.

c. Change the id to match the fscsi<id> number. Run the each script to remove the paths and to put the device in defined state, then use cfgmgr to reinitialize. This should create all of the new paths. Run these scripts one at a time and then verify that the path no longer appears in the command shown in a.

export id=1
lspath -p fscsi${id} | while read st hd fs;do echo $hd;done | sort | uniq | while read disk;do rmpath -d -l ${disk} -p fscsi${id};done
rmdev -l sfwcomm${id};rmdev -l fscsi${id};rmdev -l fcs${id}
cfgmgr -s


d. After the commands run, verify that there are no more failed paths over the port and that the port has discovered the existing paths.

Bouncing the port in this way preserves any settings stored in ODM for the fcs and fscsi devices.

KI007537

I_V1.0.0.5

I downloaded and registered the fixpack, however I'm receiving preview errors related to the DB2 and / or InfoSphere Warehouse (ISW) levels.

There are two different fix central downloads for IBM PureData System for Operational Analytics V1.0 fixpack 5.

IBM PureData System for Operational Analytics Fix Pack 5 (for systems with DB2 Version 10.1)

and

IBM PureData System for Operational Analytics Fix Pack 5 (for systems with DB2 Version 10.5)

There are a couple of scenarios where problems arise.

1. Customer has DB2 V10.1, downloads and registers the fixpack with DB2 10.5.
2. Customer has followed the instructions to uplift or upgrade DB2 to 10.5 by following the technote Upgrading an IBM PureData System for Operational Analytics Version 1.0 environment to DB2 10.5 and downloads the fixpack with DB2 10.1.
3. Customer has only partially followed the instructions to uplift or upgrade DB2 to 10.5 by following the technote Upgrading an IBM PureData System for Operational Analytics Version 1.0 environment to DB2 10.5 and downloads the fixpack with DB2 10.5., but encounters a preview errors related to the ISW level being at the incorrect version level.

-----------

1. Customer has DB2 V10.1, downloads and registers the fixpack with DB2 10.5.
2. Customer has followed the instructions to uplift or upgrade DB2 to 10.5 by following the technote Upgrading an IBM PureData System for Operational Analytics Version 1.0 environment to DB2 10.5 and downloads the fixpack with DB2 10.1.

Contact IBM Support for help to de-register the incorrect fixpack.
Download the fixpack with the correct db2 levels.
Follow the fixpack instructions as usually.


3. Customer has only partially followed the instructions to uplift or upgrade DB2 to 10.5 by following the technote Upgrading an IBM PureData System for Operational Analytics Version 1.0 environment to DB2 10.5 and downloads the fixpack with DB2 10.5. but encounters a preview errors related to the ISW level being at the incorrect version level.

This scenario is most likely to due to issues where the technote was not fully followed. This can happen due to confusion about the relationships between the InfoSphere Warehouse and DB2. Most customers understand how to upgrade DB2 and it is easy to miss that it is important to update the InfoSphere Warehouse product as well. Our Fixpack catalog does not at present support mixing DB2 10.5 and InfoSphere Warehouse 10.1 together. So the customer will need to revisit the Upgrading an IBM PureData System for Operational Analytics Version 1.0 environment to DB2 10.5 technote to verify that all of the update steps were followed and that the InfoSphere Warehouse levels are at 10.5 and the WebSphere Application Server levels are at 8.5.5.x as required as part of the technote.

Once the levels are updated per the technote the Fixpack can be resumed and the preview should no longer fail.

KI007538

I_V1.0.0.5

I_V1.1.0.1

I downloaded all of the files included in the FixCentral for the fixpack and there are extra XML files included. What are they for?

The XML files are of the following pattern:

*.fo.xml
*SG*.xml

-----------

These files were inadvertently included in the fixpack packages and should either not be downloaded or deleted.

KI007549

I_V1.0.0.5

I_V1.1.0.1

During the fixpack apply phase the fixpack fails.

----------------------------------------------------------------------------------------
/BCU_share/applmgmt/pflayer/log/pl_update.log:
--> Not much in this log except for the failure message.
----------------------------------------------------------------------------------------

[16 Sep 2017 21:14:05,518] <8126944 UPDT APPI DEBUG mgmthost> STORAGE:storage0:172.23.1.181:1:Storage firmware update failed.


----------------------------------------------------------------------------------------
/BCU_share/applmgmt/pflayer/log/pl_update.trace:
--> This excerpts shows an attempt to update the drive fw fails. The critical message is this one: "CMMVC6055E The action failed as an update is in progress.\n"],<"PLLogger=HASH(0x2000f558)"

----------------------------------------------------------------------------------------
[16 Sep 2017 21:11:38,575] <5832914 CTRL TRACE mgmthost> sleep time is not configured, defaults will be applied
[16 Sep 2017 21:13:38,576] <5832914 CTRL DEBUG mgmthost> apply: 172.23.1.181: now installing drive updates...
[16 Sep 2017 21:13:38,577] <5832914 CTRL DEBUG mgmthost> drive_id: 0:1:2:3:4:6:7:8:9:10:11:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:58:59:60:61:62:64:65:66:67:68:69:70:71
[16 Sep 2017 21:13:38,577] <5832914 CTRL DEBUG mgmthost> Number of drive id's is less than 128
[16 Sep 2017 21:13:38,578] <5832914 CTRL DEBUG mgmthost> Drive update command execution cnt : 0.
[16 Sep 2017 21:13:43,150] <5832914 CTRL TRACE mgmthost> command: ssh admin@172.23.1.181 LANG=en_US svctask applydrivesoftware -file IBM2076_DRIVE_20160923 -type firmware -drive 0:1:2:3:4:6:7:8:9:10:11:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:58:59:60:61:62:64:65:66:67:68:69:70:71
[16 Sep 2017 21:13:43,151] <5832914 CTRL TRACE mgmthost> CMMVC6055E The action failed as an update is in progress.
[16 Sep 2017 21:13:43,151] <5832914 CTRL TRACE mgmthost> Rc = 1
[16 Sep 2017 21:13:43,152] <5832914 CTRL DEBUG mgmthost> Extracted msg from NLS: apply: 172.23.1.181 ssh admin@172.23.1.181 LANG=en_US svctask applydrivesoftware -file IBM2076_DRIVE_20160923 -type firmware -drive 0:1:2:3:4:6:7:8:9:10:11:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:58:59:60:61:62:64:65:66:67:68:69:70:71 command failed.
[16 Sep 2017 21:13:43,153] <5832914 CTRL DEBUG mgmthost> apply: 172.23.1.181: error: ssh admin@172.23.1.181 LANG=en_US svctask applydrivesoftware -file IBM2076_DRIVE_20160923 -type firmware -drive 0:1:2:3:4:6:7:8:9:10:11:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:58:59:60:61:62:64:65:66:67:68:69:70:71 command failed , rc=1
[16 Sep 2017 21:13:43,153] <5832914 CTRL TRACE mgmthost> { Entering Ctrl::Updates::Storage::search_token (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storage.pm line 1127)
[16 Sep 2017 21:13:43,154] <5832914 CTRL TRACE mgmthost> Args:[["CMMVC8325E","None of the specified drives needed to be upgraded or downgraded"],[],<"PLLogger=HASH(0x2000f558)">]
[16 Sep 2017 21:13:43,155] <5832914 CTRL DEBUG mgmthost> Not able to find CMMVC8325E None of the specified drives needed to be upgraded or downgraded in the output, an unexpected error occured
[16 Sep 2017 21:13:43,155] <5832914 CTRL TRACE mgmthost> Return: 0
[16 Sep 2017 21:13:43,156] <5832914 CTRL TRACE mgmthost> Exiting Ctrl::Updates::Storage::search_token }
[16 Sep 2017 21:13:43,156] <5832914 CTRL TRACE mgmthost> { Entering Ctrl::Updates::Storage::search_token (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storage.pm line 1128)
[16 Sep 2017 21:13:43,157] <5832914 CTRL TRACE mgmthost> Args:[["CMMVC8325E","None of the specified drives needed to be upgraded or downgraded"],["CMMVC6055E The action failed as an update is in progress.\n"],<"PLLogger=HASH(0x2000f558)">]
[16 Sep 2017 21:13:43,157] <5832914 CTRL DEBUG mgmthost> Not able to find CMMVC8325E None of the specified drives needed to be upgraded or downgraded in the output, an unexpected error occured
[16 Sep 2017 21:13:43,158] <5832914 CTRL TRACE mgmthost> Return: 0
[16 Sep 2017 21:13:43,158] <5832914 CTRL TRACE mgmthost> Exiting Ctrl::Updates::Storage::search_token }
[16 Sep 2017 21:13:43,158] <5832914 CTRL TRACE mgmthost> { Entering Ctrl::Updates::Storage::search_token (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storage.pm line 1138)
[16 Sep 2017 21:13:43,159] <5832914 CTRL TRACE mgmthost> Args:[["CMMVC6546E","The current drive status is degraded"],[],<"PLLogger=HASH(0x2000f558)">]
[16 Sep 2017 21:13:43,159] <5832914 CTRL DEBUG mgmthost> Not able to find CMMVC6546E The current drive status is degraded in the output, an unexpected error occured
[16 Sep 2017 21:13:43,160] <5832914 CTRL TRACE mgmthost> Return: 0
[16 Sep 2017 21:13:43,160] <5832914 CTRL TRACE mgmthost> Exiting Ctrl::Updates::Storage::search_token }
[16 Sep 2017 21:13:43,160] <5832914 CTRL TRACE mgmthost> { Entering Ctrl::Updates::Storage::search_token (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storage.pm line 1139)
[16 Sep 2017 21:13:43,161] <5832914 CTRL TRACE mgmthost> Args:[["CMMVC6546E","The current drive status is degraded"],["CMMVC6055E The action failed as an update is in progress.\n"],<"PLLogger=HASH(0x2000f558)">]
[16 Sep 2017 21:13:43,161] <5832914 CTRL DEBUG mgmthost> Not able to find CMMVC6546E The current drive status is degraded in the output, an unexpected error occured
[16 Sep 2017 21:13:43,162] <5832914 CTRL TRACE mgmthost> Return: 0
[16 Sep 2017 21:13:43,162] <5832914 CTRL TRACE mgmthost> Exiting Ctrl::Updates::Storage::search_token }
[16 Sep 2017 21:13:43,163] <5832914 CTRL DEBUG mgmthost> Function search_token failed, exiting the loop.
[16 Sep 2017 21:13:43,163] <5832914 CTRL DEBUG mgmthost> Drive update got failed on 172.23.1.181 storage.
[16 Sep 2017 21:13:43,203] <6029570 CTRL DEBUG mgmthost> apply: storage0: apply failed
[16 Sep 2017 21:13:43,204] <6029570 CTRL TRACE mgmthost> For message id::1021
[16 Sep 2017 21:13:58,212] <6029570 CTRL TRACE mgmthost> { Entering Ctrl::Query::Status::read_status_n_details (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Updates/Storage.pm line 1210)
[16 Sep 2017 21:13:58,213] <6029570 CTRL TRACE mgmthost> Args:["172.23.1.181 storage0 storage 0 NA"]
[16 Sep 2017 21:13:58,213] <6029570 CTRL TRACE mgmthost> { Entering Ctrl::Util::util_details (Called from /opt/ibm/aixappl/pflayer/lib/Ctrl/Query/Status.pm line 45)
[16 Sep 2017 21:13:58,214] <6029570 CTRL TRACE mgmthost> Args:["172.23.1.181","storage",0]
....
16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> Command Status:Details:
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> Status: Online
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> AccessState: Unlocked
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> Model: 124
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> IPv4Address: ["172.23.1.181"]
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> Manufacturer: IBM
...
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> FWBuild: 115.54.1610251759000
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> PLLogicalName: storage0
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> MachineType: 2076
[16 Sep 2017 21:14:05,520] <8126944 UPDT APPI DEBUG mgmthost> HostName: V7_00_1
[16 Sep 2017 21:14:05,521] <8126944 UPDT APPI DEBUG mgmthost> FWLevel: 7.5.0.11
[16 Sep 2017 21:14:05,521] <8126944 UPDT APPI DEBUG mgmthost> Description: IBM Storwize V7000 Storage
[16 Sep 2017 21:14:05,521] <8126944 UPDT APPI DEBUG mgmthost> STORAGE:storage0:172.23.1.181:1:Storage firmware update failed.
[16 Sep 2017 21:14:05,521] <8126944 UPDT APPI DEBUG mgmthost> , Command Status->1
[16 Sep 2017 21:14:05,522] <8126944 UPDT ERROR mgmthost> TASK_END::13::1 of 1::StorageUPD::172.23.1.181::::RC=1::Storage update failed on 172.23.1.181
[16 Sep 2017 21:14:05,523] <8126944 UPDT APPI TRACE mgmthost> Return: 0
...
[16 Sep 2017 21:14:05,831] <8126944 UPDT APPI ERROR mgmthost> Error on nodes (172.23.1.181).
[16 Sep 2017 21:14:05,848] <8126944 UPDT APPI INFO mgmthost> STEP_END::13::StorageFW_UPD::FAILED
...
[16 Sep 2017 21:14:06,220] <8126944 UPDT APPI TRACE mgmthost> Exiting /opt/ibm/aixappl/pflayer/lib/ManageCatalogStatus.pm => ODM::change_record }
[16 Sep 2017 21:14:06,224] <8126944 UPDT APPI TRACE mgmthost> Exiting ManageCatalogStatus::update_status }
[16 Sep 2017 21:14:06,225] <8126944 UPDT APPI INFO mgmthost> PHASE_END APPLY IMPACT
[16 Sep 2017 21:14:06,225] <8126944 UPDT APPI TRACE mgmthost> For message id::640
[16 Sep 2017 21:14:06,227] <8126944 UPDT APPI ERROR mgmthost> The apply phase for the release 'bwr5' failed.
[16 Sep 2017 21:14:06,228] <8126944 UPDT RESU INFO mgmthost> PHASE_END RESUME
[16 Sep 2017 21:14:06,228] <8126944 UPDT RESU TRACE mgmthost> For message id::640
[16 Sep 2017 21:14:06,229] <8126944 UPDT RESU ERROR mgmthost> The resume phase for the release 'bwr5' failed.



----------------------------------------------------------------------------------------
lseventlog message
--> this messages shows the last message with the 2050 error code.
----------------------------------------------------------------------------------------
131 170916183340 cluster V7_00_1 alert no 009198 2050 System update completion required


----------------------------------------------------------------------------------------
V7000 lsupdate status
--> Indcates a status of system_completion_required
----------------------------------------------------------------------------------------

ssh -n superuser@172.23.1.181 'lsupdate'
status system_completion_required
event_sequence_number 131
progress
estimated_completion_time
suggested_action complete
system_new_code_level
system_forced no
system_next_node_status none
system_next_node_time
system_next_node_id
system_next_node_name




This is a documented issue when upgrading V7000 firmware from 7.3.x to 7.4.x as indicated in the 7.4.0 release notes.

https://public.dhe.ibm.com/storage/san/sanvc/release_notes/740_releasenotes.html

-----------

Using the PureData System for Operational Analytics Console, use the Service Level Access page to find the link to access the Management Interface for each of the V7000s.

Navigate to the Events page which should show an alert. Select this alert and follow the fix procedures to initiate the second phase.

Do this for each of the V7000s that has this issue. This step will take approximately 40 mins per enclosure and can be run in parallel.


After the second phase is completed. You should see the message 'System update completion finished from lseventlog.

lseventlog -fixed yes

130 170916183340 cluster V7_00_1 message no 980507 Update completed
131 170916183340 cluster V7_00_1 alert yes 009198 2050 System update completion required
132 170917022703 cluster V7_00_1 message no 980511 System update completion started
133 170917022713 node 3 node2 message no 980513 Node restarted for system update completion
134 170917022713 io_grp 0 io_grp0 message no 981102 SAS discovery occurred, configuration changes pending
135 170917022729 io_grp 0 io_grp0 message no 981103 SAS discovery occurred, configuration changes complete
136 170917022818 node 3 node2 message no 980349 Node added
137 170917022818 io_grp 0 io_grp0 message no 981102 SAS discovery occurred, configuration changes pending
138 170917022828 io_grp 0 io_grp0 message no 981103 SAS discovery occurred, configuration changes complete
139 170917025828 node 1 node1 message no 980513 Node restarted for system update completion
140 170917025828 io_grp 0 io_grp0 message no 981102 SAS discovery occurred, configuration changes pending
141 170917025828 io_grp 0 io_grp0 message no 981103 SAS discovery occurred, configuration changes complete
142 170917025939 node 1 node1 message no 980349 Node added
143 170917025941 io_grp 0 io_grp0 message no 981102 SAS discovery occurred, configuration changes pending
144 170917025941 cluster V7_00_1 message no 980512 System update completion finished
145 170917025946 io_grp 0 io_grp0 message no 981103 SAS discovery occurred, configuration changes complete


lsupdate on that host should show 'status' = success.


ssh -n superuser@172.23.1.181 'lsupdate'
status success
event_sequence_number
progress
estimated_completion_time
suggested_action start
system_new_code_level
system_forced no
system_next_node_status none
system_next_node_time
system_next_node_id
system_next_node_name



Once this is completed for all V7000s resume the apply phase.

KI007421

I_V1.0.0.5

I_V1.1.0.1

The fixpack fails with the following message in the pl_update.log file.

[07 Oct 2017 14:51:11,167] <3933062 CTRL  DEBUG  host01> iso file validation failed/not-applicable
[07 Oct 2017 14:51:11,168] <3933062 CTRL  DEBUG  host01>  Updates failed
[07 Oct 2017 14:51:11,188] <3474144 UPDT  ERROR  host01> TASK_END::2::1 of 1::HMCUPD::172.23.1.246::::RC=1::Update failed for HMC
[07 Oct 2017 14:51:11,285] <3474144 UPDT APPI DEBUG  host01> Executing query Logical_name=Management AND Solution_version=4.0.5.0, to update status of Product
[07 Oct 2017 14:51:11,340] <3474144 UPDT APPI DEBUG  host01> Executing query Sub_module_type=Management AND Solution_version=4.0.5.0, to update status of sub module
[07 Oct 2017 14:51:11,570] <3474144 UPDT APPI DEBUG  host01> Executing query Logical_name=Management AND Solution_version=4.0.5.0, to update status of Product
[07 Oct 2017 14:51:11,616] <3474144 UPDT APPI DEBUG  host01> Executing query Sub_module_type=Management AND Solution_version=4.0.5.0, to update status of sub module
[07 Oct 2017 14:51:11,735] <3474144 UPDT APPI ERROR  host01> Error on nodes (172.23.1.246 172.23.1.245).
[07 Oct 2017 14:51:11,752] <3474144 UPDT APPI INFO   host01> STEP_END::2::HMC_UPD::FAILED
[07 Oct 2017 14:51:11,756] <3474144 UPDT APPI DEBUG  host01> Error occured in apply for product hmc1
[07 Oct 2017 14:51:11,806] <3474144 UPDT APPI DEBUG  host01> Executing query Logical_name=hmc1 AND Solution_version=4.0.5.0, to update status of Product
[07 Oct 2017 14:51:11,925] <3474144 UPDT APPI ERROR  host01> Apply (impact) phase for management module has failed.
[07 Oct 2017 14:51:12,011] <3474144 UPDT APPI DEBUG  host01> Executing query Logical_name=bwr1 AND Solution_version=4.0.5.0, to update status of Solution
[07 Oct 2017 14:51:12,129] <3474144 UPDT APPI INFO   host01> PHASE_END APPLY IMPACT
[07 Oct 2017 14:51:12,131] <3474144 UPDT APPI ERROR  host01> The apply phase for the release 'bwr1' failed.
[07 Oct 2017 14:51:12,132] <3474144 UPDT RESU INFO   host01> PHASE_END RESUME
[07 Oct 2017 14:51:12,134] <3474144 UPDT RESU ERROR  host01> The resume phase for the release 'bwr1' failed.



Looking earlier in the log we see the following message:


07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01> Last login: Sat Oct  7 14:41:30 2017 from 172.23.1.1^M
[07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01> ^[[?1034hhscroot@pddrmd7hmc1:~> getupgfiles -h 172.23.1.1 -u root -d /BCU_share/bwr1/firmware/hmc/CR6/image/imports/HMC_Recovery_V8R830_5 -s
[07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01> Enter the current password for user root:
[07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01>
[07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01> The file transfer did not complete sucessfully.
[07 Oct 2017 14:51:07,582] <3998084 CTRL  DEBUG  host01> Verify the remote directory exists, all required files needed for upgrade are there,
[07 Oct 2017 14:51:07,583] <3998084 CTRL  DEBUG  host01> you have read access to both the directory and the files, and then try the operation again.
[07 Oct 2017 14:51:07,583] <3998084 CTRL  DEBUG  host01> hscroot@pddrmd7hmc1:~> echo $?
[07 Oct 2017 14:51:07,583] <3998084 CTRL  DEBUG  host01> 1
[07 Oct 2017 14:51:07,583] <3998084 CTRL  DEBUG  host01> hscroot@pddrmd7hmc1:~>
[07 Oct 2017 14:51:07,585] <3998084 CTRL  DEBUG  host01> From process 4522578: STDERR:
[07 Oct 2017 14:51:07,585] <3998084 CTRL  DEBUG  host01>
[07 Oct 2017 14:51:07,586] <3998084 CTRL  DEBUG  host01> Exit code: 1
[07 Oct 2017 14:51:07,587] <3998084 CTRL  DEBUG  host01> Command return code -> 1
[07 Oct 2017 14:51:07,588] <3998084 CTRL  DEBUG  host01> getupgfiles command failed
[07 Oct 2017 14:51:07,626] <3998084 CTRL  DEBUG  host01> Failed to upgrade release

Workaround:

KI007553

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.1

When the fixpack attempts to restart GPFS on a host, it may fail to start GPFS causing the fixpack process to fail.

Workaround:

KI007499

I_V1.0.0.5

Before starting the apply phases of the fixpack, it is necessary to apply an update to the V7000 drives. These steps can be applied while the system is online. See the linked V7000 tech note for more information.

Product ID ST900MM0006 need to update drives with firmware level B56S before running the "applydrivesoftware" command.

Data Integrity Issue when Drive Detects Unreadable Data
http://www-01.ibm.com/support/docview.wss?rs=591&uid=ssg1S1005289

-----------

1. In an ssh session log as the root user on the management host.

2. Determine the the ip addresses of all of the V7000 enclosures in the environment.

The SAN_FRAME entries in the xcluster.cfg file are V7000 enclosures.

$ grep 'SAN_FRAME[0-9][0-9]*_IP' /pschome/config/xcluster.cfg
SAN_FRAME1_IP    =      172.23.1.181
SAN_FRAME2_IP    =      172.23.1.182
SAN_FRAME3_IP    =      172.23.1.183
SAN_FRAME4_IP    =      172.23.1.184
SAN_FRAME5_IP    =      172.23.1.185
SAN_FRAME6_IP    =      172.23.1.186
SAN_FRAME7_IP    =      172.23.1.187

or

Use the following command to query the console for the storage enclosures.

$ appl_ls_hw -r storage -A M_IP_address,Description
"172.23.1.181","IBM Storwize V7000 Storage"
"172.23.1.182","IBM Storwize V7000 Storage"
"172.23.1.183","IBM Storwize V7000 Storage"
"172.23.1.184","IBM Storwize V7000 Storage"
"172.23.1.185","IBM Storwize V7000 Storage"
"172.23.1.186","IBM Storwize V7000 Storage"
"172.23.1.187","IBM Storwize V7000 Storage"


In the above examples there are several V7000 storage enclosures and the ip addresses are: 172.23.181 to 172.23.1.187

3. Determine if your system has the impacted drive. This command will provide the number of drives that match the 900 GB with type ST900MM0006.

$ grep 'SAN_FRAME[0-9]*[0-9]_IP' /pschome/config/xcluster.cfg | while read a b c d;do echo "*** ${c} ***";ssh -n superuser@${c} 'lsdrive -nohdr| while read id rest;do lsdrive $id;done' | grep -c "product_id ST900MM0006";done

*** 172.23.1.181 ***
0
*** 172.23.1.182 ***
5
*** 172.23.1.183 ***
23
*** 172.23.1.184 ***
0
*** 172.23.1.185 ***
0
*** 172.23.1.186 ***
0
*** 172.23.1.187 ***
0

4 The PureData System for Operational Analytics V1.0 FP5 image includes the necessary files to perform the drive update. These files were unpacked as part of the fixpack registration. Determine the location of the fixpack on the management host.

$ appl_ls_cat
NAME                     VERSION                       STATUS                   DESCRIPTION
bwr0                     4.0.4.0                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics
bwr1                     4.0.5.0                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics_DB2105


In the above command the fixpack files are part of the id 'bwr1'. This means the files were unpacked on the management host in /BCU_share/bwr1.

5. Determine the fix path by changing the <BWR> variable to the identifier determined in step 3 in the path /BCU_share/<BWR>/firmware/storage/2076/image/imports/drives. From the above example, the id was 'bwr1' so the path is "/BCU_share/bwr1/firmware/storage/2076/image/imports/drives".

6. Verify the fix file exists and also the cksum of the file.

$ ls -la /BCU_share/bwr1/firmware/storage/2076/image/imports/drives
total 162728
drwxr-xr-x    2 26976    19768           256 Jan 18 08:53 .
drwxr-xr-x    5 26976    19768           256 Jan 18 08:53 ..
-rw-r--r--    1 26976    19768      83313381 Jan 18 08:53 IBM2076_DRIVE_20160923


$ cksum /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923
3281318949 83313381 /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923


7. For each ip address identified in step 2. The example below uses 172.23.1.183, All V7000s can be updated concurrently.

a. Copy the image to storwize location /home/admin/upgrade

scp /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923 admin@172.23.1.183:/home/admin/upgrade
IBM2076_DRIVE_20160923                  100%   79MB  39.7MB/s   00:02

b. Update the drive using the command below:
ssh admin@172.23.1.183 "applydrivesoftware -file IBM2076_DRIVE_20160923 -all"

7. Monitor the status of drive upgrade using Isdriveupgradeprogress command. This following command like will report on the progress of all of the V7000s. Repeat this command until there is no longer any output indicating the updates have finished.

$ grep 'SAN_FRAME[0-9]*[0-9]_IP' /pschome/config/xcluster.cfg | while read a b c d;do echo "*** ${c} ***";ssh -n superuser@${c} lsdriveprogress;done
*** 172.23.1.181 ***
*** 172.23.1.182 ***
*** 172.23.1.183 ***
*** 172.23.1.184 ***
*** 172.23.1.185 ***
*** 172.23.1.186 ***
*** 172.23.1.187 ***

KI007566

I_V1.0.0.5

I_V1.0.0.6

I_V1.0.0.7

I_V1.1.0.1

I_V1.1.0.2

I_V1.1.0.3

When attempting backup or restore the core TSA domains the hareset command fails with an error similar to the following.

/usr/IBM/analytics/ha_tools/hareset: syntax error at line 854 :`else' unexpected

This is due to an errant edit as part of changes that were incorporated into the hatools in the March fixpacks (V1.0.0.5/V1.1.0.1) as part of HA Tools version 2.0.5.0.

Workaround:

KI007610

This issue only affects customers who apply PDOA V1.0.0.5 (FP5) to V1.0.0.3 (FP3).

There have been two symptoms that have appeared in the field.

The first symptom occurs after trying to run a command to change rsct / TSA policies.

(mkrsrc-api) 2621-309 Command not allowed as daemon does not have a valid license.
mkequ: 2622-009 An unexpected RMC error occurred.The RMC return code was 1.

The second symptom can occur when trying to update TSA when there is no license. The following error can show up when running installSAM:

prereqSAM: All prerequisites for the ITSAMP installation are met on operating system:  AIX 7100-05
installSAM: Cannot upgrade because no valid license was found.
installSAM: No installation was performed.
installSAM: For details, refer to the 'Error:' entries in the log file:  /tmp/installSAM.2.log

Workaround:

KI007570

I_V1.0.0.1
I_V1.0.0.2
I_V1.0.0.3
I_V1.0.0.4
I_V1.0.0.5
I_V1.1.0.1

The PDOA appliance is designed as follows:

1 DB2 9.7 copy on the management host to support IBM System Director.

1 DB2 10.1 or 10.5 DB2 copy on the management and management standby hosts supporting Warehouse Tools and DPM.

1 DB2 10.1, 10.5, 11.1 copy on all core hosts supporting the core database.

This assumption is built into the PDOA Console and can impact the following:

--> compliance checks comparing what is one the system to the validated stack

--> fixpack application (preview, prepare, apply, commit phases).

The most likely scenario is a that a customer who is very familiar with DB2 may install additional copies as part of a fixpack or special build installation. This is supported by DB2 but if the previous copy is left on the system it can cause various issues with the console with the most severe issues occurring during the fixpack application.

This issue will minimally impact customers on V1.0.0.5 or V1.1.0.1 as the non-cumulative V1.0.0.6 (FP6) / V1.1.0.2 (FP2) have significantly changed and no longer have this restriction and the compliance check for DB2 in the platform layer is not a critical function.

Workaround:

KI007499

Before starting the apply phases of the fixpack, it is necessary to apply an update to the V7000 drives. These steps can be applied while the system is online. See the linked V7000 tech note for more information.

Product ID ST1200MM0007 need to update drives with firmware level B57D before running the "applydrivesoftware" command.

Data Integrity Issue when Drive Detects Unreadable Data
http://www-01.ibm.com/support/docview.wss?rs=591&uid=ssg1S1005289

-----------

1. In an ssh session log as the root user on the management host.

2. Determine the the ip addresses of all of the V7000 enclosures in the environment.

The odd numbered SAN_FRAME entries in the xcluster.cfg file are V7000 enclosures.

$ grep 'SAN_FRAME[0-9]*[13579]_IP' /pschome/config/xcluster.cfg
SAN_FRAME1_IP    =      172.23.1.181
SAN_FRAME3_IP    =      172.23.1.183

or

Use the following command to query the console for the storage enclosures.

$ appl_ls_hw -r storage -A M_IP_address,Description
"172.23.1.181","IBM Storwize V7000 FAB-1 Storage"
"172.23.1.182","IBM FlashSystem 900 Storage"
"172.23.1.183","IBM Storwize V7000 FAB-1 Storage"
"172.23.1.184","IBM FlashSystem 900 Storage"


In the above examples there are two V7000 storage enclosures and the ip addresses are: 172.23.181 and 172.23.1.183

3. Determine if your system has the impacted drive. This command will provide the number of drives that match the 1.2 TB drive that matches the

$ grep 'SAN_FRAME[0-9]*[13579]_IP' /pschome/config/xcluster.cfg | while read a b c d;do echo "*** ${c} ***";ssh -n superuser@${c} 'lsdrive -nohdr| while read id rest;do lsdrive $id;done' | grep -c "product_id ST1200MM0007";done
*** 172.23.1.181 ***
27
*** 172.23.1.183 ***
35


4 The PureData System for Operational Analytics V1.1 FP1 image includes the necessary files to perform the drive update. These files were unpacked as part of the fixpack registration. Determine the location of the fixpack on the management host.

$ appl_ls_cat
NAME                     VERSION                       STATUS                   DESCRIPTION
bwr0                     4.0.4.0                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics
bwr1                     4.0.5.0                       Committed                Updates for IBM_PureData_System_for_Operational_Analytics_DB2105


In the above command the fixpack files are part of the id 'bwr1'. This means the files were unpacked on the management host in /BCU_share/bwr1.

5. Determine the fix path by changing the <BWR> variable to the identifier determined in step 3 in the path /BCU_share/<BWR>/firmware/storage/2076/image/imports/drives. From the above example, the id was 'bwr1' so the path is "/BCU_share/bwr1/firmware/storage/2076/image/imports/drives".

6. Verify the fix file exists and also the cksum of the file.

$ ls -la /BCU_share/bwr1/firmware/storage/2076/image/imports/drives
total 162728
drwxr-xr-x    2 26976    19768           256 Jan 18 08:53 .
drwxr-xr-x    5 26976    19768           256 Jan 18 08:53 ..
-rw-r--r--    1 26976    19768      83313381 Jan 18 08:53 IBM2076_DRIVE_20160923


$ cksum /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923
3281318949 83313381 /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923


7. For each ip address identified in step 2. The example below uses 172.23.1.183, All V7000s can be updated concurrently.

a. Copy the image to storwize location /home/admin/upgrade

scp /BCU_share/bwr1/firmware/storage/2076/image/imports/drives/IBM2076_DRIVE_20160923 admin@172.23.1.183:/home/admin/upgrade
IBM2076_DRIVE_20160923                  100%   79MB  39.7MB/s   00:02

b. Update the drive using the command below:
ssh admin@172.23.1.183 "applydrivesoftware -file IBM2076_DRIVE_20160923 -all"

7. Monitor the status of drive upgrade using Isdriveupgradeprogress command. This following command like will report on the progress of all of the V7000s. Repeat this command until there is no longer any output indicating the updates have finished.

$ grep 'SAN_FRAME[0-9]*[13579]_IP' /pschome/config/xcluster.cfg | while read a b c d;do echo "*** ${c} ***";ssh -n superuser@${c} lsdriveprogress;done
*** 172.23.1.181 ***
*** 172.23.1.183 ***

KI007470

ymptoms:
1. Fix pack apply fails

2. The miupdate log shows the following:
=====================================================
[20 Jan 2017 04:09:59,225] <6291746 CTRL TRACE reverseflash01> CMMVC5994E Error in verifying the signature of the update package.
and
20 Jan 2017 04:09:59,230] <6291746 CTRL ERROR reverseflash01> STORAGE:storage1:172.23.1.182:1:Error: The update operation on the system cannot be performed.
[20 Jan 2017 04:09:59,230] <6291746 CTRL ERROR reverseflash01> STORAGE:storage3:172.23.1.184:1:Error: The update operation on the system cannot be performed.
=====================================================

This is a known issue with the Flash900 firmware as listed in the following URL:  https://www-01.ibm.com/support/docview.wss?uid=ssg1S1009254

Workaround:

KI007523

The upgrade cannot proceed because of hardware errors.

LOG excerpts:
============

[02 Feb 2017 13:43:34,301] <3801392 UPDT ERROR reverseflash01> TASK_END::14::1 of 1::StorageUPD::172.23.1.182::::RC=1::Storage update failed on 172.23.1.182
[02 Feb 2017 13:43:34,302] <3801392 UPDT INFO reverseflash01> TASK_END::14::1 of 1::StorageUPD::172.23.1.184::::RC=0::Storage update succeeded on 172.23.1.184
[02 Feb 2017 13:43:34,497] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Logical_name=Infrastructure AND Solution_version=4.0.5.0, to update status of Product
[02 Feb 2017 13:43:34,561] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Sub_module_type=Infrastructure AND Solution_version=4.0.5.0, to update status of sub module
[02 Feb 2017 13:43:34,771] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Logical_name=Infrastructure AND Solution_version=4.0.5.0, to update status of Product
[02 Feb 2017 13:43:34,818] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Sub_module_type=Infrastructure AND Solution_version=4.0.5.0, to update status of sub module
[02 Feb 2017 13:43:34,925] <3801392 UPDT APPI ERROR reverseflash01> Error on nodes (172.23.1.182).
[02 Feb 2017 13:43:34,944] <3801392 UPDT APPI INFO reverseflash01> STEP_END::14::StorageFW_UPD::FAILED
[02 Feb 2017 13:43:34,950] <3801392 UPDT APPI DEBUG reverseflash01> Error occured in apply for product storagefw3
[02 Feb 2017 13:43:35,013] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Logical_name=storagefw3 AND Solution_version=4.0.5.0, to update status of Product
[02 Feb 2017 13:43:35,154] <3801392 UPDT APPI ERROR reverseflash01> Apply (impact) phase for solution has failed.
[02 Feb 2017 13:43:35,224] <3801392 UPDT APPI DEBUG reverseflash01> Executing query Logical_name=bwr1 AND Solution_version=4.0.5.0, to update status of Solution
[02 Feb 2017 13:43:35,349] <3801392 UPDT APPI INFO reverseflash01> PHASE_END APPLY IMPACT
[02 Feb 2017 13:43:35,351] <3801392 UPDT APPI ERROR reverseflash01> The apply phase for the release 'bwr1' failed.
[02 Feb 2017 13:43:35,352] <3801392 UPDT RESU INFO reverseflash01> PHASE_END RESUME
[02 Feb 2017 13:43:35,353] <3801392 UPDT RESU ERROR reverseflash01> The resume phase for the release 'bwr1' failed.


[02 Feb 2017 11:42:10,533] <5112002 CTRL DEBUG reverseflash01> Extracted msg from NLS: apply: 172.23.1.182 ssh admin@172.23.1.182 LANG=en_US svctask applysoftware -file IBM9840_INSTALL_1.4.5.0 c
ommand failed.
[02 Feb 2017 11:42:10,534] <5112002 CTRL DEBUG reverseflash01> apply: 172.23.1.182: error: svctask applysoftware failed, rc=1
[02 Feb 2017 11:42:10,568] <2097734 CTRL DEBUG reverseflash01> apply: storage1: apply failed
[02 Feb 2017 11:46:48,787] <4259928 CTRL DEBUG reverseflash01> get_update_status: status is <upgrading 2
[02 Feb 2017 11:46:48,787] <4259928 CTRL DEBUG reverseflash01> >

Workaround:

KI007539

When downloading the files for V1.1 FP1, I noticed that the filenames say 1.0.0.5. Did I download the right files? Is there a mistake in FixCentral?

Workaround:

KI007560

The above drive type was added as a possible drive for V7000 gen2 enclosures after the PDOA Fixpack was released and is not recognized by the V7000 test application shipped in that fixpack. This can lead to errors similar to the following during the preview stage.

[20 Dec 2017 09:34:43,469] <12386730 CTRL  TRACE  host01>  This tool has detected that the cluster contains
[20 Dec 2017 09:34:43,470] <12386730 CTRL  TRACE  host01>  one or more internal disks that are not known drive types.
[20 Dec 2017 09:34:43,470] <12386730 CTRL  TRACE  host01>  Please retry with the latest version of svcupgradetest. If this
[20 Dec 2017 09:34:43,470] <12386730 CTRL  TRACE  host01>  error is still being reported, please contact your support representative.
[20 Dec 2017 09:34:43,470] <12386730 CTRL  TRACE  host01>  +----------------------+-------------+
[20 Dec 2017 09:34:43,470] <12386730 CTRL  TRACE  host01>  | Reported model       | Drive count |
[20 Dec 2017 09:34:43,471] <12386730 CTRL  TRACE  host01>  +----------------------+-------------+
[20 Dec 2017 09:34:43,471] <12386730 CTRL  TRACE  host01>  | AL14SEB120N          | 32          |
[20 Dec 2017 09:34:43,471] <12386730 CTRL  TRACE  host01>  +----------------------+-------------+
[20 Dec 2017 09:34:43,471] <12386730 CTRL  TRACE  host01>  To see the list of entries in any of the above tables
[20 Dec 2017 09:34:43,471] <12386730 CTRL  TRACE  host01>  re-run the tool with a -d parameter added on the end.
[20 Dec 2017 09:34:43,472] <12386730 CTRL  TRACE  host01>

This will cause the preview step to fail.


No PDOA environments were shipped with this drive type prior to December, 2017. However, it is possible that a drive replacement could introduce this drive type to an existing PDOA V1.1 environment leading to this scenario.

Workaround: