ESS known issues

After completing the upgrade to 6.1.9.2 on ESS 3500 systems, essinstallcheck might incorrectly claim the wrong current versions of BIOS and BMC.

This is a false positive and can be ignored.

During a disk repair sequence by using GUI takes an extended period of time, the window might time out and exit the repair.

essrun --add-node or --add-ems

• Review quorum rules. For more information, see https://www.ibm.com/docs/en/storagescale/5.1.9?topic=failure-selecting-quorumnodes.
• Determine the quorum configuration.

  mmlscluster and mmlscluster -s

• Change selected nodes to non-quorum.

  mmchnode -N <io node or cluster node> --noquorum

• Add the new nodes.

  essrun --add-node or --add-ems

• Verify the configuration by using the mmlscluster and mmlscluster -s commands.

During a disk repair on the IBM Storage Scale System 6000 node, the amber LED might not illuminate, which makes it difficult to locate the drive.

esscallhomeconf fails on the P9 EMS in a co-existence environment.

In coexistence environments, the call home for IBM Storage Scale System 6000 will be from the Utility Node EMS only at this time.

Utility Node EMS exclusively manages call home events for IBM Storage Scale System 6000 in coexistence environments until further notice.

Unified call home enablement might not be consistent on the Utility Protocol node.

After updating to ESS 6.1.9.1, you might encounter the call home degraded error:

‘callhome_ptfupdates_failed’ when running ‘mmhealth node show’

After updating to ESS 6.1.9.1, in the GUI when navigating to the Storage section, and then to Declustered Arrays you might encounter an error. The performance collector did not return any data.

After updating to ESS 6.1.9.1, when you are checking pmsensors and pmcollector, you might encounter an error, such as the hostname is not set.

This is the error occurs in this case:

[root@ess3500 ~]# systemctl status pmsensors

● pmsensors.service - zimon sensor daemon
Loaded: loaded
(/usr/lib/systemd/system/pmsensors.service; enabled; vendor preset: disabled)
Active: active (running) since Fri
2023-12-22 09:30:08 MST; 14min ago
Process: 1272577 ExecStartPre=/bin/sh -c if [ -z "ess3500rw6a.test.net" -o "ess3500rw6a.test.net" = "localhost*" ] ;
then /usr/bin/logger "pmsensors: ERROR:
hostname is not set <>

This error is a false positive and can be ignored.

After host-side update of a Utility node, the VM doe not start after you reboot it. An error occurs when shared libraries are loading.

librdmacm.so.1: cannot open shared object file: No such file or directory

For ESS Utility Node protocol node, the mmhealth command does not monitor HAL.

The errors shown are intermittent and recoverable. They are not suppressed by the system yet can be safely ignored.

No action required.

GUI configuration: You are asked to create a second user for GUI on Utility Node even though one user is created from the Power9 EMS node.

Configure a second user with a new name. (co-existence).

While configuring the GUI in the Utility Node, you are asked to create a GUI user. Ensure that the user does not exist. GUI users are shared across, GUIs which means the user can login with an existent user in to the new GUI. For more information about GUI configuration, see Utility Node co-existence with Power9 EMS.

All events of pair canister are not surfacing.

Other events will point you to the problem.

No known issue

An enclosure and fan LEDs are not turning on.

The IBM Storage Scale System 6000 LEDs might not turn on as expected. The s6k_ledmgmt_v03.sh script in the samples directory might be used to control the LEDS.

No known issue

The selinux enable command fails on IBM Storage Scale System 6000 nodes.

Due to changes in Red Hat 9.x, deployment is currently restricts enablement of SELinux by using the essrun or manually in an ESS environment. This restriction applies only to RH9.x ESS nodes Utility (EMS/Protocol) and IBM Storage Scale System 6000 only.

This restriction will be removed in an upcoming release.

Drive fault LED (amber LED) might not light during replacement of the drive.

Determine pdisk physical slot location of the drive, which is being replaced by using the mmvdisk pdisk list command.

mmvdisk pdisk list --rg all --replace

recovery group pdisk priority
FRU (type) location
-------------- ------------ -------- -
-------------- --------
rg1 e1s24 9.83
3.84TB NVMe Tie Rack yyyyy U13-16,
Enclosure 5149-F48-78L00XX Drive 24

In this example physical slot locations of pdisks e1s24 being replaced is in slot 24 of enclosure 78L00XX.

When you are creating an ESS Utility Node Protocol VM in an ESS Utility Node system by using essrun command, you might encounter a shared connection failure.

essrun -N <node> cesvm --create --vm-name <vm name>

1. Run the following sed command in the ESS Utility Node EMS container, and then run the essrun command:

   sed -i '/fatal_error = False/a\    small_setup = False' /opt/ibm/ess/deploy/ansible/roles/kvmCreation/files/hostCheck

2. Ensure that four spaces between "a\" and "small_setup".

The GUI setup wizard might fail due to an empty drop-down selection.

During the GUI setup, a wizard portion of a newly installed system with an ESS Utility Node EMS you may encounter an empty drop-down box for the Management Server. If seen, you will not be able to continue the wizard.

1. Exit the GUI.
2. Access the command line on the ESS Utility Node EMS and run the following command:

   mmaddcompspec default --replace

   
3. Re-enter the GUI and complete the wizard setup.

When creating the ESS Utility Node Protocol VM you might encounter the following error:

[ERROR] Failed to create a bridge named on node localhost

When attempting setup ESS Utility Node Protocol services by using the IBM Storage Scale toolkit you might see the following error:

 [ INFO ] ERROR! couldn't resolve module/action 'selinux'. This often indicates a misspelling, missing collection, or incorrect module path. [ INFO ]
 The error appears to be in '/usr/lpp/mmfs/5.1.9.1/ansible-toolkit/ansible/collections/ansible_collections/ibm/spectrum_scale/roles/core_prepare/tasks/prepare.yml':
 line 3, column 3, but may [ INFO ] be elsewhere in the file 

1. Install Ansible-collection that is part of the Ansible galaxy suite.
2. On the EMSVM run the following command:

   scp ansible-collections.tar.gz utilityVM:/root/.
    tar -xvf ansible-collections.tar.gz
    ansible-galaxy collection list

After deploying ESS Utility Node Protocol VM, you are instructed to set the IP address of the management bridge by using ess_ssr_setup script. You might encounter the following issue while executing this command:

 root@esstest-utilitynode2-essvm ~]# ess_ssr_setup 2023-12-07 20:56:28,429
INFO: This is version 122 2023-12-07 20:56:28,451 ERROR: Cannot parse lscpu output Exception happened: 1

The firmware update command in ESS 3500 Hybrid or Capacity models might fail with the following error:

‘diagnostic results: pass-through os error: Cannot allocate memory.’

ERROR:   We cannot connect or check of EMSVM-23E because it is not created

If this happens you can start your VM by using the standard virsh commands from the ESS Utility Node host.

When updating the ESS Utility Node EMS host by using the essrun command, you might encounter a failure when connecting from the EMS VM to the EMS host.

This failure is due to incorrect settings in the connections between the EMS host and the EMS VM.

When accessing the Dual-Pane GUI statistics page you might be unable to display data on the second pane.

If this encountered, you will see Na:Na:Na at the bottom left portion of the second pane.

To fix this, clear the cache of your browser, and re-enter the GUI.

The deployment or restarting of the container will fail, if SELinux is enabled in the EMS.

If SELinux enabled in EMS, the container will fail with an error as follows:

Error: failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/8d4329e232d61495d37ad0b818d74a29262465fc5a3df1323e048ecd182ce806/userdata/shm": invalid argument 

The firmware update for ESS 3000 might fail, if it is coming from an N-2 level.

When running mmchfirmware --type storage-enclosure, on ESS 3000 you might see this error:

BIOS: Fail! Required uas driver module failed to load

insmod /lib/modules/$(uname -r)/kernel/drivers/usb/storage/usb-storage.ko.xz

You might encounter false positive 4U102 5 volt regulator failures in the mmlsenclosure and mmhealth node event log.

In the mmhealth node event log:

power_supply_failed WARNING Power supply 5vra_id2 is FAILED.
power_supply_failed WARNING Power supply 5vra_id2 is FAILED.

If this is seen contact support to verify false positive condition.

For the ESS 5000 4u106 enclosures, it might take longer to gather firmware information than in previous releases.

Due to changes in tools the mmlsfirmware command may take longer than expected when gathering information.

In the Hardware panel of a system using a utility-node EMS, a node might show up as Unknown.

1. Run the mmsysmoncontrol restart command on the nodes showing up as Unknown.
2. Run the mmhealth node show --refresh command on all nodes.
3. Check hardware states in the GUI.

In a utility node by using Infiniband (IB) high-speed networking, the typical use of essgennetworks to create the network bond will fail.

For IB systems you must use other means to create the network bonds on the Utility-Node EMS. The following tools will work: nmtui, nmcli.

After updating to 6.1.8.1, essinstallcheck may show an error saying that mmvdisk settings do not match to best practices.

1. To resolve this issue, you need issue following command to remove these 4 configuration:

   mmchconfig
   nsdRAIDDiskPerformanceMinLimitPct=DELETE,nsdRAIDDi
   skPerformanceUpdateInterval=DELETE,nsdRAIDEventLogTo
   Console=DELETE,nsdRAIDSSDPerformanceMinLimitPct=DELETE

2. Issue server configure --update again:

   mmvdisk server configure --update --node-class <mmvdisknode class> --recycle one

When updating enclosure firmware on a daisy-chained ESS 3500 with 5 or more 4U102 enclosures, you must update each enclosure individually by using the serial number instead of the normal parallel method.

[ERROR] Expected quantity of NVME model: Micron_7450_MTFDKBA960TFR is 2, but only found: 1
[ERROR] Expected quantity of NVME model: Micron_7300_MTFDHBA960TDF is 2, but only found: 1

You will be asked to enter a ‘resolvable’ container short name in the BYOE environment. However, this fails if it is resolvable in the /etc/hosts file. In the BYOE environment, you must remove the resolvable name from the /etc/hosts file or use a different name that is not resolvable for the command to complete successfully.

An example of failure:

EMS hostname must be the Management network (also called Xcat). Other networks can be aliases (A) or canonical names (CNAME) on DNS and/or on hosts file. 
Is the current EMS FQDN tucbyoe1vm.test.net correct? (y/n):y 
Remember NOT to add the domain name test.net to the input Please type the desired and resolvable container short hostname [changeme] : cemsbyo
2023-06-14 21:23:20,845 ERROR: The container short name cemsbyo resolves to IP address 10.88.0.5 
2023-06-14 21:23:20,845 ERROR: The container long name cemsbyo.test.net resolves to IP address 10.88.0.5 2023-06-14 21:23:20,845 ERROR: Container name can be resolved. This is not supported in this EMS. Do not add the container to /etc/hosts nor DNS, and try again. Or use a different container name that cannot be resolved.

Using this to develop a known issue for serializing mmchfirmware updates in a daisy-chain system.

When updating enclosure firmware on a daisy-chained ESS 3500 with five or more 4u102 enclosures, you must update each enclosure individually by using a serial number instead of the normal parallel method.

Example:

[root@ems9 ~]# mmlsenclosure all -N ess3500rw6a-hs
 needs
 serial number product id firmware level service nodes
 ------------- ---------- -------------- ------- ------
 78E400Q 5141-FN2 E11Q yes ess3500rw6a-hs.test.net
 78T2468 5147-102 4E29,4E29 no ess3500rw6a-hs.test.net
 78T246A 5147-102 4E2A,4E29 no ess3500rw6a-hs.test.net
 78T246C 5147-102 4E29,4E29 no ess3500rw6a-hs.test.net
 78T246E 5147-102 4E29,4E29 no ess3500rw6a-hs.test.net
 78T254a 5147-102 4E29,4E29 no ess3500rw6a-hs.test.net
[root@ems9 ~]#

mmchfirmware --type storage-enclosure --serial-number <serial number>

Enablement of IPv6 for RoCE by using Service Network includes problematic entry in /etc/sysconfig/networkscripts/ifcfg-bond-bond0.

1. Create a bond by using essgennetworks utility.
2. Remove the ‘comment out’ line from the /etc/sysconf/network-scripts/ifcfg-bond-bond0 file of the node.
3. Run nmcli con reload.
4. Run nmcli con down bond-bond0; nmcli con up bond-bond0.
5. Run all other essgennetworks commands to enable RoCE as per procedure.
   Note: With this workaround the last RoCE enablement command should work as designed.
   Example:

   essgennetworks -N essio51 --suffix=-ce --interface
   enP48p1s0f0,enP48p1s0f1 --bond bond0 --enableRoCE --mtu
   9000
   [INFO] Starting network generation...
   [INFO] nodelist: essio51
   [INFO] suffix used for network hostname:-ce
   [INFO] Interface(s) available on node essio51-ce
   [INFO] Considered interface(s) of node essio51-ce are
   ['enP48p1s0f0', 'enP48p1s0f1', 'bond0'] with RDMA Port
   ['mlx5_2', 'mlx5_3', 'mlx5_bond_0'] for this operation
   [INFO] Supported Mellanox RoCE card found at node
   essio51
   [INFO] Supported version of Mellanox OFED found at node
   essio51-ce
   [INFO] Bond validation passed and found bonds bond0 has
   been created using same physical network adapter at node
   essio51-ce
   [INFO] Bond MTU validation passed and found bonds MTU
   set to 9000 at node essio51-ce
   [INFO] Interface bond0 have the IPv4 Address assigned at
   node essio51-ce
   [INFO] Interface bond0 have the IPv6 Address assigned at
   node essio51-ce
   [INFO] Interface MTU also set to 9000 at node essio51-ce
   [INFO] Interface enP48p1s0f0 have the IPv4 Address
   assigned at node essio51-ce
   [INFO] Interface enP48p1s0f1 have the IPv4 Address
   ESS3500
   5147-102
   ESS3200
   ESS5000
   Page 2 of 9
   assigned at node essio51-ce
   [INFO] Interface enP48p1s0f0 have the IPv6 Address
   assigned at node essio51-ce
   [INFO] Interface enP48p1s0f1 have the IPv6 Address
   assigned at node essio51-ce
   [INFO] Enabling RDMA for Ports ['mlx5_bond_0', 'mlx5_2',
   'mlx5_3']
   [INFO] Enabled RDMA i.e. RoCE using bond bond0
   [INFO] Enabled RDMA i.e. RoCE using interfaces
   enP48p1s0f0,enP48p1s0f1
   [INFO] Please recycle the GPFS daemon on those nodes
   where RoCE has been enabled.

• During GUI wizard setup or during GUI edit components, you will be prompted twice for specifying rack location of the backup EMS. Do the following steps to resolve this issue:
  1. During the first prompt for the rack location of the management servers, leave the location as black for the backup EMS.
  2. Click Next to go to the panel specifying the location for other nodes and choose the rack location for the backup EMS in this panel.
• During the GUI wizard setup or during GUI edit components, when actions are run via the GUI after users specify the rack locations, the GUI action could fail due to the backup EMS being added twice into the component database. Do the following steps to resolve this issue:
  1. Ignore the error and select click Close button to close the window.

  2. Click Finish to continue.

1. Monitor the mmfs.log.latest file.
2. If any suspicious errors are printed to indicating disk paths and/or IOM are missed, then run mmgetpdisktopology and pass the output to topsummary to find out which path(s) are missed.

1. Run the mmlscomp command from EMS to find the component ID for the backup EMS.
2. Run the mmdelcomp <componend_id> command (component_id is the component ID obtained from above step 1.)
3. Return to the GUI Hardware panel and click Edit Rack Components, which is on left-top of the server list on the right side.

4. Choose Yes, discover new servers and enclosures first. This takes many minutes.

5. Click Next to follow the screen prompt to complete Hardware Components setup.

• If SUDO is already configured before deploying esscallhomeconf, then the workaround is to:
  1. Disable SUDO (in relevant nodes where it is enabled).
  2. Deploy configuration of Callhome (in a default root mode).
• If SUDO is not enabled yet, then the recommendation is to:
  1. Deploy Callhome first (Can be achieved through GUI setup or manually via esscallhomeconf. For more information, see ESS documentation.
  2. Enable SUDO on the cluster node(s).

• ESS 5000

1. Create bond using essgennetworks utility.
2. Remove the ‘comment out’ line in the /etc/sysconf/network-scripts/ifcfg-bond-bond0 file of the spcific node.
3. Run nmcli con reload.
4. Run nmcli con down bond-bond0; nmcli con up bondbond0.
5. Run all other essgennetworks commands to enable RoCE as per procedure.
   Note: With this workaround the last RoCE enablement command should work as designed.
   Example:

   essgennetworks -N essio51 --suffix=-ce --interface
   enP48p1s0f0,enP48p1s0f1 --bond bond0 --enableRoCE
   --mtu 9000
   [INFO] Starting network generation...
   [INFO] nodelist: essio51
   [INFO] suffix used for network hostname:-ce
   [INFO] Interface(s) available on node essio51-ce
   [INFO] Considered interface(s) of node essio51-ce are
   ['enP48p1s0f0', 'enP48p1s0f1', 'bond0'] with RDMA Port
   ['mlx5_2', 'mlx5_3', 'mlx5_bond_0'] for this operation
   [INFO] Supported Mellanox RoCE card found at node
   essio51
   [INFO] Supported version of Mellanox OFED found at
   node essio51-ce
   [INFO] Bond validation passed and found bonds bond0
   has been created using same physical network adapter
   at node essio51-ce
   [INFO] Bond MTU validation passed and found bonds
   MTU set to 9000 at node essio51-ce
   [INFO] Interface bond0 have the IPv4 Address
   assigned at node essio51-ce
   [INFO] Interface bond0 have the IPv6 Address
   assigned at node essio51-ce
   [INFO] Interface MTU also set to 9000 at node essio51-
   ce
   [INFO] Interface enP48p1s0f0 have the IPv4 Address
   assigned at node essio51-ce
   [INFO] Interface enP48p1s0f1 have the IPv4 Address
   assigned at node essio51-ce
   ESS5000
   Page 5 of 9
   [INFO] Interface enP48p1s0f0 have the IPv6 Address
   assigned at node essio51-ce
   [INFO] Interface enP48p1s0f1 have the IPv6 Address
   assigned at node essio51-ce
   [INFO] Enabling RDMA for Ports ['mlx5_bond_0',
   'mlx5_2','mlx5_3']
   [INFO] Enabled RDMA i.e. RoCE using bond bond0
   [INFO] Enabled RDMA i.e. RoCE using interfaces
   enP48p1s0f0,enP48p1s0f1
   [INFO] Please recycle the GPFS daemon on those
   nodes where RoCE has been enabled.

• Issue the mmvdisk sed enroll command after creating a recovery group and before creating user vdisk sets.
• Contact IBM Support, if you issued the mmvdisk sed enroll command after creating user vdisk sets.

# essfindmissingdisks
[INFO]  Start find missing disk paths
[INFO]  nodelist: localhost
[INFO]  May take long time to complete search of all drive paths
[INFO]  Checking node:  localhost
[INFO]  Checking missing disk paths from node localhost
[INFO]  GNR server: name ess3500a-hs.test.net arch x86_64 model ESS3500-5141-FN2 serial 78E400XA
[INFO]  Enclosure 78E40XA sees 22 disks (22 SSDs, 0 HDDs)
[INFO]  Enclosure 78T254A sees 102 disks (0 SSDs, 102 HDDs)
[INFO]  Enclosure 78T246A sees 102 disks (0 SSDs, 102 HDDs)
[ERROR] GNR server disk topology: ESS 3500 H2 (2 HBA 24 NVMe 2 Full 4U102) (match: 96/100)
[INFO]  GNR configuration: 3 enclosures, 22 SSDs, 2 empty slots, 220 disks total, 0 NVRAM partitions
[ERROR] Location 78E40XA-1 appears empty but should have an SSD
[ERROR] Location 78E40XA-2 appears empty but should have an SSD
[[ERROR] essfindmissingdisks detected error in system. Please review output carefully.
[root@ess3500a ~]#

This error indicates that one path to the drive might be in a ‘stuck-state’ condition whereas other path is healthy.

The ESS system reboots unexpected because mpt3sas messages fill logs.

System crashed with 'swiotlb buffer is full' then 'scsi_dma_map failed' errors

.44 mpt3sas driver will be available in ESS 6.1.8.

Customer may encounter false positive intermittent fan module failures in /var/log/messages. It is also possible that a call home will be generated for each fan module.

mmsysmon[7819]: [W] Event raised: Fan
fan_module1_id4 has a fault.
mmsysmon[7819]: [W] Event raised: Fan
fan_module1_id4 state is FAILED.

If this is seen contact IBM support to verify false positive condition.

In ESS 5000, if using Y-Type cables (used in HDR switches), for running high-speed network, the mmhealth node show -N ioNode1-ib,ioNode2-ib NETWORK might show ib_rdma_port_width_low(mlx5_0/1, mlx5_1/1, mlx5_4/1).

1. Check existing anomaly in HDR-Y cables.
2. Contact IBM Support for help to update /usr/lpp/mmfs /lib/mmsysmon/NetworkService.py and /usr/lpp/mmfs/lib/mmsysmon/network.json with appropriate code.
3. After patching, restart mmsysmon to apply changes. Example: systemctl restart mmsysmon.

4. Issue the mmhealth command to verify whether the condition is alleviated.

1. Reboot EMS.
2. Restart the container.
3. Ensure that selinux-policy is up-to-date by issuing the yum update selinux-policy command.
4. Reinstall pcp-selinux by issuing the yum reinstall pcp-selinux command.

mmaddcallback: Callback identifier "MIGRATION" already exists or was specified multiple times.

• ESS 3000
• ESS 3200
• ESS 5000
• ESS 3500
• ESS 3500 (4u102)

Delete the callback and create the file system again.