mmvdisk server command

Manages mmvdisk recovery group servers for IBM Spectrum Scale RAID.

Synopsis

mmvdisk server configure {--node-class NcName | -N Node[,Node...]}  
                         [--maxblocksize {2M | 4M | 8M | 16M}]
                         [--pagepool {nM |nG | n%}]
                         [--custom-config Attribute[=value][,Attribute[=value]...]]
                         [--update]
                         [--recycle {none |one |all | Number}]
or
mmvdisk server configure --node-class NcName --verify [-Y]
or
mmvdisk server unconfigure {--node-class NcName | -N Node[,Node...]}
                           [--recycle {none | one | all | Number}]
or
mmvdisk server change --node-class NcName
                      {[--pagepool {nM | nG | n%}] [--vdisk-space n%] [--map-memory n%]}   
                      [--recycle {none | one | all | Number}]
or
mmvdisk server list [-Y]
or
mmvdisk server list {--node-class NcName |-N Node[,Node...] | --recovery-group RgName} 
                    [{[--version][--config][--disk-topology ][--disk-list DiskExpr]} [--fanout N]] Start of change[-Y]End of change
or
mmvdisk server list -N Node --disk-topology {[--disk-topology][--disk-list DiskExpr]} -L  [-Y]

Availability

Available on all IBM Spectrum Scale editions.

Description

Use the mmvdisk server command to manage mmvdisk recovery group servers.

Before an IBM Spectrum Scale RAID recovery group can be created, the mmvdisk node class for the intended recovery group or recovery group pair must be configured to run IBM Spectrum Scale RAID and the IBM Spectrum Scale daemon must be restarted on each server in the node class. The mmvdisk server configure command examines the nodes in an mmvdisk node class, verifies that the node class members meet the same memory and disk topology requirements, and sets the appropriate configuration parameters. The mmvdisk server configure command can be used to optionally recycle the IBM Spectrum Scale daemon on the node class to ensure that the new configuration is in effect.

The configuration parameters set by mmvdisk server configure depend on the amount of server memory and the type of server disk topology. All node class members must have, within ten percent, the same amount of memory, and all node class members must have the same general type of server disk topology: either a paired recovery group (Elastic Storage Server) disk topology or a scale-out recovery group (IBM Spectrum® Scale Erasure Code Edition) disk topology or a shared recovery group (IBM Elastic Storage® System 3000) disk topology. The mmvdisk server configure command will not configure a node class where the server memory differs by more than ten percent or where a mixture of recovery group topologies is present.

The mmvdisk server configure command can be used to optionally recycle the IBM Spectrum Scale daemon on the recovery group server node class to ensure that the new configuration is in effect for a subsequent mmvdisk recoverygroup create command.

The mmvdisk server configure command can be used with the --update option on a node class where a recovery group is already present. This can be useful when an updated version of the mmvdisk command uses new configuration parameters, or when the server memory or disk topology has changed and the configuration parameters based on memory or the number of disks should be recalculated. The --recycle option can be used with --update to make the updated configuration parameters take effect by restarting the IBM Spectrum Scale daemon on the servers in the node class.

The mmvdisk server configure command can also be used to configure individual nodes for IBM Spectrum Scale RAID to prepare for maintenance use cases where servers are replaced or added to a recovery group and its node class.

The --verify option of the mmvdisk server configure command verifies whether or not the configuration settings for an mmvdisk server node class are as expected. The --verify option will check the real memory and server disk topology for each node in the node class, and check whether the IBM Spectrum Scale RAID configuration attributes for the node class are set to the expected values for the memory and topology. The mmvdisk server configure --verify command can be used to check whether a new release of IBM Spectrum Scale RAID has updated best practice configuration values; to check whether configuration values are affected by changes that have been made to server memory or server disk topology; or to check whether accidental changes have been made to the node class configuration. Start of changeThe mmvdisk command bypasses the check for the attributes that belong to customized configuration attributes of node class. The customized configuration attributes might be set by the --custom-config option or the mmvdisk nodeclass change command.End of change If an unexpected configuration is reported, the mmvdisk server configure --update command can be used to reset the node class to the correct configuration values.

The mmvdisk server unconfigure command will remove IBM Spectrum Scale RAID configuration from a node class where a recovery group was either deleted or never created. The mmvdisk server unconfigure command can also be used to unconfigure IBM Spectrum Scale RAID on individual nodes for maintenance use cases where servers are replaced or deleted from a recovery group and its node class.

Use the mmvdisk server change command to safely change the server memory configuration of an mmvdisk server node class. The memory configuration of a node class affects the number and size of vdisk sets that can be defined in the recovery groups of the node class. The mmvdisk server change command will not allow changes that reduce the available memory below what is required by the vdisk set definitions in the node class. The three options that affect node class server memory for vdisk sets are:
  1. The --pagepool option, which adjusts the node class IBM Spectrum Scale pagepool attribute.
  2. The --vdisk-space option, which adjusts the node class nsdRAIDBufferPoolSizePct attribute for the percentage of pagepool dedicated to vdisks and IBM Spectrum Scale RAID.
  3. The --map-memory option, which adjusts the node class nsdRAIDNonStealableBufPct attribute for the percentage of vdisk space dedicated to the track address maps for each vdisk.

The mmvdisk server change command should only very rarely be necessary. The default values for the server memory configuration options should be suitable for most circumstances. The pagepool default is dependent on the real server memory and the recovery group type in the node class. The vdisk space default is always 80% of pagepool, and the map memory default is always 50% of the vdisk space. In the event that, for example, the map memory needs to be increased to address very large vdisks with very small block sizes, the mmvdisk server change --node-class NC --map-memory 60% command can be used to safely adjust the server memory configuration in node class NC.

The --recycle one option can be used with the mmvdisk server change command to make any changes take effect by restarting IBM Spectrum Scale one node at a time in the node class.

The mmvdisk server list command, without any options, will list all the configured IBM Spectrum Scale RAID servers in the cluster. It identifies all recovery group servers and all nodes configured to be recovery group servers, regardless of whether the servers are members of an mmvdisk node class. This can be useful for planning server maintenance and planning the conversion of recovery groups and servers to mmvdisk management.

The mmvdisk server list command supports several options useful for showing server characteristics. The --node-class option will show whether the servers in a node class are active and the recovery group they serve. The --version option can be used to show the IBM Spectrum Scale and operating system versions for servers. The --config option can be used to show important aspects of the IBM Spectrum Scale RAID server configuration.

The mmvdisk server list --disk-topology option is important for reporting the supported IBM Spectrum Scale RAID disk topology on a node class or server. It should be used before mmvdisk server configure on a node class, to verify that all the node class members have the same server disk topology. It can also be used to help identify disk configuration problems that prevent nodes from acting as recovery group servers.

Start of changeThe mmvdisk server list --disk-list diskExpr option is important for reporting IBM Spectrum Scale RAID disk list on a node class or server. diskExpr is a subset of disk list which might be useful if you do not want to use all disks on a node to create recovery group.End of change

Parameters

mmvdisk server configure
Configure nodes or an mmvdisk node class to be IBM Spectrum Scale RAID servers. The nodes or node class must not already be serving a recovery group or already be configured, except when the --update option is used for a node class where a recovery group is present.
mmvdisk server unconfigure
Unconfigure nodes or an mmvdisk node class to no longer be IBM Spectrum Scale RAID servers. The nodes or node class must no longer be associated with a recovery group.
mmvdisk server change
Change server memory configuration attributes for an mmvdisk server node class. The pagepool, vdisk buffer space, and vdisk map memory can be adjusted for a node class. The mmvdisk server change command will not allow changes that reduce the available memory below what is required for the vdisk set definitions in the node class.
mmvdisk server list
List IBM Spectrum Scale RAID servers and their characteristics. A node is considered to be a server if it is in the server list of a recovery group, or if it is member of an mmvdisk node class, or if it has the IBM Spectrum Scale RAID nsdRAIDTracks attribute set.
--node-class NcName
Specifies a single mmvdisk node class name to be configured, unconfigured, or listed.
-N Node[,Node...]
Specifies individual nodes to be configured, unconfigured, or listed.
--recycle {none | one | all | Number}
With mmvdisk server configure or mmvdisk server unconfigure, specifies the number of nodes to be simultaneously restarted so that IBM Spectrum Scale configuration changes can take effect. The default is none, meaning that the administrator is responsible for using the mmshutdown and mmstartup commands to recycle the nodes or node class. The keyword one will recycle one node at a time, the keyword all will recycle all specified nodes simultaneously, or a Number of nodes can be chosen to be recycled at the same time. Care should be taken when recycling nodes if it is desired to maintain quorum availability in the IBM Spectrum Scale cluster.
--pagepool {nM | nG | n%}
With mmvdisk server configure and mmvdisk server change, choose a specific IBM Spectrum Scale pagepool value. Normally, mmvdisk chooses the best appropriate pagepool size for the real memory and disk topology in the server node class, so this option should only be used under instruction from IBM®.
--maxblocksize {2M | 4M | 8M | 16M}
With mmvdisk server configure, also set the IBM Spectrum Scale cluster maxblocksize value. Even though maxblocksize is not a server-specific configuration value, IBM Spectrum Scale RAID is often used with large file system block sizes, and this option provides a convenient way to set the maximum file system block size. This would typically be done only when configuring the first recovery group in a cluster, and it requires that the administrator ensure that the IBM Spectrum Scale daemon on all cluster nodes is recycled (not only on the recovery group server nodes).
Start of change--custom-config Attribute[=value][,Attribute[=value]...]End of change
Start of changeWith mmvdisk server configure, also set the IBM Spectrum Scale cluster customized configuration value. When mmvdisk (See IBM Spectrum Scale RAID: Administration) executes, it calls mmchconfig to set the value of attributes and add these attributes to node class's customized configuration attributes.End of change
--update
With mmvdisk server configure for a node class that already has a recovery group present, update the node class configuration parameters according to the current mmvdisk version and current server memory and disk topology.
--verify
With the mmvdisk server configure command, verify whether or not the IBM Spectrum Scale RAID configuration attributes for an mmvdisk server node class are set to the expected values.
--vdisk-space n%
With the mmvdisk server change command, specify the percentage of pagepool that should be reserved as space for vdisks and IBM Spectrum Scale RAID. This corresponds to the nsdRAIDBufferPoolSizePct configuration attribute, and the default value is 80%. Only in rare circumstances should this value need to be changed, so this option should only be used under instruction from IBM.
--map-memory n%
With the mmvdisk server change command, specify the percentage of the vdisk space that should be reserved to map the track addresses of vdisks. This corresponds to the nsdRAIDNonStealableBufPct configuration attribute, and the default value is 50%. Only in rare circumstances should this value need to be changed, so this option should only be used under instruction from IBM.
--recovery-group RgName
With mmvdisk server list, lists the servers for the named recovery group.
--version
With mmvdisk server list, lists the IBM Spectrum Scale and operating system version for the servers.
--config
With mmvdisk server list, lists the IBM Spectrum Scale RAID configuration for the servers.
--disk-topology
With mmvdisk server list, lists the discovered IBM Spectrum Scale RAID disk topology on the servers.
Start of change--disk-list DiskExprEnd of change
Start of changeWith mmvdisk server list, lists the discovered IBM Spectrum Scale RAID disks on the servers. If DiskExpr is specified, mmvdisk shows only the list of disks that belong to DiskExpr. For more information, see mmvdisk recoverygroup command.End of change
--fanout N
With mmvdisk server list --disk-topology, chooses how many servers to query simultaneously for their disk topologies. The default is 32. A fanout of 1 can be useful when disk fabric errors with twin-tailed server pairs are suspected.
-L
With mmvdisk server list --disk-topology and a single node, provide a detailed listing of the discovered IBM Spectrum Scale RAID disk topology. This is very useful when disk topology errors are suspected.
-Y
Specifies that the mmvdisk server list and mmvdisk server configure --verify commands produce colon-delimited raw output.

Exit status

0
Successful completion.
nonzero
A failure has occurred.

Security

You must have root authority to run the mmvdisk server command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages.

Examples

  • To display the IBM Spectrum Scale RAID disk topology of the servers in mmvdisk node class, ESS01:
    mmvdisk server list --node-class ESS01 --disk-topology

The system displays output similar to the following:

 node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     1  server01.gpfs.net                 no          100/100  ESS GL6
     2  server02.gpfs.net                 no          100/100  ESS GL6
  • Start of changeTo display the IBM Spectrum Scale RAID disk topology and disk list of the servers in mmvdisk node class, nc3500: 
    mmvdisk server list --node-class nc3500 --disk-topology --disk-list
    End of change
The system displays output similar to the following:

 node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     2  c145f11san08a                     no          100/100  ESS 3500 FN2 24 NVMe
     1  c145f11san08b                     no          100/100  ESS 3500 FN2 24 NVMe

                                      disk           list of
type  size(Mb)  spin   server         count       disk/location
----  --------  -----  -------------  -----  ------------------------
NVMe  3662830   0      c145f11san08a  24     nvme[0-23]n1 / 78E4001-[1-24]
                       c145f11san08b  24     nvme[0-23]n1 / 78E4001-[1-24]
  • To configure mmvdisk node class ESS01 to be IBM Spectrum Scale RAID recovery group servers and restart the daemons one at a time to make the configuration take effect: 
    mmvdisk server configure --node-class ESS01 --recycle one

The system displays output similar to the following:


mmvdisk: Checking resources for specified nodes.
mmvdisk: Node 'server01.gpfs.net' has a paired recovery group disk topology.
mmvdisk: Node 'server02.gpfs.net' has a paired recovery group disk topology.
mmvdisk: Node class 'ESS01' has a paired recovery group disk topology.
mmvdisk: Setting configuration for node class 'ESS01'.
mmvdisk: Node class 'ESS01' is now configured to be recovery group servers.
mmvdisk: Restarting GPFS on the following nodes:
mmvdisk:     server01.gpfs.net
mmvdisk: Restarting GPFS on the following nodes:
mmvdisk:     server02.gpfs.net
  • To show the status of the newly-configured mmvdisk node class ESS01:
    mmvdisk server list --node-class ESS01

The system displays output similar to the following:

 node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  server01.gpfs.net                 yes      configured, idle
     2  server02.gpfs.net                 yes      configured, idle

This shows that the servers are configured and running, but IBM Spectrum Scale RAID is idle because recovery groups have not yet been created.

  • To display the disk topology of the scale-out recovery group servers of recovery group RG01:
    mmvdisk server list --recovery-group RG01 --disk-topology

The system displays output similar to the following:

 node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     1  s01                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     2  s02                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     3  s03                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     4  s04                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     5  s05                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
  • To display the current status of the servers of recovery group RG01:
    mmvdisk server list --recovery-group RG01

The system displays output similar to the following:

 node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  s01                               yes      serving RG01: LG003, LG008
     2  s02                               yes      serving RG01: root, LG005, LG010
     3  s03                               yes      serving RG01: LG002, LG007
     4  s04                               yes      serving RG01: LG001, LG006
     5  s05                               yes      serving RG01: LG004, LG009
  • In an mmvdisk server node class where vdisk sets are defined, a vdisk space of 20% and a map memory of 30% will almost certainly be a memory configuration error, and the mmvdisk server change command will not allow it:
    mmvdisk server change --node-class ESS --vdisk-space 20% --map-memory 30%

The system displays output similar to the following:


mmvdisk: Checking resources for specified nodes.
mmvdisk: The requested change to node class 'ESS':
mmvdisk:   --vdisk-space 20% (config attribute 'nsdRAIDBufferPoolSizePct')
mmvdisk:   --map-memory 30% (config attribute 'nsdRAIDNonStealableBufPct')
mmvdisk: results in an available vdisk set map memory of 4558 MiB.
mmvdisk: This is less than the required vdisk set map memory of 26 GiB
mmvdisk: based on the current vdisk set definitions in the node class.
mmvdisk: Command failed. Examine previous error messages to determine cause.
This example is shown only as a reference. Please be reminded that the mmvdisk server change command should almost never be necessary, since the default values for the server memory configuration options are suitable for most circumstances.
  • To verify that the mmvdisk server node class ESS is properly configured:
    mmvdisk server configure --node-class ESS --verify

The system displays output similar to the following:


mmvdisk: Checking resources for specified nodes.
mmvdisk: Node class 'ESS' has 123 GiB total real memory per server.
mmvdisk: Node class 'ESS' has a paired recovery group disk topology.
mmvdisk: Node class 'ESS' has server disk topology 'ESS GL6'.
mmvdisk: Node class 'ESS' uses 'default.paired' recovery group configuration.

daemon configuration attribute      expected value  configured value
----------------------------------  --------------  ----------------
pagepool                            79665522278     as expected
nsdRAIDTracks                       128K            as expected
nsdRAIDBufferPoolSizePct            80              as expected
nsdRAIDNonStealableBufPct           50              as expected
pagepoolMaxPhysMemPct               90              as expected
nspdQueues                          64              as expected
nsdRAIDBlockDeviceMaxSectorsKB      0               as expected
nsdRAIDBlockDeviceNrRequests        0               as expected
nsdRAIDBlockDeviceQueueDepth        0               as expected
nsdRAIDBlockDeviceScheduler         off             as expected
nsdRAIDFlusherFWLogHighWatermarkMB  1000            as expected
nsdRAIDMaxPdiskQueueDepth           128             as expected
nsdRAIDSmallThreadRatio             2               as expected
nsdRAIDThreadsPerQueue              16              as expected
ignorePrefetchLUNCount              yes             as expected
maxBufferDescs                      2m              as expected
maxFilesToCache                     128k            as expected
maxMBpS                             30000           as expected
maxStatCache                        128k            as expected
nsdMaxWorkerThreads                 3842            as expected
nsdMinWorkerThreads                 3842            as expected
nsdMultiQueue                       256             as expected
numaMemoryInterleave                yes             as expected
panicOnIOHang                       yes             as expected
pitWorkerThreadsPerNode             32              as expected
workerThreads                       1024            as expected

mmvdisk: All configuration attribute values are as expected or customized.

See also

Location

/usr/lpp/mmfs/bin