subscribe iconSubscribe to this information
POWER6 information

Installing and configuring the InfiniBand switch

Use this procedure to install and configure InfiniBand switches.

It is possible to perform some of the tasks in this procedure in a method other than that which is described. If you use other methods for configuring switches, you must review a few key points in the installation process outlined in this procedure. These key points are related to the order and coordination of tasks and configuration settings that are required in a cluster environment.

Parent topic: Installing and configuring vendor InfiniBand switches

Key points for installing and configuring the InfiniBand switch

Review the following list of key points before beginning the switch installation process:
  1. Power on the InfiniBand switches and configure their IP addresses before attaching them to the cluster virtual local area network (VLAN). Alternatively, you must add each switch to the cluster VLAN individually and change the default IP address before adding another switch.
    Note: The switch vendor documentation refers to the Ethernet connection for switch management as the service VLAN.
  2. Set the static IP addresses on the switches for the cluster VLAN.
    Notes:
    • If a switch has multiple managed spines or management modules, each one requires its own IP address, in addition to an overall chassis IP address.
    • You also need to set up the default gateway.
    • If an InfiniBand switch has multiple Ethernet connections for the cluster VLAN, and the cluster has multiple cluster VLANs for redundancy, the Ethernet ports on the switch must connect to the same cluster VLAN.
  3. Update the switch firmware code as required. See the IBM® Clusters with the InfiniBand Switch Web site for information regarding switch code levels.
  4. Set the switch name.
  5. Temporarily stop the embedded subnet manager and performance manager from running. Depending on the configuration, this might be a permanent state.
  6. Set up logging:
    1. Enable full logging.
    2. Enable the full logging format.
    3. Point switch logs to the Cluster Systems Management/Management Server (CSM/MS).
  7. Set the chassis maximum transfer unit (MTU) value according to the installation plan.
  8. If the switch is not running an embedded subnet manager, complete the following tasks:
    1. Ensure that the embedded subnet manager is disabled.
    2. Disable the performance manager.
    3. Disable the default broadcast group.
  9. If the switch is running an embedded subnet manager, complete the following tasks:
    1. Use the license key to enable the embedded subnet manager to be run on the switch.
    2. Set up the priority based on the fabric management work sheet.
    3. Set the global identifier (GID) prefix value according to the installation plan. See the QLogic switch planning work sheets or Planning for global identifier prefixes.
    4. If this is a high-performance computing (HPC) environment, set the LID Mask Control (LMC) value to 2.
    5. Set the broadcast MTU value according to the installation plan. See the QLogic switch planning work sheets or Planning for maximum transfer units (MTUs).
  10. Point to the Network Time Protocol (NTP) server.
  11. Instruct the customer to verify that the switch is detected by the CSM/MS by using the verify detection step in the Verifying the InfiniBand network topology and operation topic.
Note: If you are expanding an existing cluster, also consider using the QLogic switch command help. On the command-line interface (CLI), use the help command name command. Otherwise, see the users guides for information about the commands and to identify the appropriate command in its procedural documentation.

Installing and configuring InfiniBand switches

Complete the following procedure to install and configure your InfiniBand switches:

Note: The tasks W1 through W6 described in the following steps are based on the major steps found in the Figure 1.
  1. Review this procedure and determine whether the Fabric Management Server has the Fast Fabric Toolset installed and is on the cluster VLAN before you finish this procedure. If Fast Fabric tools are available, you can customize the multiple switches simultaneously after you have them configured with unique IP addresses and they are attached to the cluster VLAN. If you do not have Fast Fabric tools ready, you need to customize each switch individually. In that case, you might want to do the customization step right after you set up the switch management IP address and give it a name.
  2. W1: Physically place frames and switches on the data center floor, and complete the following steps:
    1. Review the vendor documentation for each switch model that you are installing.
    2. Physically install the InfiniBand switches into 19-inch frames (or racks) and attach power cables to the switches according to the instructions for the InfiniBand switch model. This automatically powers on the switches. There is no power switch for the switches.
      Note: Do not connect the Ethernet connections for the cluster VLAN at this time.
  3. W2: Set up the Ethernet interface for the cluster VLAN by setting the switch to a fixed IP address, which is provided by the customer. See the QLogic switch planning work sheets. Use the procedure in the vendor documentation for setting switch addresses.
    Notes:
    • You can attach a notebook to the serial port of the switch, or you can attach each switch individually to the cluster VLAN, use the default address to get into the CLI, and customize the static IP address.
    • As indicated in Planning for QLogic InfiniBand switch configurations, QLogic switches with managed spine modules have multiple addresses. There is an address for each managed spine, as well as an overall chassis address that is used by whichever spine is master at any given time.
    • If you are customizing the IP address of the switch by accessing the CLI through the serial port on the switch, you might want to leave the CLI open to perform the rest of the customization. This is not necessary if the Fast Fabric Toolset has been installed and can access the switches, because with Fast Fabric tools, you can update multiple switches simultaneously.
    • For QLogic switches, the key commands are setChassisIpAddr and setDefaultRoute.
    • Use an appropriate subnet mask when setting up the IP addresses.
  4. Set the switch name. For QLogic switches, use the setIBNodeDesc command.
  5. Disable the subnet manager functions and performance manager functions. If embedded subnet management is used, this is reversed after the network cabling is completed.
    • Ensure that the embedded subnet manager is not running by using the smControl stop command.
    • Ensure that the embedded subnet manager does not start at boot by using the smConfig startAtBoot no command.
    • Ensure that the performance manager is not running by using the smPmBmStart disable command.
  6. W3: Attach the switch to the cluster VLAN.
    Note: If the switch has multiple Ethernet connections, they must all attach to the same Ethernet subnet.
  7. W4: For QLogic switches, if the Fast Fabric Toolset is installed on the fabric management server verify that the Fast Fabric tools can access the switch. Referring to the Fast Fabric Toolset Users Guide, use a simple query command or ping test to the switch. For example, the pingall command could be used as long as you point to the switch chassis and not to the servers or nodes.
  8. W5: Verify that the switch code matches the latest supported level indicated in IBM Clusters with the InfiniBand Switch Web site. Check the switch software level by using a method described in vendor's switch users guides. These guides also describe how to update the switch's software, which is available on the vendor's Web site. For QLogic switches, one of the following guides and methods are suggested:
    • You can check each switch individually by using a command on its command line interface (CLI). This command can be found in the switch users guide for the model.
    • If the Fast Fabric Toolset is installed on the fabric management server, you can check the code levels of multiple switch simultaneously by using techniques found in the Fast Fabric Toolset Users Guide.
    • You can use the fwVersion command. If this command is issued by using Fast Fabric tools, the cmdall command can be used to issue this command to all switches simultaneously.
    • For updating multiple switches simultaneously, use the Fast Fabric Toolset.
  9. W6: Finalize the configuration for each InfiniBand switch.

    You are setting up the final switch and subnet manager configuration. You planned the following values in the planning phase (see Planning InfiniBand network cabling and configuration and the QLogic switch planning work sheets).

    • Subnet manager priority
    • MTU
    • LMC
    • GID prefix
    • Node appearance and disappearance log threshold

    For QLogic switches, the pertinent commands and user manuals and methods to be used by this procedure follow:

    • You can work with each switch individually by using a command on its CLI.
    • If the Fast Fabric Toolset is installed on the fabric management server, you can check the code levels of multiple switch simultaneously by using techniques found in the Fast Fabric Toolset Users Guide. Set the chassis maximum transfer unit (MTU) value according to the installation plan. See the QLogic switch planning work sheets or Planning for maximum transfer units (MTUs).
    • For setting chassis MTU use the ismChassisSetMtuvalue command on each switch (4 equals 2 KB; 5 equals 4 KB).
    • For each embedded subnet manager, use the following commands for final configuration:
      • For the priority: smPrioritypriority
      • For LMC=2: smMasterLMC=2
      • For 4 KB broadcast MTU with default pkey: smDefBcGroup 0xFFFF 5rate (rate: 3 equals SDR; 6 equals DDR rate)
      • For 2 KB broadcast MTU with default pkey: smDefBcGroup 0xFFFF 4rate (rate: 3 equals SDR; 6 equals DDR rate)
      • For GID prefix: smGidPrefixGID-prefix value
      • For node appearance or disappearance threshold of 10: smAppearanceMsgThresh 10
    1. If this switch has an embedded subnet manager, complete the following steps:
      1. Enable the subnet manager for operation by using the license key. Do not start the embedded subnet manager; you will start it later, in the procedure Attaching cables to the InfiniBand network. Use the addKeykey command.
      2. Set the GID-prefix value according to the installation plan. See the QLogic switch planning work sheets or Planning for global identifier prefixes.
      3. If this is a high-performance computing (HPC) environment, set the LMC value to 2.
    2. Set the broadcast MTU value according to the installation plan. See the QLogic switch planning work sheets or Planning for maximum transfer units (MTUs).
    3. If applicable, point to the NTP server. For QLogic switches, this is done by using the time command. Details are in the Switch Users Guide. Typical commands from the fast fabric management server are as follows. If remote command processing is set up on the CSM/MS, you can use the dsh command instead of cmdall. Remember to use the --devicetype IBSwitch::Qlogic command to access the switches.
      1. If applicable, set the time by using the Network Time Protocol (NTP) server: cmdall -C 'time -S [NTP server IP-address]
      2. If no NTP server is present, set the local time: cmdall –C ‘time –T hhmmss[mmddyyyy]'
      3. Set the time zone, where X is the offset of the time zone from GMT: cmdall -C 'timeZoneConf X'
      4. Set the daylight saving time, where X is the offset of the time zone from GMT: cmdall -C 'timeDSTTimeoutX'

      If you are also responsible for cabling the InfiniBand network, proceed to Attaching cables to the InfiniBand network. Otherwise, you can return to the overview of the installation section to find your next set of installation tasks.

Other installation tasks involving final configuration of switches are:

Send feedback | Rate this page

Last updated: Tue, February 08, 2011