Planning for more than four fabric ports per node

You can use more than four fabric ports per node to improve peak load I/O performance, but careful planning is needed.

A fabric port is a Fibre Channel port. If you use more than four fabric ports per node, you must either use the localfcportmask and partnerfcportmask commands or be careful with your fabric zoning configuration.

Careful zoning improves resilience and prevents lease expires due to port errors, low fabric buffer credits, shared adapter resources, or push-back from remote links. Table 1 displays a port allocation scheme.
Table 1. Port allocation scheme
Adapter / Port 4 ports 8 ports 12 ports 16 ports SAN Fabric
Adapter 1 Port 1 Host and Storage Host and Storage Host and Storage Host and Storage A
Adapter 1 Port 2 Host and Storage Host and Storage Host and Storage Host and Storage B
Adapter 1 Port 3 Intracluster and Replication Intracluster Intracluster Intracluster A
Adapter 1 Port 4 Intracluster and Replication Intracluster Intracluster Intracluster B
Adapter 2 Port 1 - Host and Storage Host and Storage Host and Storage A
Adapter 2 Port 2 - Host and Storage Host and Storage Host and Storage B
Adapter 2 Port 3 - Intracluster or Replication Replication or Host and Storage Replication or Host and Storage A
Adapter 2 Port 4 - Intracluster or Replication Replication or Host and Storage Replication or Host and Storage B
Adapter 3 Port 1 - - Host and Storage Host and Storage A
Adapter 3 Port 2 - - Host and Storage Host and Storage B
Adapter 3 Port 3 - - Intracluster Intracluster A
Adapter 3 Port 4 - - Intracluster Intracluster B
Adapter 4 Port 1 - - - Host and Storage A
Adapter 4 Port 2 - - - Host and Storage B
Adapter 4 Port 3 - - - Replication or Host and Storage A
Adapter 4 Port 4 - - - Replication or Host and Storage B
localfcportmask 1100 11001100 or 00001100 110000001100 0000110000001100 -
remotefcportmask 1100 00000000 or 11000000 000011000000 1100000011000000 -
  • Host refers to host objects defined in the system.
  • Replication refers to nodes that are part of a different cluster.
  • Storage refers to controller objects defined in the system if external storage is being used.
  • Intracluster refers to nodes within the same cluster.
  • The word "and" indicates that both types are used.
  • The word "or" indicates that one of the options must be selected. If using replication, preference should be given to replication.

If a node receives more than 16 logins from another node, then it causes node error 860.

You can ensure that no more than 16 logins are received by following these guidelines:
  • Zone the SAN fabric to reduce the number of paths between the nodes.
  • Apply a local Fibre Channel port mask (if the nodes are within the same system) or partner Fibre Channel port mask (if the nodes are in different systems) to reduce the number of ports that are being used for node-to-node communication.
  • Provide a combination of the two (zoning and port masks).
To avoid receiving node error 860 and to maximize performance on your system, follow these guidelines:
  • For redundancy, use a minimum of two ports. If ports are being dedicated for different types of traffic, assign a minimum of two ports for each type of traffic.
  • Within a system, up to 50% of the overall data that is transferred across the SAN is transmitted between the nodes. However, for read-intensive workloads, the figure is much less. Therefore, if ports are being dedicated for different types of traffic, assign between 1/4 and 1/2 of the overall ports for node-to-node communication within a system.
  • For replication between systems, the connection between the systems is usually the bottleneck. Other than for redundancy, there is no point in having more SAN connections than there is bandwidth. For example, if two sites are connected with a 10 Gbps link, two 8 Gbps ports are sufficient. Systems usually are configured with two ports per node for replication traffic primarily for redundancy. For systems with larger numbers of nodes, it might be appropriate to have connections only from a subset of the nodes for replication to a remote system. The system automatically forwards replication traffic between local nodes so that all nodes can still participate in replication.
In the examples that follow, consider this configuration that is shown in Figure 1:
Figure 1. Fabric port configuration
This figure shows an example configuration of four nodes where each adapter has four ports
  • Four nodes, where nodes 1 and 2 are in a system on site A. Nodes 3 and 4 are in a system on site B.
  • Each site has 2 switches (switch 1 and 2 on site A; switch 3 and 4 on site B).
  • Each node has 2 adapters (A and B).
  • Each adapter has 4 ports where the port is named P nodeid adapterid portnumber. For example, port 3 on adapter B of node 4 is named P4B3.

Example 1: Zoning only

Each node in the system is configured in the following way:
  • Four ports are used for host, controller, and local node connections.
  • Four ports are used for replication.

The following zones are required for two sites, A and B:

Zone sets for site A:
  • Zone for storage controllers and local node connections (Fibre Channel adapter HBA)
    • Zone 1 - [Node ports P1A1, P1A2, P2A1, P2A2 plus all storage controller ports in site A on switch 1]
    • Zone 2 - [Node ports P1A3, P1A4, P2A3, P2A4 plus all storage controller ports in site A on switch 2]
  • Zones for host attachment (Fibre Channel adapter HBA A)
    • The same ports on the first adapter are zoned to be visible to hosts. It is recommended that each host is in a separate zone, and a host with two ports that are connected to two fabrics have two zones, one per fabric. The actual number of zones might be two multiplied by the number of hosts. A host with four ports that are connected to two fabrics should have four zones. In this case, the actual number of zones might be four multiplied by the number of hosts. In the four-port host case, the zones for one host might be:
      • Zone 3 - [host port, P1A1 and P2A1 for multipath fail over] on switch 1
      • Zone 4 - [host port, P1A2 and P2A2 for multipath fail over] on switch 1
      • Zone 5 - [host port, P1A3 and P2A3 for multipath fail over] on switch 2
      • Zone 6 - [host port, P1A4 and P2A4 for multipath fail over] on switch 2
      If the host only had two ports, it might have Zone 3 and Zone 5, or Zone 4 and Zone 6.
  • Zones for replication (Fibre Channel adapter HBA B only):
    To ensure replication use the second Fibre Channel adapter exclusively, it is required that each local or remote port pair has its own zone:
    • Zone 7 - [P1B1,P3B1]
    • Zone 8 - [P1B1,P3B2]
    • Zone 9 - [P1B1,P4B1]
    • Zone 10 - [P1B1,P4B2]
    • Zone 11 - [P1B2,P3B1]
    • Zone 12 - [P1B2,P3B2]
    • Zone 13 - [P1B2,P4B1]
    • Zone 14 - [P1B2,P4B2]
    • Zone 15 - [P2B1,P3B1]
    • Zone 16 - [P2B1,P3B2]
    • Zone 17 - [P2B1,P4B1]
    • Zone 18 - [P2B1,P4B2]
    • Zone 18 - [P2B1,P4B2]
    • Zone 19 - [P2B1,P4B2]
    • Zone 20 - [P2B2,P3B2]
    • Zone 21 - [P2B2,P4B1]
    • Zone 22 - [P2B2,P4B2]

Zone sets for site B:

This zone set is essentially a mirror image of the zoning for site A.
  • Zone for storage controllers and local node connections (Fibre Channel adapter HBA A):
    • Zone 23 - [Node ports P3A1, P3A2, P4A1, P4A2 plus all storage controller ports in site B on switch 3]
    • Zone 24 - [Node ports P3A3, P3A4, P4A3, P4A4 plus all storage controller ports in site B on switch 4]
  • Zones for host attachment (Fibre Channel adapter HBA A)
    • The same ports on the first adapter are zoned to be visible to hosts. Only zones for one host are listed. It is recommended that each host is in a separate zone. A host with two ports that are connected to two fabrics has two zones, one per fabric. The actual number of zones might be two multiplied by the number of hosts. A host with four ports that are connected to two fabrics should have four zones. In this case, the actual number of zones might be four multiplied by the number of hosts. In the four-port host case, the zones for one host might be:
      • Zone 25 - [host ports, P3A1 and P4A1 for multipath fail over] on switch 3
      • Zone 26 - [host ports, P3A2 and P4A2 for multipath fail over] on switch 3
      • Zone 27 - [host ports, P3A3 and P4A3 for multipath fail over] on switch 4
      • Zone 28 - [host ports, P3A4 and P4A4 for multipath fail over] on switch 4
      If the host had only two ports, it might have Zone 25 and Zone 27, or Zone 26 and Zone 28.
  • Zones for replication (Fibre Channel adapter HBA B only):

    To ensure replication use the second Fibre Channel adapter exclusively, it is required that each local or remote port pair has its own zone:

    • Zone 29 - [P1B3,P3B3]
    • Zone 30 - [P1B3,P3B4]
    • Zone 31 - [P1B3,P4B3]
    • Zone 32 - [P1B3,P4B4]
    • Zone 33 - [P1B4,P3B3]
    • Zone 34 - [P1B4,P3B4]
    • Zone 35 - [P1B4,P4B3]
    • Zone 36 - [P1B4,P4B4]
    • Zone 37 - [P2B3,P3B3]
    • Zone 38 - [P2B3,P3B4]
    • Zone 39 - [P2B3,P4B3]
    • Zone 40 - [P2B3,P4B4]
    • Zone 41 - [P2B4,P3B3]
    • Zone 42 - [P2B4,P3B4]
    • Zone 43 - [P2B4,P4B3]
    • Zone 44 - [P2B4,P4B4]

Example 2: Port masking only

Assuming the same configuration as in example 1, you can achieve the same result by using port masking rather than zoning. The positions in the mask represent the Fibre Channel I/O port IDs with ID 1 in the rightmost position. These IDs are shown by the lsportfc command. In this example ports A1, A2, A3, A4, B1, B2, B3, and B4 correspond to FC I/O port IDs 1, 2, 3, 4, 5, 6, 7 and 8.

For remote partnership communication, apply a mask that limits communication to ports B1, B2, B3, and B4 by applying a port mask of 11110000 to both systems. Use the command chsystem -partnerfcportmask 11110000.

For local node-to-node communication, apply a mask that limits communication to ports A1, A2, A3, and A4 by applying a port mask of 00001111 to both systems. Use the command chsystem -localfcportmask 00001111.

It is then possible to have a single zone that contains all the node ports, as the port masking ensures that the number of logins is no more than 16.

With this setup, zoning for host connectivity and storage connectivity does not need to consider the rules about more than 16 logins.