Etherchannel Performance Issue due to Adapter’s LLDP Capability.

 When LPAR1 Etherchannel is in primary mode, while establishing network traffic through it, the switch will learn that the Mac address associated to its port 10 is the same as that of ent2 (that is,     Mac address of ent0). Consequently any ingress packets that the switch receives that is intended for ent2 of LPAR1 will be passed to the port10 by the network switch. Similarly, when the LPAR1 Etherchannel fails over to its backup, the Etherchannel sends Gratuitous Address Resolution Protocol (ARP) packets that have the source Mac address as of ent2 (that is,Mac address of ent0) through the adapter ent1. Hence, the network switch relearns that the port associated to the Mac address of LPAR1 ent2 is now port 11. That means, from then onwards, any ingress packets that the switch gets that is intended for ent2 of LPAR1 must be passed to the port 11.

Please note that the LLDP being a Link Layer protocol, the adapter will transmit and receive LLDP packets without the knowledge of LPAR’s OS. So when the LPAR1 Etherchannel being in backup mode, the ent0 adapter will still transmit LLDP packets. That means when Etherchannel being in backup mode, the network switch gets network packets having same source Mac address from ent0 (the LLDP packets) to its port 10 and from ent1 (the network traffic from ent2) to its port 11, this confuses the switch, because the switch learns the Mac address to port mapping based on network traffic as well. This confusion in the switch can cause poor network performance due to packet loss because while the Etherchannel is in Back up mode, ingress packets that the switch gets that are intended for ent2 of LPAR1 can be passed to the port 10 sometimes and hence those packets will be lost.

There are the 3 possible solutions to this problem and their advantages and disadvantages:

• Starting from AIX version 7200-05-03 and 7300-00, AIX added a feature to enable and disable the Mellanox adapter’s LLDP capability. Since ETS is the only AIX feature that needs the adapter LLDP capability, from this level of AIX onwards, the adapter will enable its LLDP capability only when the ETS is enabled, otherwise the LLDP capability of the adapter will be disabled. By disabling ETS, the above mentioned Etherchannel performance issue can be resolved but the disadvantage here is that you cannot use the ETS feature when Etherchannel is configured in NIB mode with this way.

By default, ETS is disabled in AIX and hence forms AIX version 7200-05-03 and 7300-00 onwards, the Mellanox Adapter’s LLDP capability will be disabled by default.

To disable ETS on an adapter, enter the following command: 

# chdev -l entX -a ets_mode=disabled

• By setting the Etherchannel optional field ‘Enable Alternate Address’ to ‘Yes’, you can specify a Mac address that you want to use for an Etherchannel instead of it using the Mac address of the primary adapter. Here the difficulty is, you must identify a unique Mac address that doesn’t exist in the network to assign to the Etherchannel. If you can do that, aligning with the above example, the adapters ent0, ent1, and ent2 will have different Mac address and hence the above-mentioned performance issue will not exist.

To enable alternate address on an existing Etherchannel permanently, enter the following command:

            # chdev -l entX -a alt_addr=0x......
            # chdev -l entX -a use_alt_addr=yes

• By setting the Etherchannel attribute ‘mac_swap’ to ‘Yes’. If you enable mac_swap, when Etherchannel fails over to backup Mode, then its underlying primary and backup adapters Mac addresses will be swapped. That means in the above example, when ent2 fails back to ent1, ent1 will get the Mac address of ent0 and ent0 will get the Mac address of ent1 and hence you can avoid the above-mentioned Etherchannel performance issue.