IBM Support

IBM WebSphere Load Balancer required adapter interface options

Troubleshooting


Problem

For best performance, the Load Balancer requires certain Ethernet interfaces offload features to be disabled. The IBM WebSphere Load Balancer does not support these functions that modify the contents of cluster traffic. Packets that do not conform to the standard IP protocol are discarded by the Load Balancer.

Symptom

Clients experience a noticeable performance delay or failed requests.
When a response is not received for a packet, the sending workstation retransmits the packet. After several failed attempts, most software disables the offload feature for the failing packet. The packet is successfully received when the offload feature is disabled.

Cause

Features that are known to be problematic with the Load Balancer are:
  •  Checksum offload (TCP, UDP, or IP)
  •  Large send or receive (LSO, LRO)
  •  TCP segmentation offload (TSO)
  •  Generic send or receive (GSO, GRO)
  •  Jumbo frames
A server can modify data in packets as the contents are processed. As such, a network trace tool and the Load Balancer can report different information in a packet.
When checksum offload is enabled, there is typically a noticeable delay with client requests. Network traces capture packets sent from the load balancers but these packets are not captured in network traces collected on the backend servers.
With the large send or receive, generic send or receive, and jumbo frames, the Load Balancer discards packets larger than the maximum transmit unit. The executor report indicates a count of the number of packets discarded.  Network traces collected at the Load Balancer capture inbound packets larger than the maximum Ethernet frame size but a copy of the packet is not transmitted to a backend server.

Diagnosing The Problem

If packets are not forwarded or there appears to be a long delay in forwarding packets, a network trace is necessary to see the malformed packets. Review network trace instructions to avoid problems from concurrent execution of network trace tools and the Load Balancer. Examine the packets captured in the problematic connection. If the IP or TCP checksums are incorrect, the checksum offload feature must be disabled.


Example of a trace :
Internet Protocol, Src: 3.210.180.203 (3.210.180.203), Dst: 3.210.180.215 (3.210.180.215)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 47
Identification: 0xea49 (59977)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0xdf38 [correct]
Good: True
Bad : False
Source: 3.210.180.203 (3.210.180.203)
Destination: 3.210.180.215 (3.210.180.215)
Transmission Control Protocol, Src Port: http (80), Dst Port: 1610 (1610), Seq: 1, Ack: 9, Len: 7
Source port: http (80)
Destination port: 1610 (1610)
Sequence number: 1 (relative sequence number)
Next sequence number: 8 (relative sequence number)
Acknowledgement number: 9 (relative ack number)
Header length: 20 bytes
Flags: 0x0018 (PSH, ACK)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 1... = Push: Set
.... .0.. = Reset: Not set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x7168 [incorrect, should be 0x3abf]
Features such as large send or receive, generic send or receive, or jumbo frames, can produce packets larger r than the maximum Ethernet frame size (1518 bytes).
image-20220113163652-1

NOTE-Often, the packet transmitted 3 times. The first 2 with the offload function enabled and the third with the offload disabled. Because of this, Load Balancer operates with offload function enabled but the performance is unacceptable. Connections might be terminated prematurely by the backend server or the client after repeated transmission failures occur.

Review your OS for documentation for disabling these adapter settings. Some common methods are listed but as technologies change and if different features and configurations are used, these commands are not sufficient.

For Windows®, the adapter properties can be examined to determine whether any of the offload features are enabled.

For AIX®, the command 'ifconfig -a' shows the checksum offload and large send settings.
 > ifconfig -a
en0: flags=1e084863,814c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,
                          64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

For Linux®, use the ethtool command to examine the adapter features.
ethtool -k eno16780032
Features for eno16780032:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on

Resolving The Problem

Review your OS for documentation to disable these adapter settings. Some common methods are listed. Different commands can exist in virtualization software, shared interfaces, specific hardware, or different operating system releases.

For Windows™:
Open the Network Connections and select View Properties for each Ethernet interface. Select the Configure button next to the adapter. Select the Advanced tab. If the checksum offload feature is listed, it needs to be disabled.

For AIX®:  The largesend, large_receive, checksum_offload, chksum_offload, mtu_bypass, and jumbo_frames features need to be disabled if present.
ifconfig <logical interface> -largesend
ifconfig <logical interface> -checksum_offload
ODM level:
chdev -l <physical interface> -a mtu_bypass=off
chdev -l <physical interface> -a chksum_offload=no
VIOS:
chdev -l <physical interface> -a large_receive=0
chdev -l <physical interface> -a chksum_offload=no
Native command:
lsattr -El <physical interface> -a -largesend
lsattr -El <physical interface> -a -chksum_offload
Note: An interface must be down before the jumbo_frames setting can be modified. An example of a logical interface is en0. An example of the physical interface is ent0.
chdev -l <logical interface> -a state=down
chdev -l <logical interface> -a state=detach
chdev -l <physical interface> -a jumbo_frames=no
chdev -l <logical interface> -a state=up
The change can require a reboot or refresh of the network.
mkdev -l <name>
For Linux™:
ethtool -K <interface> rx off tx off
ethtool -K <interface> tso off
ethtool -K <interface> gso off

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdIqAAK","label":"IBM Edge Load Balancer"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"8.5.0;8.5.5;9.0.0;9.0.5"}]

Document Information

Modified date:
31 May 2023

UID

swg21285822

Page Feedback