Troubleshooting
Problem
Request failures occur when the load balancer is configured to distribute traffic based on MAC or encapsulated MAC (eMAC) forwarding.
Cause
MAC forwarding is the default forwarding method. With MAC or eMAC forwarding, responses from the server to the client are not routed on the same path as the request traffic that flows through the load balancer. The traffic indicates the load balancer cluster address as the source of the packets but the cluster address is associated with the load balancer, not the server. Some software and hardware blocks this traffic because it is identified as suspicious, possibly harmful traffic.
Environment
Software and hardware known to have configuration options to block traffic include operating systems, switches, routers, and firewalls. This issue occurs only when load balancer forwarding is configured as MAC or encapsulated MAC forwarding. The forwarding methods are explained in detail: How does the Load Balancer in Edge Component work?
A simple test to run to see whether traffic is being blocked due to device filtering is to temporarily define the server to use NAT forwarding at the load balancer. NAT forwarded traffic is not blocked by any of these filtering mechanisms.
Diagnosing The Problem
In order to determine whether traffic is blocked, you must be able to follow the flow of traffic from the client to the server and traffic from the server back to the client. If traffic is light, you can determine the flow by watching the packet statistics on each machine. Traffic is likely too high in a non-isolated environment. For most systems, network trace analysis would be the more effective means to determine whether traffic is being blocked. Each operating system contains network trace capabilities that can be used. The load balancer's must gather information details network trace syntax on various operating systems.
First trace the traffic on the load balancer machine (or examine the load balancer packet statistics if capable of isolating requests). For each packet that the client sends, the corresponding cluster, port, and server forwarded count increases. The network trace captures two copies of each packet. If you examine the MAC header of the packets, the first packet has a destination MAC of the load balancer ethernet MAC address. The second packet has a destination MAC of the server's ethernet MAC address.
If packets are correctly forwarded from the load balancer machine, gather a network trace on the backend server machines. A single copy of each packet that was reported at the load balancer is recorded and response packets that have the client IP as the destination IP. If you do not see packets with the client IP as the destination, traffic is not leaving the server machine.
If traffic is observed leaving the server machine, examine a network trace on the client machine. If traffic is not observed at the client machine, some device in between the two machines is blocking the traffic. Issue a traceroute command between the server and client machines to determine the network path. Research documentation for the routers and firewalls along the path to determine whether they have configuration settings to block the traffic. You can also locate different client machines along the network path and test connections through the load balancer to isolate where traffic is being blocked.
Resolving The Problem
If traffic is forwarding by the load balancer but not seen at the server, ensure that the server and the load balancer are on the same subnet. Traffic between the load balancer and the server cannot be routed to different subnets. Ensure that the cluster address is configured on the loopback interface.
If traffic is received at the server but traffic to the client is not observed leaving the server machine, ensure that the server software is listening on the port used by the traffic. With the Linux™ operating system, disable reverse path filtering:
sysctl -w net.ipv4.conf.<inteface_name>.rp_filter=0
Research switch and hub documentation for settings similar to the terms provided or try moving the server to a different type of switch or hub.
Software that blocks the response packets from MAC-based forwarded requests is known by several different names. Once you determine where traffic is blocked, review the documentation for the product and look for settings related to:
- Asymmetric routing
- TCP State bypass
- Direct server return (DSR)
- Triangulation
- Stateful and stateless routing (stateless routing needs to be allowed)
- MAC forwarding
The vendor-specific documentation explains how to configure the product to prevent traffic from being blocked.
Document Location
Worldwide
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"Edge Component;Load Balancer","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
15 April 2020
UID
ibm16116956