Troubleshooting
Problem
The MAC Forwarding method has some advantages over the Forwarding methods used by many Load Balancers. MAC Forwarding only forwards packets received from client to backend servers. The SourceIP=Clientip remains intact so the backend responds back to source and not the Edge LB as would be the case for NAT where SourceIP is replaced by a returnaddress on the Load Balancer. This means MAC Forwarding only handles half the traffic of NAT or can handle twice the traffic of NAT and so provides better performance and throughput. Since MAC Forwarding keeps SourceIP=Clientip it is easy to log clientip in the backend server access logs where as with NAT the clientip is lost and Edge is not a server type that has an equivalent to an access log. It is easy to get excited about setting up an Edge Load Balancer and configuring for MAC forwarding but the difficult work is in configuring the backend servers. The backend servers can be configured and tested for MAC forwarding without the Edge server once a clusterip address has been assigned. Pre-testing the clusterip address on each backend server will help ensure successful MAC Forwarding config of the Edge Load Balancer. To cluster a number of tcpip addresses under a single ClusterIP/VirtualIP most load balancers replace the destinationip=clusterip in the packet with a serverip of a backend server as defined in the load balancer cluster config. This prevents duplicate ip of the clusterip address since only the load balancer owns the clusterip on the network so clusterip stops at the load balancer. This usually means that SourceIP=Clientip has to change also to a returnaddress of the load balancer. This in effect makes the load balancer a proxy point remaining inbetween client and backend. In MAC Forwarding the Source and Destination IP address remain intact in the packet. Only the MAC Address for the Destination is changed to a MAC address of a backend server defined in the Load Balancer config. In MAC Forwarding the backend server receives a packet with sourceip=clientip and destinationip=clusterip therefore the backend server and client will be unware of the packet forwarder (Edge LB) but more importantly each backend server must also have the clusterip aliased. How can we alias this clusterip on multiple machines without creating a DUP IP situation on the network? We do this by aliasing the clusterip on a loopback adapter/interface on each backend server. Loopbacks do not announce or advertise their ip address on the network (except for Linux but arp suppression can be configured for the loopback to control the behavior). Backend server network adapters/interfaces can be configured to accept packets for ipaddresses aliased on the loopback. This may even be default behavior for the OS in use. This allows the backend server to own the clusterip within its own local scope of the machine but not on the network and so preventing dup ip. This also supports configuring the backend HTTP or LDAP server to listen and startup on the clusterip. MAC Forwarding usually requires the Edge, Backend Servers and Clusterip be part of the same TCPIP subnet. IP routing stops at the subnet router and is then routed as Local Area Network Ethernet using MAC address. Only the subnet router has the ARP table of MAC to ipaddresses for that subnet. Notice you do not configure the backend MAC addresses in the Edge config. You configure the backend ipaddress or hostname. Edge gets the MAC addresses of the backends from the subnet router. Edge MAC Forwarding is very dependant on ARP (Address Resolution Protocol), normal ethernet behavior and proper communication with the subnet router.
Diagnosing The Problem
In MAC Forwarding the backend server receives a packet with sourceip=clientip and destinationip=clusterip so there are three ways we can test the clusterip with the backend server long before we setup the Edge server. This will help verify you will not run into issues with doing MAC forwarding in your network.
Backend servers may be HTTP webservers, LDAP servers or other. Edge LB supports a number of different protocols because it works at the packet level below TCPIP. For the purpose of this technote and examples we will assume the most common, HTTP webservers.
To setup the machine and backend server you had to start with an ipaddress. Some may call that a maintenance or adminstrative ip. For the purpose of this technote we will refer to that ipaddress as serverip.
Test 1:
You do not need the Edge server to go to your network or tcpip administrator to request a unique clusterip for the subnet. Once you have that address start by aliasing that clusterip onto the network interface/adapter of one of the backend servers that the Edge LB will forward to. Then configure its HTTP (or other type) server to additionally listen on the clusterip. The HTTP server must also listen on the serverip defined in the Edge config because Edge advisors will target that ip. Then use any client to test that backend server will serve on the clusterip - HTTP://<clusterip-or-its-hostname>/<defaultpage>
After the first backend server is tested and works setup the loopback and unalias the clusterip off the network interface and onto the loopback. Be sure to consult the Edge guide for proper subnet masks for clusterip on loopback. For linux enable the arp suppression for the loopback then configure it such that the loopback and arp suppression maintain on a reboot of the linux server. This is also a good time to test that the arp suppression is working as a ping of the clusterip from a client machine should fail.
Alias the clusterip onto the next backend server and work through same tests.
Test 2:
The Local Test:
With the clusterip on the loopback if you open a browser on that backend server and issue a request for the clusterip HTTP://<clusterip/hostname/<defaultpage> it will start down the Server's OSI Network stack until it reaches the TCPIP Layer which will see that the clusterip is local on the loopback. This will loop the request back up the network stack to the HTTP Server. So the request should only and always be served from the local HTTP server. One way to be sure is to create a test page or modify the default page to include a servername or other identifier. I recommend a different background color for quickest recognition. This will also help out when you test the Edge Server Load Balancing.
The local test is also useful if you think you have SSL problems with one of the backend servers. This test is a great way to check that the HTTP server is configured correctly to serve on the clusterip.
Test 3:
This is the test defined in the Edge Load Balancer Administration Guide. This test uses ARP commands to assign the clusterip to a backend server macaddress in the subnet router ARP table, This creates a packet as would be forwarded from the Edge LB.
For the ARP test you will need a client machine on the same subnet as the backend servers. It will help later on if this client is not an Edge server or Backend server but. if the Edge server is not running or setup yet (executor and dsserver stopped) then that machine may suffice for ARP test. Stopping executor then dsserver on the Backup Edge server is also another way to create a test client.
Perform the following steps from the test client on the same subnet
when the Load Balancer is not running and cluster is unconfigured:
1. Issue the command: arp -d <clusterip>
2. Issue the command: ping <clusterip>
There should be no response. If there is a response to the ping, ensure
that you did not alias the cluster address to the interface. Ensure
that no machine has a published arp entry to the cluster address.
3. Ping the backend server ip:
Ping <backend serverip> - (the one that will be in the EDGE dscontrol server add)
then immediately issue the command:
arp -a
In the output from the command, you should see the MAC address of your
backend server. Issue the command:
arp -s <clusterip> <backendserver_mac_address>
The arp -s tells the local subnet router to associate <clusterip> to
the backendserver_mac_address (for the purpose of our testing).
4. Ping the cluster. You should get a response. Issue a http, telnet,
or other request that is addressed to the cluster that you expect your
backend server to handle. Ensure that it works properly.
5. if Step 4 is successful Issue the command: arp -d <clusterip>
to remove the address from the arp table.
6. Ping the cluster. There should be no response.
Note: If there is a response, issue an arp -a to get the MAC address of the misconfigured machine. Then, repeat steps 1 through 6.
7. If successful then perform same steps with next backend server.
-------------------
Even if Test 1 and 2 are successful it is important to verify that test 3 works since this most closely simulates MAC forwarding. Win2008 presents a classic situation where Test 1 and Test 2 would work but Test 3 would fail because Win2008 loopbacks need additional config to allow ipforwarding or to change the loopback behavior to older Win2003 weak network behavior before the ethernet adapter will accept a packet for the clusterip that is on the loopback.
While these tests are easier to do prior to Edge setup they can also help as problem isolation tests for Edge Mac Forwarding.
Resolving The Problem
Edge support experience reveals that the majority of PMRs with concerns about MAC Forwarding problems are due to config/operation of the backend server loopback, HTTP Server or the ability to respond back to client. The above tests can help you prepare for or isolate problems with Edge Load Balancer MAC forwarding.
-------------------
Additional Edge Hint Tips:
- Disable Checksum Offload on the Edge LB network interface - ref doc:
Checksum problems with Load Balancer
http://www.ibm.com/support/docview.wss?uid=swg21285822
- How to configure the loopback adapter on Microsoft Windows 2008.
http://www.ibm.com/support/docview.wss?uid=swg21304795
- Do not think of Edge the same as an HTTP Server, LDAP server, JVM appserver or other server that works above the TCPIP OSI layer. The forwarding component of Edge LB, the Executor, works at a packet level below the TCPIP layer. Those TCPIP ports you put in the config are the ports the backend servers are listening on. Edge LB does not listen on a TCPIP port. Edge will not respond to a PING <clusterip> because it will just forward the PING packet. Think of it more like a network device packet switcher or firewall. Since HTTP/HTTPS is above the TCPIP layer then Edge does not do any HTTP or HTTPS processing in forwarding packets. There is no SSL or keystore configuration required to support HTTPS. Edge LB will forward those SSL packets as is. The SSL handshake occurs with the backend server. Forwarding at the packet level allows Edge LB to forward packets for many different types of protocols.
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21587834