IBM Support

MustGather: IBM WebSphere Edge Load Balancer (Dispatcher component)

Troubleshooting


Problem

Collecting data for Dispatcher problems with the WebSphere® Application Server Load Balancer process. Review this information to familiarize yourself with the troubleshooting and data collection process. 
Be aware there are two Edge Load Balancer (LB) products:
  • Edge Load Balancer for IPV4 - short name is LLB for legacy Load Balancer (kernel intrusive) also known by its process name "ibmlb" (formerly "ibmnd"). This product is out of service.
  • Edge Load Balancer for IPV4 and IPV6 - short name is ULB for User space Load Balancer (minimal kernel-dependent code) also known by its process name "ibmlb2".

Resolving The Problem

If you already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server Products.

Load Balancer MustGather information
If high availability is configured, there is a primary and a standby load balancer. When the Load Balancer is configured with high availability, data collection is required from both load balancers.
If the reported problem involves communication with the backend servers, execute data collection from the backend servers as well as the load balancer.
For traffic forwarding problems, use a client workstation on the same subnet as the Load Balancer cluster. After communication between the cluster and client workstation is attempted, run the server collection tool on the client workstation.
Before data collection:
If possible, configure the Load Balancer and servers to respond to traffic when data is collected. 
  • If a Load Balancer is stopping unexpectedly or the server.log file contains entries with the keyword "Exception", a programming fault occurred. A core file needs to be provided. Ensure that your operating system is configured to produce an application core file. Review information about core file creation to ensure that the necessary data is collected.
  • Increase the default log level as the first command in the configuration:
    dscontrol set loglevel 5
    dscontrol set logsize unlimited
  • For high availability problems, increase the reach log level and log size.
    dscontrol manager reach set loglevel 5
    dscontrol manager reach set logsize unlimited
  • For advisor problems, increase the advisor log level and log size.
    dscontrol advisor loglevel protocol port 5
    dscontrol advisor logsize protocol port unlimited
Data collection:
Download the Load Balancer problem determination tool (lbpd) to collect the necessary data. On unix operating systems, place the script in the /opt/IBM/WebSphere/Edge/ULB/servers/bin directory and set execute permission on the tool after transfer to the load balancer machine. The Load Balancer problem determination tool creates the lbpmr.jar file in the 'PD' directory that is created in the ULB home directory.
lbpd-linux.sh  [version: 10.31.2023]
lbpd-aix.sh      [version: 10.31.2023]
lbpd-windows.bat    [version: 06.15.2022]
If an error occurs executing lbpd-linux.sh or lbpd-aix.sh, see note[1].
If the reported problem is related to forwarding traffic or the backend servers, download and run the server tool on each server. On unix systems, set execute permission on the tool after transfer to the server workstation. The tool creates a text file (txt) called the hostname of the workstation in the local directory.
server-linux.sh    [version: 04.07.2022]
server-aix.sh        [version: 04.07.2022]
server-windows.bat   [version: 04.07.2022]
If an error occurs executing server-linux.sh or server-aix.sh, see note[1].
Note[1] - The end of line character can be corrupted during download or file movement. On unix systems, if an error occurs indicating "Wrong interpreter", "No such file", or contains "^M" in the message, the end of line character is likely corrupted.  There are several ways to correct the problem:
  • Edit the file with vi editor. Positioned at the start of the file, press the Esc key and enter:  
    :%s/^V^M//g
!wq
    Where ^V is the Ctrl key + V pressed at the same time; ^M is the Ctrl key + M pressed at the same time.  
  • Edit the file with the vim editor. Positioned at the start of the file, press the Esc key and enter:  
    :set ff=unix
!wq
  • Download and use the dos2unix program to correct the line endings.
  • Use the sed editor to correct the line end character with the command:  image-20221115133554-1

Core file creation
The Load Balancer might crash when a programming error occurs. The executor is responsible for processing all packets and is most likely to experience a crash. Configure your operating system to produce a core file to ensure that the necessary data is available if a problem occurs.
  • If the Load Balancer experiences a fatal programming error on AIX, a system core is produced by default. If the system core is not created, ensure that the defined dump device has sufficient space.
  • For the Linux operating system, user limits must be sufficient size.
    ulimit -c unlimited
    • To configure Red Hat servers to produce core files, follow the instructions provided by Red Hat. If the abrt service is used, edit the /etc/abrt/abrt-action-save-package-data.conf file. Change the OpenGPGCheck and OpenUnpackaged settings.
      OpenGPGCheck=no
      OpenUnpackaged=yes
      Restart the abrtd service after changes are saved.
    • To configure Ubuntu servers to produce core files, the apport software must be configured.
    • To configure SLES servers, follow the instructions provided by SUSE.
  • Microsoft Windows does not produce core files by default. Settings must be modified in the registry and the systems needs to be rebooted for the changes to take effect.  The application name to use for the new registry key is ibmlb2.exe.
More data might be requested based on the reported problem. Information regarding extra data collection is provided.
  • Network traces help identify discarded or delayed traffic. AIX users must not run network traces until the Load Balancer is configured to allow for network tracing.
  • Executor tracing is helpful for unexpected program behavior.

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdIqAAK","label":"IBM Edge Load Balancer"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions"}]

Document Information

Modified date:
31 October 2023

UID

swg21140797

Page Feedback