IBM Support

MustGather: WebSphere® Load Balancer executor tracing

Troubleshooting


Problem

Collecting data for problems with the WebSphere® Application Server Edge component, Load Balancer packet-forwarding process (the executor) is described. This information applies only to the Load Balancer Dispatcher component. Unless directed by IBM® support to collect an executor trace, use caution increasing logging to ensure load balancer performance is not adversely affected.

Environment

The executor module interacts with the operating system network layer and interface cards. Each operating system designed the network layer in a different fashion so the load balancer executor tracing has differences on each operating system.
If the load balancer is running on a Linux™ operating system, executor tracing is written to /var/log/messages with the operating system syslog API. Messages might not be reported with the default Linux™ rate limiting settings. If there are reports of lost or suppressed messages in the log, tune the operating system rsyslog.conf and journald.conf rate limit settings.
If the load balancer is running on the Windows™ operating system, executor tracing is written to the ibmlb2.log file located in the %IBMULBPATH%\servers\logs\dispatcher directory. Data in the file is only accessible after the executor is stopped.
If the load balancer is running on the AIX™ operating system, executor trace information is not written to a file until requested.  The executor runs as a kernel extension on AIX™ so it stores the trace information in an internal 64kb memory buffer. The load balancer provides a tool, lb_kext, to write this buffer to a file.  The lb_kext tool is located in the /opt/IBM/WebSphere/Edge/ULB/servers/bin directory.
Syntax:
./lb_kext -l -t <secs> -n <collections>
A collection of the data in the executor buffer is collected every "-t" second until the "-n" number of collections completes. Each collection is written to a file called ibmulb_log<collection> file located in the working directory. The first collection filename is appended with the value specified (-n) and the last collection has the file name ibmulb_log1.

Diagnosing The Problem

The executor component is responsible for receiving and analyzing every packet on each interface of the Load Balancer machine. The executor tracing provides details about these packets. By default, executor logging is sparse to prevent excessive log growth and performance degradation. IBM® support might request executor logging increased to resolve a reported problem or a user can increase logging to investigate load balancer operations in more detail.
The lbcommand (lbcommand.sh on unix) tool provided in the IBMULB_ROOT\servers\bin directory configures different tracing specifications for the executor.
To enable all tracing (not recommended unless directed by IBM® for problem determination), set the logmask value to -1. The output of this command lists the individual trace specifications that are enabled.
When the logmask value is set to -1, the list of trace specifications enabled is all available specifications.
This example is taken from the Windows™ operating system:
C:\Program Files\IBM\WebSphere\Edge\ULB\servers\bin>lbcommand 0 set logmask -1
LBStackCommand: 0 set logmask -1
executor key: (CONFIG) 0/0/0/0 action: 4(SET) rc: 1(SUCCESS)
  children: 9.42.6.103/0/0/0
  addable: n/a
  setable:
    logmask: 0xffffffffffffffff ( ipv4 ipv6 ip tcpudp linkedlist executor command
cluster port server config iface ifcollection ifinstance os semaphore arp hashtable
neigh select icmpv6 ha icmp fragment memory rule useserver srvset nat err intact
discard forward held pstats rmreap error warn )
    (ha)role: 1 (SOLO)
    (ha)takeoverstrategy: 3 (undef)
    (ha)replicatestrategy: 0 (NOREP)
    (ha)reachscore: 0
    (ha)timeout: 2
    (ha)reachaddrs: 0 0 0 0 0 0 0 0 0 0 0 0
    (ha)reachstatus: 0 0 0 0 0 0 0 0 0 0 0 0
    (ha)port: 0
    (ha)manualtakeoverrequested: 0
    nfa: 1.2.3.4
    clientgateway: 1.2.3.1
    clientgateway_ipv6: 0:0:0:0:0:0:0:0
    clientcollocate: 0
    Legacy: cps: 0    Legacy: bps: 0    Legacy: maxclusters: 0    Legacy: maxports: 0
    Legacy: maxservers: 0
    Legacy: fintimeout: 20
    Legacy: staletimeout: 300
    Legacy: stickytime: 0
    Legacy: weightbound: 20
    Legacy: configfilename:
  getable:
    starttime: 01 Sep 2020 04:02:10 EDT
    version: 8.5.5-18 - 20200901-002637 [WSBLD346] NT/amd64/cl
    stats/ipv4: in:6148 [ fwd:0 err:0 notforus:6124 discard:16 ]:6140 gen:8
    stats/ipv6: in:65 [ fwd:0 err:0 notforus:65 discard:0 ]:65 gen:0
    stats/arp:  in:4405 [ fwd:13 err:0 notforus:4392 discard:0 ]:4405 gen:13
    stats/icmp:   in:16 [ fwd:0 err:0 notforus:0 discard:0 ]:0 gen:8
    stats/icmpv6: in:15 [ fwd:0 err:0 notforus:15 discard:0 ]:15 gen:0
    clientgwneighstat: 0x1 ( RXOK )
    clientgwneighstat_ipv6: 0x0 ( undef )
    (ha)state:    3 (NA)
    (ha)substate: 1 (NA)
    (ha)lastevent: 0
The individual trace specifications:
Specification Details
ipv4 Debugging IPv4 TTL, fragment reassembly, checksum calculations
ipv6 Debugging IPv6 extension handling, neighbor unknown for clientgateway6
ip Debugging for subnet matching
tcpudp Debugging for packets not matching LB configuration
linkedlist Debugging of linked list internal containers
executor Debugging for executor settings
command Debugging of general command failures
cluster Debugging for cluster settings
port Debugging for port settings
server Debugging for server settings
config Currently unused
iface Currently unused
ifcollection Debugging network interface operations
ifinstance Debugging IP addresses associated with network interfaces
os Debugging operating system specific interfaces
semaphore Currently unused
arp Debugging ARP handling
hashtable Debugging internal hash table structure operations
neigh Debugging neighbor discovery processing
select Debugging server selection process
icmpv6 Debugging icmpv6 protocol handling
ha Debugging high availability operations
icmp Debugging icmp protocol handling
fragment Debugging IP packet fragment operations
memory Currently unused
rule Debugging for rule settings
useserver Debugging for server assignment and use counts for rules
srvset Debugging of server set weight verification
nat Debugging of NAT specific packet forwarding
err Errors detected in packets
intact Listing of packets received not matching LB forwarding configuration
discard Listing of packets discarded and not forwarded
forward Listing of packets forwarded
held Listing of packets buffered, waiting for additional data (currently unused)
pstats Packet statistics (reported every 1/2 second)
rmreap Debugging of memory releases (garbage collection)
error Forwarding errors, default trace specification
warn Issues which may prevent forwarding but are currently not causing errors
Usage examples:
Individual trace specification:
lbcommand<.sh> 0 set logmask pstat
Multiple trace specifications:
lbcommand<.sh> 0 set logmask pstat+rmreap
Add trace specification to existing tracing:
lbcommand<.sh> 0 set logmask +warn
Remove trace specification from existing tracing:
lbcommand<.sh> 0 set logmask -pstat
Restore tracing to default:
lbcommand<.sh> 0 set logmask error
Disable all executor tracing:
lbcommand<.sh> 0 set logmask 0
View current executor tracing:
lbcommand<.sh> 0 get
Unix example:
/opt/IBM/WebSphere/Edge/ULB/servers/bin> ./lbcommand.sh 0 get
LBStackCommand: 0 get
executor key: (CONFIG) 0/0/0/0 action: 3(GET) rc: 1(SUCCESS)
  addable: n/a
  setable:
    logmask: 0x200 ( error )
    (ha)role: 1 (SOLO)
    (ha)takeoverstrategy: 3 (undef)
    (ha)replicatestrategy: 0 (NOREP)
    (ha)reachscore: 0
    (ha)timeout: 2
    (ha)reachaddrs: 0 0 0 0 0 0 0 0 0 0 0 0
    (ha)reachstatus: 0 0 0 0 0 0 0 0 0 0 0 0
    (ha)port: 0
    (ha)manualtakeoverrequested: 0
    nfa: 1.2.3.4
    clientgateway: 0.0.0.0
    clientgateway_ipv6: 0:0:0:0:0:0:0:0
    clientcollocate: 0
    Legacy: cps: 0    Legacy: bps: 0    Legacy: maxclusters: 0    Legacy: maxports: 0
    Legacy: maxservers: 0
    Legacy: fintimeout: 30
    Legacy: staletimeout: 300
    Legacy: stickytime: 0
    Legacy: weightbound: 20
    Legacy: configfilename:
  getable:
    starttime: 0
    version: 8.5.5-17 - 20200624-072128 [paix614] AIX/ppc64/xlc80
    stats/ipv4: in:194 [ fwd:0 err:0 notforus:194 discard:0 ]:194 gen:0
    stats/ipv6: in:0 [ fwd:0 err:0 notforus:0 discard:0 ]:0 gen:0
    stats/arp:  in:879 [ fwd:0 err:0 notforus:1052 discard:0 ]:1052 gen:0
    stats/icmp:   in:0 [ fwd:0 err:0 notforus:0 discard:0 ]:0 gen:0
    stats/icmpv6: in:0 [ fwd:0 err:0 notforus:0 discard:0 ]:0 gen:0
    clientgwneighstat: 0x0 ( undef )
    clientgwneighstat_ipv6: 0x0 ( undef )
    (ha)state:    3 (NA)
    (ha)substate: 1 (NA)
    (ha)lastevent: 0
Trace specifications are valid only when the executor is started. The lbcommand or lb_kext (AIX™ only) command fail when the executor is not running.
When the executor is stopped and restarted, executor tracing is reset to the default of error tracing.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSAW57","label":"WebSphere Application Server Network Deployment"},"ARM Category":[{"code":"a8m50000000Cd9WAAS","label":"Edge LoadBalancer->Dispatcher issues"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5.0;9.0.0"}]

Document Information

Modified date:
02 September 2020

UID

ibm16325369

Page Feedback