Troubleshooting
Problem
The WebSphere Application Server Java Proxy and Java On Demand Router use an asynchronous thread to write log entries to files such as the proxy.log to improve request/response performance. If there is a disk or network filesystem issue, log entries asynchronously queued in native memory may accumulate and consume excessive native memory.
Symptom
Native OutOfMemoryError (OOM) on 32-bit or exhausted physical memory and paging on 64-bit.
Cause
By design
Diagnosing The Problem
In both 32-bit and 64-bit cases, an operating system core file is required to diagnose whether or not asynchronous log entries are the cause of excessive native memory usage (in recent versions of IBM Java, a system core is automatically produced on OOM by default in addition to a PHD heapdump, javacore.txt, etc.). In the IBM Memory Analyzer Tool with the IBM Extensions for Memory Analyzer, load the .dmp file (in recent versions of IBM Java, jextract is not required) and select Open Query Browser > IBM Extensions > Java SE Runtime > DirectByteBuffers. In the main summary view, the following line shows how many live DBBs there are:
=> Sum DirectByteBuffer capacity not available for GC: 2670433848 (2.48 GB)
The DirectByteBuffers query also opens a second tab named "directbytebuffers," which lists all DirectByteBuffer objects. Click the "Show as Histogram" button to aggregate these instances. Select all histogram rows, right click and click Show objects by class > by incoming references. Expand the rows with the largest "Objects" column values. For this issue, you will see something like the following, with most DirectByteBuffers referenced by a java.util.LinkedList$Link which is referenced by com.ibm.ws.http.channel.impl.LoggerOffThread:
Class Name | Objects |
----------------------------------------------------------------------------------------------
java.nio.DirectByteBuffer | 495,472 |
|- com.ibm.ws.buffermgmt.impl.PooledWsByteBufferImpl | 484,915 |
| |- com.ibm.ws.buffermgmt.impl.PooledWsByteBufferImpl | 484,915 |
| |- java.util.LinkedList$Link | 474,161 |
| | '- java.util.LinkedList$Link | 474,166 |
| | |- java.util.LinkedList$Link | 474,166 |
| | |- java.util.LinkedList | 100 |
| | | |- com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl | 60 |
| | | |- com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl| 38 |
| | | |- com.ibm.ws.http.logging.impl.LoggerOffThread$WorkerThread | 1 |
----------------------------------------------------------------------------------------------
In the 32-bit case, the OOM javacore.txt file has three symptoms that point to the problem, although a system core file is required to prove it:
1) 1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
2) Large number and size of Direct Byte Buffer Allocations:
5MEMUSER | | | +--Direct Byte Buffers: 2,701,155,264 bytes / 496711 allocations
3) LoggerOffThread in the logData method:
3XMTHREADINFO "Thread-52" J9VMThread:0x09374200, j9thread_t:0xAB9B965C, java/lang/Thread:0xBC43AE28, state:CW, prio=5...
4XESTACKTRACE at com/ibm/ws/http/logging/impl/LoggerOffThread$WorkerThread.logData(LoggerOffThread.java:601(Compiled Code))...
Resolving The Problem
The issue must be resolved by correcting the filesystem issue (example causes include NFS errors or performance problems, disk freezes, operating system CPU starvation, etc.).
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21690814