Troubleshooting
Problem
The WebSphere Application Server thread pools can leak ThreadLocal objects when threads are destroyed (java.lang.ThreadLocal). ThreadLocal objects can have a significant native memory component, so you can see a native memory leak if there is a ThreadLocal leak.
Symptom
The process size for the application server will grow quite large. You might also see the following symptoms:
| JVMDBG001: malloc failed to allocate … |
| Failed to fork OS thread |
| Or |
| cannot create anymore threads |
- OutOfMemoryError’s thrown
- Malloc errors in the SystemOut.log file:
- Errors creating threads:
Cause
The WebSphere Application Server thread pools can leak ThreadLocal objects when thread pool threads are destroyed. ThreadLocal objects are Java™ heap objects, but they have a native memory component which is not released until the ThreadLocal object is garbage collected. Since the ThreadLocals of the destroyed threads are not always garbage collected, you end up with a leak of native components
Environment
WebSphere Application Server versions 7.0, 8.0, 8.5.5 and 9
Diagnosing The Problem
Check javacores for high index WebContainer threads, or default thread pool threads, anything over 300 is a potential leak:
"WebContainer : 1003" (TID:0x37D62000
"Default : 338" (TID:109934D0
"TCPChannel.DCS : 303" (TID:0x4D720000
The number beside the thread name is the index, the number of threads that have been created for that thread pool.
So, there have been 1004 WebContainer threads created in the first example (starts at 0). This does not mean that there are this many still alive, as most of them will have been destroyed. An index is never reused.
"WebContainer : 1003" (TID:0x37D62000
"Default : 338" (TID:109934D0
"TCPChannel.DCS : 303" (TID:0x4D720000
The number beside the thread name is the index, the number of threads that have been created for that thread pool.
So, there have been 1004 WebContainer threads created in the first example (starts at 0). This does not mean that there are this many still alive, as most of them will have been destroyed. An index is never reused.
Resolving The Problem
Using a fixed size thread pool, will avoid thread pool shrinking, which will avoid threads from being destroyed. If they are not destroyed then no ThreadLocal objects will get left behind and the native memory leak should no longer occur.
To set a thread pool to be fixed sized, set the minimum size to be the same as the maximum size. This will eliminate the thread pool from shrinking once it has reached its maximum size. Keep the maximum pool size the same as it was previously, as it is the maximum size which is the throttle for threads. Changing the max size, would be changing the performance of the application.
Depending on which thread pool is leaking, you will need to change the pool size in the administrative console for that pool. The two most common threads pools are the WebContainer and the Default thread pool, and instructions for each are below. The TCPChannel.DCS thread pool is also one which may cause problems, but is not as common.
IBM support will be able to tell you which thread pool to set to a fixed size. You will also be fine in setting both the Default thread pool and the WebContainer thread pool to be fixed size pools, if you are not sure which one to change.
Web container thread pool:
The application server will need to be restarted for the setting to take effect.
Default thread pool:
The application server will need to be restarted for the setting to take effect.
To set a thread pool to be fixed sized, set the minimum size to be the same as the maximum size. This will eliminate the thread pool from shrinking once it has reached its maximum size. Keep the maximum pool size the same as it was previously, as it is the maximum size which is the throttle for threads. Changing the max size, would be changing the performance of the application.
Depending on which thread pool is leaking, you will need to change the pool size in the administrative console for that pool. The two most common threads pools are the WebContainer and the Default thread pool, and instructions for each are below. The TCPChannel.DCS thread pool is also one which may cause problems, but is not as common.
IBM support will be able to tell you which thread pool to set to a fixed size. You will also be fine in setting both the Default thread pool and the WebContainer thread pool to be fixed size pools, if you are not sure which one to change.
Web container thread pool:
- In the administrative console click Servers > Application Servers > server_name.
- Near the bottom of the page, open Additional Properties, Thread Pools.
- Click on WebContainer.
- Change the Minimum Size field to whatever the Maximum is set to, probably 50.
- Ensure that the "Allow thread allocation ?" checkbox is not checked.
- Click OK and then save the configuration.
The application server will need to be restarted for the setting to take effect.
Default thread pool:
- In the administrative console click Servers > Application Servers > server_name.
- Near the bottom of the page, open Additional Properties, Thread Pools.
- Click on Default.
- Change the Minimum Size field to whatever the Maximum is set to, probably 30.
- Ensure that the "Allow thread allocation ?" checkbox is not checked.
- Click OK and then save the configuration.
The application server will need to be restarted for the setting to take effect.
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Out of Memory","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
18 November 2019
UID
swg21368248