Fixes are available
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
When using the WebSphere asynchbeans WorkManager with the Object pool service enabled, the following error may occur in the logs. [2/13/15 2:09:22:934 GMT] 0000004a WorkTimeoutLi E ASYN0066E: The method <null> threw an Throwable: {1}. Also a memory leak related to the java/util/concurrent/ConcurrentLinkedQueue$Node holding com/ibm/ws/asynchbeans/WorkTimeoutListener objects is possible. The memory analyzer tools may report leak suspects like below. A Linked List Data Structure Detected. 212,757,328 bytes (30.13 %) of Java heap is used by 13,297,159 instances of java/util/concurrent/ConcurrentLinkedQueue$Node Contains the following object: - 13,297,158 instances of com/ibm/ws/asynchbeans/WorkTimeoutListener holding 425,509,056 bytes - Contained under array of java/util/HashMap$Entry holding 591,810,208 bytes at 0x97390bc8 Tree : com.ibm.ws.objectpool.ObjectPoolImpl @ 0x93b2ff88 | 24 | 566,417,296 | 75.00% '- com.ibm.ws.objectpool.FastObjectPoolImpl @ 0x949ca290 | 32 | 566,417,272 | 75.00% |- java.util.concurrent.ConcurrentLinkedQueue @ 0x949ca2c0 | 16 | 566,417,200 | 75.00% | |- java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x9e52a730 | 16 | 566,417,136 | 75.00% | | '- java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x9e52a740 | 16 | 566,417,120 | 75.00% | | '- java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x9e52a750 | 16 | 566,417,104 | 75.00% | | '- java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x9e52a760 | 16 | 566,417,088 | 75.00% | | '- java.util.concurrent.ConcurrentLinkedQueue$Node @ 0x9e52a770| 16 | 566,417,072 | 75.00%
Local fix
N/A
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server * **************************************************************** * PROBLEM DESCRIPTION: Asyncbeans logs ASYN0066E and can * * lead to a slow memory leak * **************************************************************** * RECOMMENDATION: * **************************************************************** The asynchbeans component uses the Object pool service to pool the WorkTimeoutListener alarm objects needed to keep track of the Work Timeout. The Object pool internally uses the unbounded queue based linked nodes structure java/util/concurrent/ConcurrentLinkedQueue for maintaing the objects pooled within it. When the work starts, a WorkTimeoutListener alarm object is obtained from the pool and likewise when the work completes, the same is returned back to the pool as part of the cleanup. However there is a timing window possible that causes the cleanup operation to be performed by both the work thread as well as the worktimeout alarm thread. This cause the pooled object to be returned back twice. Since the datastructure java/util/concurrent/ConcurrentLinkedQueue is unbounded, when the object is returned back twice, there is a chance for the Object pool to leak since the objects that are returned gets added to the tail of the queue. This APAR is applicable : a) When you notice the following error messages logged repeatedly in the SystemOut.log WorkTimeoutLi E ASYN0066E: The method <null> threw an Throwable: {1}. b) When heapdump analysis shows memory leak suspects indicating large number of java/util/concurrent/ConcurrentLinkedQueue$Node instances containing com/ibm/ws/asynchbeans/WorkTimeoutListener instances. Note : The leak can be very slow and it can take several days for the memory leak and OOM to manifest. Example : 212,757,328 bytes (30.13 %) of Java heap is used by 13,297,159 instances of java/util/concurrent/ConcurrentLinkedQueue$Node Contains the following object: - 13,297,158 instances of com/ibm/ws/asynchbeans/WorkTimeoutListener holding 425,509,056 bytes The cause of this issue is that the APAR PI05902 corrected a problem where alarms were getting fired incorrectly. That is, WorkTimeoutListener alarms were getting fired even if the work had been completed well before the timeout. However there is a possibility where cleanup can occur twice due to timings with which work gets completed and worktimeout expires. This can cause the WorkTimeoutListener to get returned back to the object pool twice leading to the accumulation of WorkTimeoutListener objects and ASYN0066E.
Problem conclusion
The asynchbeans code has been corrected to handle the cleanup of the WorkTimeoutListener alarm objects properly. The fix for this APAR is currently targeted for inclusion in fix packs 8.5.5.6, 8.0.0.11, and 7.0.0.39. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI32169
Reported component name
WEBSPHERE APP S
Reported component ID
5724J0800
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2015-01-06
Closed date
2015-04-03
Last modified date
2015-04-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE APP S
Fixed component ID
5724J0800
Applicable component levels
R700 PSY
UP
R800 PSY
UP
R850 PSY
UP
Document Information
Modified date:
28 April 2022