Troubleshooting
Problem
When reviewing thread dumps taken against the WebSphere Application Server, there are several thread states that appear to be hanging, but are actually in an idle state. This document will explain what threads would usually be misidentified as a problem thread when actually it's idle.
Diagnosing The Problem
During high CPU, threads are likely performing operations rapidly, and as a result thread dumps taken that are correlated with the CPU utilization data will map the highest CPU usage thread to a thread that is actually in a wait/idle state. The thread dump is only capturing a single instance of all the running threads and rapidly executing operations may have completed their workload just before the javacore was taken, and new workload likely was being handled by a different thread.
Sometimes a hung thread is misinterpreted because it is in a wait state and therefore it's not seen as changing when comparing multiple thread dumps.
Sometimes a hung thread is misinterpreted because it is in a wait state and therefore it's not seen as changing when comparing multiple thread dumps.
Resolving The Problem
Since these threads are already idle, they are not the cause of any high CPU or hangs that may have been detected.
If diagnosing high CPU, you will need to look at trends in all thread dumps captured to determine what is consuming the CPU usage, or capture profiling data (for example from HealthCenter).
Example Idle Thread Stacks
Idle Thread (Waiting for Work)
Usually seen in WebContainer and Default Executor threads, but can be seen elsewhere which indicates a thread waiting for new work to arrive:
Traditional WebSphere Application Server:
at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:222)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2127)
at com/ibm/ws/util/BoundedBuffer$GetQueueLock.await(BoundedBuffer.java:285)
at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:424)
at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:817)
at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:934)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1704)
Liberty:
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:218(Compiled Code))
at com/ibm/ws/threading/internal/BoundedBuffer.waitGet_(BoundedBuffer.java:176(Compiled Code))
at com/ibm/ws/threading/internal/BoundedBuffer.take(BoundedBuffer.java:647(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1085(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java/lang/Thread.run(Thread.java:818)
Idle Thread (Waiting for Work)
Similar thread stacks without the concurrent locks were seen in earlier versions of WebSphere Application Server, which are also waiting for work (or may be seen on some non-WebContainer threads that have a thread pool):
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:192)
at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:543)
at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:819)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1544)
z/OS
Traditional WebSphere Application Server:
at com/ibm/ws390/orb/CommonBridge.getWork(Native Method)
at com/ibm/ws390/orb/CommonBridge.getAndProcessWork(CommonBridge.java:653)
at com/ibm/ws390/orb/CommonBridge.runApplicationThread(CommonBridge.java:614)
at com/ibm/ws/util/ThreadPool$ZOSWorker.run(ThreadPool.java:2127)
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"100% CPU Usage","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0.0.0;8.5.5;8.5;8.0;7.0;6.1","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
21 August 2020
UID
swg21672204