Question & Answer
Question
How do you correlate a thread consuming High CPU with a particular thread stack in the native_stdout.log?
Answer
This document explains how to correlate data from the MustGather below which utilizes GlancePlus:
The following files will be used to determine the thread that has high CPU:
- hpux_glance.out
- native_stdout.log
Sample output from hpux_glance.out
| -----16:26:11 gbl cpu utilization = 93% alive processes = 213 active processes = 16 priority queue = 0.1 run queue = 0.7 context switch rate = 333 network packet rate = 1.6 disk phys io rate = 29.1 memory pageout rate = 0.0 system call rate = 41834 process fork rate = 0.0 proc completion rate= 0.4 highest syscall rate= 9332 for system call write arm compl tran rate = 0.0 Process name pid ktid pri cpu% scall% dsptch cpu-swtch shell-cmd ------------ --- ---- --- ---- ------ ------ --------- --------- java 11671 5134935 154 2 0 98 0 /opt/IBM/WebSphere/AppServer60 java 11671 5551377 223 82 8 334 0 /opt/IBM/WebSphere/AppServer60 |
1. The hpux_glance.sh script outputs 13 times every 25 seconds and tracks CPU usage. Each section is separated by a timestamp. CPU usage is indicated in the cpu% column. By default it displays only java processes/threads.
2. The pid column contains the process ID. If the PID is for an application server, the
<SERVER_NAME>.pid contains the matching PID. You can find this file in the server logs directory. In the above example, the PID is 11671.
3. The ktid column identifies an individual thread. In this example, the thread 5551377 is utilizing 82% of the CPU.
Sample thread dump from native_stdout.log
"WebContainer : 3" daemon prio=10 tid=03025200 nid=92 lwp_id=5551377 runnable [0x5e4e4000..0x5e4e34f0]
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at com.ibm.ejs.ras.WrappingFileOutputStream.write(WrappingFileOutputStream.java:362)
- locked <67acb728> (a com.ibm.ejs.ras.WrappingFileOutputStream)
...
4. Search the native_stdout.log for the ktid. The ktid has a one-to-one correspondence with the lwp_id. In the example, the matching thread has lwp_id equal to 5551377.
5. Review the thread dump to reveal the methods most likely causing the high CPU.
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"100% CPU Usage","Platform":[{"code":"PF010","label":"HP-UX"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21367002