Troubleshooting: Performance monitoring infrastructure (PMI) and performance tool problems

 

You will find common questions in this section that may help resolve your PMI problems.

1. How can PMI be enabled?

You can enable PMI through the administrative console or through the wsadmin. Instructions can be found in the Knowledge Center:

Administrative Console instructions:

https://www.ibm.com/support/knowledgecenter/en/SSEQTP_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tprf_prfstartadmin.html

wsadmin instructions:

https://www.ibm.com/support/knowledgecenter/en/SSEQTP_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tprf_command.html


2. How can you find the available counters per component?

The following Knowledge Center link contains all the components that have PMI counters available to be monitored:

https://www.ibm.com/support/knowledgecenter/en/SSEQTP_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rprf_dataorg.html

If you scroll down to the Subtopics section, you can click on the component you'd like to monitor, and it will bring up another page that contains all of the available counters.


3. How can you review the data for PMI?

There are a few options available to review the data. The following Knowledge Center links will walk you through the options:

Viewing PMI data with Tivoli Performance Viewer
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tprf_tpvperfmodules.html


Viewing Data with the Tivoli Performance Viewer
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/uprf_tpvserverview.html


Logging performance data with Tivoli Performance Viewer
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tprf_tpvlogdata.html

4. How do I know the performance impact of a counter?

For each counter, the overhead is documented as low, medium, or high in the Knowledge Center. Check the ‘Overhead’ column for each counter to see if there could be a low or high performance impact. For example, if you look at the CreateCount counter for the J2C connection pool in the link below, you can see that its overhead is low which would mean it has a low performance impact:
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rprf_datacounter3.html


5. Why is it possible to get an Out of Memory situation when special counters are enabled?

One known Out of Memory (OOM) issue related to PMI metrics is with the WebAppModules which are enabled by default. In a production environment which handles a lot of requests, it is possible
that the URLCounters of the WebAppModules can cause some high memory usage. This is because they may need to track a lot of content.

If you're running into this type of issue, you have two options:

1. Increase the max heap size

2. Disable the URL PMI counters:

In the administrative console, go to:

PMI > 'server_name' > configuration > custom > webApplications > 'application_name' > 'servlet_name' > URL Counters > disable URIRequestCount and URIServiceTime

After that save the changes and restart the server.


6. Why do you notice that some counters are grayed out even though you have already enabled PMI for that specific category?

This is often seen when you set the counters under the Configuration tab and forget to restart the server. A server restart is necessary if the counters are set under the Configuration tab. If you don't want to do a server restart, you can set the counters under the Runtime tab. You can find these tabs at the following location in the administrative console:

Monitoring and Tuning > Performance Monitoring Infrastructure (PMI) > 'server_name'


7. Can using Intelligent Management / PMI cause slowness in your environment?

High CPU usage due to intelligent management is most likely caused by the problem reported in APAR PI48619 EXCESSIVE APPENDCUSTOMSETSTRING CALLS CAUSE HIGH CPU:

http://www-01.ibm.com/support/docview.wss?uid=swg1PI48619


8. How can you use logging from Tivoli Performance Viewer (TPV)?

Refer to the following link for information on how to use logging from TPV:

https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tpvlogdata.html

You can also log Tivoli Performance Viewer data using scripting. For more information on this, please refer to the following link:

http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/txml_loggingtpvdata.html


9. Why do you just notice active Messaging Engine Counters on the server with the active Messaging engine when checking PMI for the whole cluster?

PMI Counters are lazily loaded. This means they would only be available when transactions start happening for the Messaging Engine. Hence, the counters would only be visible for a running server with an active Messaging Engine.


10. Why do you see some PMI counters with a negative value even though they should not have a negative value?

A. For WAS Liberty, it is possible to see a negative value being reported for the ActiveCount for JMX session stat counter.

B. In WAS Liberty 16.0.0.2 and older, the value of FreeConnectionCount attribute is likely to be negative when under load. It's fixed in 16.0.0.3 and newer versions.

C. In tWAS, the value of Method calls and Avg Resp Time will show a value of -1 if the following two counters are disabled:

WSEJBStats.MethodStats.MethodLevelCallCount
WSEJBStats.MethodStats.MethodLevelResponseTime

A "-1" value in this case simply means that the data of the requested
PMI stats are not available. This is most likely due to the fact that the
aforementioned two counters are not enabled so that TPV cannot fetch
the values of the method calls and response time stats.

Knowledge Center has an article on PMI levels required for different TPV
summary report:
http://www.ibm.com/support/knowledgecenter/SSAW57_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rprf_summaryreports.html

To find WSEJBStats counters in a PMI tree, please refer to the knowledge
center link below:

http://www.ibm.com/support/knowledgecenter/en/SSEQTJ_7.0.0/com.ibm.websphere.javadoc.doc/web/apidocs/com/ibm/websphere/pmi/stat/WSEJBStats.html

11. How do you use the Performance Servlet, and how do you send the results to IBM?

The Performance Servlet (PerfServlet) is used for simple end-to-end retrieval of performance data that any tool, provided by either IBM or a third-party vendor, can handle. This servlet provides a way to use an HTTP request to query the performance metrics for an entire WebSphere Application Server administrative domain.

The performance data can be output as an XML document where the leaves of the structure provides the actual observations of performance data and the paths to the leaves that provide the context.

For more information about the PerfServlet, please refer to the following IBM Knowledge Center page and its subtopics:

https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_devprfservlet.html

The XML document can be then sent over to IBM for support. Please follow the instructions on how to send documents to IBM support from the following link:

http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21153852


12. How do you log data for longer periods of time and then view that data?

The Tivoli Performance Viewer (TPV) is designed for a quick monitoring viewer for performance data. It is not recommended as a long-term monitoring solution. TPV runs as a Deployment Manager process, and if it is run for longer periods of time, it may impact the overall performance of the Deployment Manager.

To have log data monitored for longer periods of time, we recommend that you develop your own monitoring applications, which we have provided the necessary APIs for. Please refer to the following IBM Knowledge Center link, for more information:

https://www.ibm.com/support/knowledgecenter/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tprf_pmi.html

What to do next
If the preceding troubleshooting steps did not help solve your problem, see the MustGather for PMI problems to continue investigation.