How to generate, download, and analyze the dump files from your Java Liberty application running on Cloud Foundry.

Cloud Foundry (CF) is a Platform-as-a-Service (PaaS) offering on IBM Cloud. Through CF, you can deploy your own application in different languages, such as Node.js, PHP, Java Liberty, etc. 

In this article, I’m are going to share how to generate and download dump files from your Java Liberty application running on Cloud Foundry. Then I’ll show an analysis example using a tool called IBM Thread and Monitor Dump Analyzer for Java.

The analysis method might be useful if you want to investigate thread hang and performance problems. For this example, I prepared a demo app that was having a problem with performance. I then collected multiple thread dump files (javacore) to analyze why the app was running slow.

Generating dump files

In order to generate dump files, you need to install ibmcloud cli on your cmd or terminal. We will use pkill -3 java to take the snapshot of the running Java:

1. Log in to IBM Cloud and select the account that your app is running:

   > ibmcloud login

2. Specify the CF organization where your app is running:

   > ibmcloud target --cf

3. List all apps running in your organization and get the app name:

   > ibmcloud cf apps

4. Once we observe the performance issue, run the command and generate dump files. Make sure you replace the <appname> to the actual app name you obtained in Step 3:

   > ibmcloud cf ssh <appname> -c "pkill -3 java"

5. Repeat Step 4 a few more times to capture if there are any hanging threads over time.

How to download dump files

In order to download the dump files for analysis, use the following commands:

1. List dump files you generated:

   > ibmcloud cf ssh <appname> -c "ls -l dumps"

2. To download dump files, run the following command. In this example, we download javacore files:

   > ibmcloud cf ssh <appname> -i <instance_id> -c "cat dumps/<dump_file_name>" > <local_dump_file_name>

How to use IBM Thread and Monitor Dump Analyzer for Java

Follow the IBM Thread and Monitor Dump Analyzer for Java (TMDA) link and download the tool on your machine. Then, follow these steps:

1. Once it is downloaded, run the tool:

2. Go to File > Open Thread Dumps and open the javacore files. 
3. Select all javacore files and click Compare Threads:

4. Compare threads shows how the threads are running. We’ve imported 10 javacore files here, so there will be 10 columns. Each row represents a thread running when the dump was captured. Threads in green mean they are running threads, whereas ones in red means these threads are stopped or blocked:

What can we tell from the dump files?

In this example, we found threads that were running for a few minutes. We checked the detail of the thread and it turned out it is was a query to the backend database.

We checked the problematic thread in Monitor Detail. In the very first javacore collected, it showed there were three threads blocked by the long running thread. In the later javacore, we saw the number of blocked threads increased to 24.

We can conclude that the issue is due to fact that the query running against the DB had not yet finished. This is an extreme case, but you can use this tool to troubleshoot or analyze memory- or thread-related problems. 

Where to get more help

Due to the nature of PaaS, performance issues in general are outside of the IBM Cloud support scope. However, using this analysis tool, you might be able to narrow down where the cause of the problem lies. 

For any questions about Cloud Foundry or other information about the platform, please contact IBM Cloud support.

If you have questions for Java Liberty itself, please open a ticket from IBM Support portal.

The TMDA tool is provided as is without any warranty or support; however, we try to fix and enhance the tool as time permits. The tool was originally created by Jinwoo Hwang. Since Mr. Hwang’s departure from IBM, the tool is maintained as time permits by Kevin Grigorenko.