MustGather: Troubleshooting TM1 Application Tier Performance Issues

Problem

If your TM1 Web application experiences a perceived hang, or performance issues, there are a number of diagnostic data files that are useful for diagnosing the problem. This MustGather document describes the steps and settings that are required to be in place in order to generate those diagnostic data files.

Collecting Data

To troubleshoot TM1 Web Application Tier performance issues prepare to gather the following information - either by supplying existing files or enabling additional logging - that should then be supplied from the time (or shortly after) the performance issue is happening:

1. Environment information:

   1. TM1 Server and TM1 Web software versions

   2. Operating System version

   3. Optionally, include any server hardware or Virtual Machine specification, i.e. CPUs and RAM, that may be available

   4. Provide the tm1web_config.xml file, that can be found in the \tm1_64\webapps\tm1web\WEB-INF\configuration folder

2. Set the following Loggers to DEBUG level, by referring to these instructions to edit the log4j.properties file (the logging settings are dynamic - any changes you make will be detected while the service is running, but wait a minute or so before replicating the issue and then prepare to supply the tm1web.log file, that can be found in the \tm1_64\webapps\tm1web\WEB-INF\logs folder):

   1. log4j.logger.com.ibm.cognos=DEBUG

   2. log4j.logger.com.cognos=DEBUG

   3. log4j.logger.com.ibm.cognos.tm1=DEBUG

3. Garbage Collection logging should be enabled for TM1 Web performance issues (the performance impact of GC logging is negligible so, if GC Logging is not already enabled, enable it before reproducing the issue):

   1. To enable GC logging see the following instructions - once GC logging is enabled refer to the jvm.options file, that is located in the \tm1_64\wlp\usr\servers\tm1\ folder, for the -Xverbosegclog line which defines the location of the required log file and then prepare to supply the file

4. Dumps are very useful for diagnosing hangs and memory and performance issues and a Javacore Thread Dump or System Core Dump can be manually invoked using the following instructions while the performance issue is happening:

   1. Background: These steps assume use of Planning Analytics and the WebSphere Liberty Profile. Using the server command included with Planning Analytics to create a dump means that a) there is no need to identify the OS PID of the specific java.exe process, b) there is a platform-agnostic method being used and c) this command stops all Java threads to reduce potential corruption of the resulting dump

   2. Important: For a System Core Dump the output file can be many GBs in size. If disk space is not an issue then use the server command as shown in the steps below. If disk space is an issue then omit the parameter --include=system when executing the server command (which will result in smaller output as the larger .dmp file will not be written along with the .txt file)

   1. Windows:

      1. Open a Command Prompt as Administrator

      2. Navigate to the wlp directory where Planning Analytics was installed (e.g. \ibm\cognos\tm1_64\wlp\bin)

      3. While the performance problem is reproducing, execute this command server javadump tm1 --include=system (this step will be repeated 3 times, 5 minutes apart)

      4. There should be an indicator of the dump being created, e.g. Dumping server tm1.

      5. After a few moments there should be a message(s) saying: Server tm1 dump complete in <file>. which will indicate the location of the created file(s), the output is of the form javacore.[date].[time].[PID].[index].txt and (if the --include=system parameter was used) core.[date].[time].[PID].[index].dmp

      6. Wait approximately 5 minutes and repeat Step iii twice more

      7. Note the location of the output files and prepare to supply them

   2. Linux / UNIX:

      1. Open a Terminal

      2. Navigate to the wlp directory where Planning Analytics was installed (e.g. /ibm/cognos/tm1_64/wlp/bin)

      3. While the performance problem is reproducing, execute this command ./server javadump tm1 --include=system (this step will be repeated 3 times, 5 minutes apart)

      4. There should be an indicator of the dump being created, e.g. Dumping server tm1.

      5. After a few moments there should be a message(s) saying: Server tm1 dump complete in <file>. which will indicate the location of the created file(s), the output is of the form javacore.[date].[time].[PID].[index].txt and (if the --include=system parameter was used) core.[date].[time].[PID].[index].dmp

      6. Wait approximately 5 minutes and repeat Step iii twice more

      7. Note the location of the output files and prepare to supply them