IBM Support

OutOfMemory error on the Impact GUI server, because of too many Derby daemon threads

Troubleshooting


Problem

The number of threads grows on the ImpactUI server, causing a native OutOfMemory condition. 

Symptom

The ImpactUI server grows unresponsive as it runs out of memory. 
On running out of memory, for this scenario, the generated javacore file will contain the following information :
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
The number of threads in the javacore file will run to thousands. Normally, the server will have hundreds of threads, not thousands.
The messages.log for the ImpactUI server will contain an error showing the problem is related to restdb, as follows:
java.sql.SQLException: Failed to start database 'restdb' with class loader com.ibm.ws.classloading.internal.AppClassLoader@84b8ecd0, see the next exception for details.

Cause

The ImpactUI server contains a derby database, called restdb, used for ActiveMQ related functionality. This database is internal to Impact. If IP resolution fails for restdb, the number of derby threads grows to cause a native OutOfMemory condition on the ImpactUI server.
ActiveMQ properties are stored in ActiveMQBroker.properties in the home directory for the ImpactUI server. The ActiveMQBroker.transportConnectorURIs property has an IP Address and port. By default, the ipaddress used is 0.0.0.0. On servers where binding to 0.0.0.0 is problematic, the restdb will not function correctly and an OutOfMemory condition can occur. 

Diagnosing The Problem

Look for errors for restdb in the messages.log in wlp/usr/servers/logs directory.

On Unix, use kill -QUIT to generate a javacore file and monitor the number of threads for the ImpactUI process. The expected number of threads is less than 500.

The server.xml trace specification (for the ImpactUI) can be set to see exactly what properties are set for ActiveMQ. The log output will be in wlp/usr/servers/logs/trace.log

<logging traceSpecification="com.ibm.tivoli.rest.event=all"/>

Resolving The Problem

The resolution is to use a Fully Qualified Domain Name (FQDN) or an actual IP address of the machine, when setting the ActiveMQBroker.transportConnectorURIs property in the ActiveMQBroker.properties file.

Change the property from this (default):

ActiveMQBroker.transportConnectorURIs=nio://0.0.0.0:16399
To this:
ActiveMQBroker.transportConnectorURIs =nio://ValidHostOrIP:SomeUnusedPort
i.e.
ActiveMQBroker.transportConnectorURIs=nio://my.ip.add.ress:16399

Or

ActiveMQBroker.transportConnectorURIs=nio://my.fully.qualified.hostname.com:16399
Changing the property requires a server restart.
With 7.1.0-TIV-NCI-FP0030, APAR IJ45801 will replace 0.0.0.0 in the ActiveMQBroker.properties automatically.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSB2GF","label":"Netcool\/Impact"},"ARM Category":[{"code":"a8m500000008ZuvAAE","label":"Impact-\u003EGUIServer"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1.0"}]

Document Information

Modified date:
22 March 2023

UID

ibm16962183

Page Feedback