IBM Support

QRadar: Information to gather in case of Out of memory(OOM) errors

Question & Answer


Question

When the QRadar Console reports an out of memory error, what information is necessary to gather?

Cause

Out of memory errors trigger when a process tries to request or allocate more memory than the defined maximum memory allocation for that process.
Symptoms
When an out of memory error triggers, the following messages are received in
GUI
GUI error
Event processor
[main] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR] [NOT:0030003100][IPADDRESS/- -] [-/- -]Out of memory discovered
 
Note
The bold IPADDRESS is the host that is having the out of memory error.
Logs
In /var/log/qradar.log
OutOfMemoryMonitor[12345]: Starting out-of-memory monitoring (enabled: yes)…
OutOfMemoryMonitor[12345]: Discovered out-of-memory error for [servicename] process.
OutOfMemoryMonitor[12345]: out-of-memory TYPE: [servicename], PID: 123456
OutOfMemoryMonitor[12345]: Discovered out-of-memory error for [servicename](type: [servicename], pid: 123456).
[servicename][123456]: JVMDUMP030E Cannot write dump to file /store/jheap/[servicename]/[servicename].*.dmp: File exists
 
 
Note
The bold [servicename] is the process having the out of memory error.

Answer

Review the following common causes of out of memory errors:
In case that none of the previous documentations solved the out of memory error, gather the following information before you restart the service that is affected and create the support case:
  1. Errors found in the /var/log/qradar.log
     
  2. Check /store/jheap/[servicename]/ to see whether a dump file name: "[servicename].system.dmp" or "[servicename].javacore.dmp" was created at the same date and time of the out of memory error reported:
    # cd /store/jheap/[servicename]/
    # ls -lh
     
    1. If the dump file does not match the date and time, and there are error messages like in the next example in /var/log/qradar.log:
      [servicename][123456]: JVMDUMP030E Cannot write dump to file /store/jheap/[servicename]/[servicename].*.dmp: File exists
       
      Erase the old dump file with the following command to allow a new dump to be created:
      # rm -r /store/jheap/[servicename]/[servicename].*.dmp
       
    2. If the dump file was created and it matches the date and time, create the following folder:
      # mkdir /store/ibm_support/[servicename-date]
       
    3. Compress the file by using the gzip tool and move the file:
      # gzip [servicename].*.dmp
      # mv /store/jhead/[servicename]/file.gz /store/ibm_support/[servicename-date      ]
       
  3. If the dump created a core.[pid] file in the same directory /store/jheap/[servicename]/, note the timestamp and file size, but it is not necessary to gather.
     
  4. Gather get_logs from console and affected managed hosts. For more information, see How to collect log files for QRadar support from the user interface.
     
  5. If you have a security policy about providing logs to support teams, you can sanitize the logs before you upload them to your support case. For more information, see QRadar: Sanitizing logs before you open a support case.
     
  6. Create a case with support for the specific service that is having the out of memory error and provide the information gathered.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
31 January 2023

UID

ibm16568037

Page Feedback