IBM Support

QRadar: Tomcat Can Restart From Many Offenses

Troubleshooting


Problem

When you have many Offenses in QRadar, some Dashboards, reports, or searches can restart Tomcat.

Symptom

In /var/log/qradar.log:
grep -i txsentry /var/log/qradar.error | less +G
Will return content similar to:
com.q1labs.hostcontext.tx.TxSentry: (..) Found a process on host {CONSOLE}: tomcat, pid={PID}, TX age={t} secs
TX on host {CONSOLE}: pid={PID} age={T} IP={loopback} port={port} locks={N} query='SELECT DISTINCT t0.id, t0.attackerCount, (..)

Cause

TXSentry is a feature of stopping a service that is taking too long on a task is working as designed to protect the overall system.

Diagnosing The Problem

psql -U qradar -c "select * from q_table_size;"
If you see "offense_attacker_target_link" over 1 GB, you are hitting this issue. In addition, anything over 1 GB needs to be cleaned up for performance issues but tables such as Reference Sets might not be related depending on the tables indicated in the error.

Resolving The Problem

Cleaning up tables, decreasing time to live, or decreasing retention periods are an option for some environments, but the simplest option is to increase the default timeout:
  1. Log in to the Web Console.
  2. Click Admin tab.
  3. Select System Settings.
  4. Select: Advanced. 
  5. Search for Transaction Max Time Limit and set it to 30 minutes.

    Important:
    QRadar continues to collect events when you deploy the full configuration. When the event collection service must restart, QRadar does not restart it automatically. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
  6. From the Admin tab, click Advanced >  Deploy Full Configuration after making this change.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"TS004076797","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3.2;7.3.3;7.4.0;7.4.1"}]

Document Information

Modified date:
08 December 2020

UID

ibm16324601