Troubleshooting
Problem
The Tomcat service can be up and running from the backend but the User Interface (UI) is not available for the users, resulting in the disruption of the availability of QRadar.
Symptom
The following warnings are seen in the /var/log/qradar.error related to "Broker not responding":
WARNING [localhost-startStop-1] Broker not responding [HELLO(10)] for 120 seconds. Still trying...
WARNING [W2003]: Broker not responding [COMMIT_TRANSACTION(46)] for 360 seconds. Still trying...
WARNING [W2003]: Broker not responding [CREATE_SESSION(68)] for 840 seconds. Still trying...
The following stack trace might be seen from Tomcat in /var/log/qradar.error:
Aug 01 00:44:11 console tomcat[30248]: isClosed true
Aug 01 00:44:11 console tomcat[30248]: isConnectedToHABroker false
Aug 01 00:44:11 console tomcat[30248]: isQueueConnection false
Aug 01 00:44:11 console tomcat[30248]: isStopped true
Aug 01 00:44:11 console tomcat[30248]: isTopicConnection false
Aug 01 00:44:11 console tomcat[30248]: protocolHandlerIsClosed false
Aug 01 00:44:11 console tomcat[30248]: readChannnelIsClosed false
Aug 01 00:44:11 console tomcat[30248]: recoverInProcess false
Aug 01 00:44:11 console tomcat[30248]: readChannnelReceivedGoodByeReply false
Aug 01 00:44:11 console tomcat[30248]: reconnecting false
Cause
The problem originates because of the IMQ process, which is responsible for the communication between components on a managed host.
This process overloads its memory and fails to fulfill the communication between components. This problem can be reviewed by looking at the IMQ logs in /opt/openmq/mq/var/instances/imqbroker/log:
[B1089]: In low memory condition, Broker is attempting to free up resources
[B1088]: Entering Memory State RED from previous state ORANGE - allocated memory is 385420K, 98% of total memory used
[B1088]: Entering Memory State ORANGE from previous state RED - allocated memory is 359344K, 91% of total memory used
Resolving The Problem
The issue is fixed by restarting hostservices in order to restart the IMQ process.
Note: The restart of QRadar services affects other functionalities such as correlations, searches, offenses creation, and other functionalities. Check the following link for more information QRadar: Core services and the impact of restarting services.
Follow this sequence of commands to fix this problem:
- SSH into the QRadar console.
Stop hostcontext:systemctl stop hostcontext- Stop Tomcat:
systemctl stop tomcat - Restart hostservices:
systemctl restart hostservices - Start Tomcat:
systemctl start tomcat - Start hostcontext:
systemctl start hostcontext
Result
Access through the graphical interface is restored. If the error persists, contact QRadar Support.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSV4BL","label":"IBM QRadar"},"ARM Category":[{"code":"a8m500000008YT1AAM","label":"ATS-Infrasec-\u003ESiteProtector-\u003EUI"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
14 November 2022
UID
ibm16838721