Troubleshooting
Problem
In QRadar version 7.4.x and later, application "STOPPED" errors are reported in the output of the command /opt/qradar/support/recon ps.
Symptom
In the ouput of the command /opt/qradar/support/qappmanager QRadar Log Source Management application with app instance ID 1051 is STOPPED and the Pulse app with app instance ID 1052 is RUNNING.
/opt/qradar/support/qappmanager
APP DEFINITIONS (SIO=Single Instance Only, MTS=Multi-tenancy Safe):
ID | Name | Version | Status | Installed | Memory | Instances | SIO | MTS | Errors
------------------------------------------------------------------------------------------------------------------------
1001 | QRadar Log Source Management | 6.1.0 | COMPLETED | 2021-01-13 10:37 | 100 | 1 | t | t |
1002 | pulse.full_name | 2.2.5 | COMPLETED | 2021-01-13 10:40 | 350 | 1 | t | t |
APP INSTANCES (IID=Instance ID, DID=Definition ID, MHN=Managed Host Name, AHT=Application Host Type, SP=Security Profile):
IID | DID | Name | Status | Task Status | Installed | MHN | AHT | Memory | SP | Errors
-----------------------------------------------------------------------------------------------------------------------------------------------
1051 | 1001 | QRadar Log Source Management | STOPPED | COMPLETED | 2021-01-13 10:37 | qradar | LOCAL | 0 | |
1052 | 1002 | pulse.full_name | RUNNING | COMPLETED | 2021-01-13 10:40 | qradar | LOCAL | 350 | |
Total memory used by LOCAL app instances: 350MB
OPTIONS:
0) Quit
1) Help
10) App definition - list all
11) App definition - list authorized
12) App definition - show manifest
13) App definition - cancel install
14) App definition - delete
20) App instance - list all
21) App instance - list authorized
22) App instance - create
23) App instance - start
24) App instance - stop
25) App instance - authorize
26) App instance - change authorized user
27) App instance - change security profile
28) App instance - cancel install
29) App instance - delete
30) Augmented security profiles - list
31) Admin user - add augmented security profile
32) Admin user - remove augmented security profile
40) App containers - list
Choose option:
From the recon ps output, errors are reported for the QRadar Log Source Management application with app instance ID 1051.
/opt/qradar/support/recon ps
App-ID Name Managed Host ID Workload ID Service Name AB Container Name CDEGH Port IJKL
1051 QRadar Log Source Management 53 apps qapp-1051 +- qapp-1051 +-+-- 5000 ----
1052 pulse.full_name 53 apps qapp-1052 -n 0
0 ui ui ++ ui +++++ 5000 ++++
0 graphql graphql ++ graphql +++++ 5000 ++++
Legend:
Symbols:
n - Not Applicable
- - Failure
* - Warning
+ - Success
Checks:
Service:
A - Service exists in the workload file
B - Service is set to started
Container:
C - Container is in ConMan workload file
D - Container environment file exists
E - Container image is in si-registry
G - Container Systemd Units are started
H - Container exists and is running in Docker
Port:
I - Container IP are in firewall main filter rules
J - Container IP and port is in iptables NAT filter rules
K - Container port has routes through Traefik
L - Container port is responsive on debug path
Remediations:
B on Service qapp-1051:
The application is not started.
Go to https://ibm.biz/recon_doc for application troubleshooting information. Choose your version of QRadar from the 'Change version or product' drop-down menu.
D on Container qapp-1051:
Config file is missing.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.
G on Container qapp-1051:
The config file for container qapp-1051 was not found at /etc/conman/container@6233626133204147840.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.
H on Container qapp-1051:
The app was not found in docker.
1. Put conwrap into debug mode by typing 'conman-support set-config -p CONWRAP_LOG_LEVEL -v DEBUG apps qapp-1051 qapp-1051'.
2. Check the logs at 'journalctl -u container@6233626133204147840.service' for errors.
If the Conwrap vault token is being rejected, check that the token works, is not expired, and that vault-qrd is active.
If the dockerApps network is down:
a) Bring up the interface in ifconfig by typing 'ifconfig dockerApps up' and review /var/log/messages
b) Make sure the /etc/docker/network.d/dockerApps.txt file exists.
K on Port 5000:
Unable to connect to the container qapp-1051 through traefik.
1. Ensure that the traefik service is running by typing 'systemctl is-active traefik' and ensure it started without errors by typing 'journalctl -u traefik'.
2. Ensure that iptables has no rules which might be blocking communication to port 14433.
3. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.
L on Port 5000:
Unable to connect to the container qapp-1051 on the debug endpoint: /qapp-1051/1hv5jpfueiiid with error: Received an invalid HTTP response from Traefik endpoint on: /qapp-1051/1hv5jpfueiiid/debug
1. Ensure that the traefik service is started by typing 'systemctl is-active traefik'.
2. Ensure that iptables has no rules which might be blocking communication to the container's port 5000.
3. Ensure that the web service inside the container is active by using the recon support tool to enter the container and inspect the supervisord logs in 'recon connect apps qapp-1051 qapp-1051 cat /store/log/'.
4. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.
A on Service qapp-1052:
Confirm the host ID for this app is the same as the host ID for this host. If it is, do a deploy.
Cause
Recon utility code.
Environment
QRadar 7.4.x and later.
Diagnosing The Problem
Run the command, /opt/qradar/support/recon ps with all or some applications set to STOPPED in the application instance database table.
Resolving The Problem
In the output of the command, /opt/qradar/support/qappmanager if an application's status is shown as "STOPPED". All information related to that application in the Remediations section of the command, /opt/qradar/support/recon ps can be ignored and treated as benign.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSV4BL","label":"IBM QRadar"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.4.0;7.5.0"}]
Was this topic helpful?
Document Information
Modified date:
15 February 2023
UID
ibm16836939