Troubleshooting
Problem
How do you use recon ps to view logs for QRadar applications?
Environment
In QRadar Version 7.3.2 and later the support tool for application troubleshooting is named recon. Recon replaces the QRadar V7.3.1 and earlier troubleshooting application named qapp_utils730.py.
Resolving The Problem
Recon is a tool designed to aid the troubleshooting of containers and container management on the QRadar Console or App Host. Recon features multiple commands for this purpose. Similar to tools like ps and docker ps, recon ps allows you see an overview of what containers are currently running on the system and available properties for them. Recon uses integration with the DrQ tool to diagnose problems that can occur and show a brief chart. This command also lists possible remediation for each potential error found. Due to API limitations, Recon does not know what managed host it's being run on. Recon lists what applications exist according to the QRadar API, and also what containers are currently running on the current host. They might not be the same set. When given an app ID, or a combination of workload, service, and container, recon connect allows you to run a specific command inside of a specific container.
Procedure
- Using SSH, log in to the QRadar Console as the root user.
Note: The following commands must be run on the host where the Applications are running. If you are running an App Host, you need to SSH to it to run the commands. - To locate the application ID for your app, type the following command.
A list of installed applications and their App-ID values are output to the screen. The App-ID is a unique numeric value. Administrators can use the numeric App-ID value to connect to the container for a specific application./opt/qradar/support/recon ps
If no issues are detected, the recon command output might look like the following example:/opt/qradar/support/recon ps App-ID Name Managed Host ID Workload ID Service Name AB Container Name CDEGH Port IJKL 1151 QRadar Log Source Management 53 apps qapp-1151 ++ qapp-1151 +++++ 5000 ++++ 1201 User Analytics 53 apps qapp-1201 ++ qapp-1201 +++++ 5000 ++++ 1102 QRadar Assistant 53 apps qapp-1102 ++ qapp-1102 +++++ 5000 ++++ 1103 QRadar Use Case Manager 53 apps qapp-1103 ++ qapp-1103 +++++ 5000 ++++ Legend: Symbols: n - Not Applicable - - Failure * - Warning + - Success Checks: Service: A - Service exists in the workload file B - Service is set to started Container: C - Container is in ConMan workload file D - Container environment file exists E - Container image is in si-registry G - Container Systemd Units are started H - Container exists and is running in Docker Port: I - Container IP are in firewall main filter rules J - Container IP and port is in iptables NAT filter rules K - Container port has routes through Traefik L - Container port is responsive on debug path - To connect to the app container, type the following command.
A shell is opened to the application's container. Administrators can browse this directory to review files, logs, or configurations for the application./opt/qradar/support/recon connect 1005
- To review an application log for connection errors, type the following command.
less /store/docker/volumes/qapp-<appID>/log/app.log - If you want to do a test connection from QRadar resilient Application to another server (like proxy, auth server) to check the reachability, run the following command:
Example output.sh-4.4$ curl -v telnet://X.X.X.11:443* Rebuilt URL to: telnet://X.X.X.11:443/ * Trying X.X.X.11... * TCP_NODELAY set * Connected to X.X.X.11 (X.X.X.11) port 443 (#0)
Note: Do not use the telnet command inside the application container to perform the reachability test as the telnet command is not available inside the application container.
Possible errors returned.
If the following message is returned, confirm that IBM QRadar Console UI Login page is accessible.
Unable to communicate with API. Received error: An API error occurred. The HTTP status code of 404 was returned. The API returned the body of:
&ps.StatusResult{Check:(*ps.StatusCheck)(nil), Message:"error getting response for url https://qradar-qr750.local:9000/v1/api/workloads: Get https://qradar-qr750.local:9000/v1/api/workloads: dial tcp xxx.xxx.xxx.xxx:9000: connect: connection refused", Remediation:"", Value:1}
App-ID Name Managed Host ID Workload ID Service Name Container Name Port
0 Failed to decode workloads - 0
Legend:
Symbols:
n - Not Applicable
- - Failure
* - Warning
+ - Success
Checks:
Service:
A - Service exists in the workload file
If an application fails to start the following message is displayed along with the remediation steps required.
/opt/qradar/support/recon ps
App-ID Name Managed Host ID Workload ID Service Name AB Container Name CDEGH Port IJKL
1151 QRadar Log Source Management 53 apps qapp-1151 ++ qapp-1151 +++++ 5000 ++++
1201 User Analytics 53 apps qapp-1201 ++ qapp-1201 +++++ 5000 ++++
1103 QRadar Use Case Manager 53 apps qapp-1103 +- qapp-1103 +-+-- 5000 ----
1102 QRadar Assistant 53 apps qapp-1102 ++ qapp-1102 +++++ 5000 ++++
Legend:
Symbols:
n - Not Applicable
- - Failure
* - Warning
+ - Success
Checks:
Service:
A - Service exists in the workload file
B - Service is set to started
Container:
C - Container is in ConMan workload file
D - Container environment file exists
E - Container image is in si-registry
G - Container Systemd Units are started
H - Container exists and is running in Docker
Port:
I - Container IP are in firewall main filter rules
J - Container IP and port is in iptables NAT filter rules
K - Container port has routes through Traefik
L - Container port is responsive on debug path
Remediations:
B on Service qapp-1103:
The application is not started.
Go to https://ibm.biz/recon_doc for application troubleshooting information. Choose your version of QRadar from the 'Change version or product' drop-down menu.
D on Container qapp-1103:
Config file is missing.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.
G on Container qapp-1103:
The config file for container qapp-1103 was not found at /etc/conman/container@14814818600457963407.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.
H on Container qapp-1103:
The app was not found in docker.
1. Put conwrap into debug mode by typing 'conman-support set-config -p CONWRAP_LOG_LEVEL -v DEBUG apps qapp-1103 qapp-1103'.
2. Check the logs at 'journalctl -u container@14814818600457963407.service' for errors.
If the Conwrap vault token is being rejected, check that the token works, is not expired, and that vault-qrd is active.
If the dockerApps network is down:
a) Bring up the interface in ifconfig by typing 'ifconfig dockerApps up' and review /var/log/messages
b) Make sure the /etc/docker/network.d/dockerApps.txt file exists.
K on Port 5000:
Unable to connect to the container qapp-1103 through traefik.
1. Ensure that the traefik service is running by typing 'systemctl is-active traefik' and ensure it started without errors by typing 'journalctl -u traefik'.
2. Ensure that iptables has no rules which might be blocking communication to port 14433.
3. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.
L on Port 5000:
Unable to connect to the container qapp-1103 on the debug endpoint: /qapp-1103/flask with error: Received an invalid HTTP response from Traefik endpoint on: /qapp-1103/flask/debug
1. Ensure that the traefik service is started by typing 'systemctl is-active traefik'.
2. Ensure that iptables has no rules which might be blocking communication to the container's port 5000.
3. Ensure that the web service inside the container is active by using the recon support tool to enter the container and inspect the supervisord logs in 'recon connect apps qapp-1103 qapp-1103 cat /store/log/'.
4. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
09 May 2024
UID
ibm11079655