Troubleshooting
Problem
Collect troubleshooting data for problems with IBM Security SOAR AppHost. Gathering this information before you contact IBM support helps familiarize you with the troubleshooting process and save you time.
Environment
IBM Security QRadar SOAR configured with App Host or Edge Gateway
Resolving The Problem
Collecting system details and logs on the App Host or Edge Gateway Server as well as the SOAR server
Process for versions prior to v1.13.1.582:
sudo appHostPackageLogs # - (Output is: Logs have been gathered and stored in apphost_logs_<apphost_name>_20210908_125310.tar.gz)
sudo kubectl get pods -A > appHostPods.txt
sudo kubectl describe nodes > appHostNodes.txt
sudo kubectl get events > appHostEvents.txt
sudo kubectl describe pod coredns -n kube-system > appHostCoreDNS.txt
sudo kubectl get pods -A -l apps.isc.ibm.com/app-type=app -L app.kubernetes.io/instance > pods.txt
sudo manageAppHost showconfig # - (choose the app host that the offending application is deployed to)
sudo manageAppHost checkconnection > check_connection.txt
sudo cat /etc/systemd/system/k3s.service.env > k3s_proxy_env.txt
sudo env | grep -i proxy > proxy_env.txt
sudo netstat -nr > netstat.txt
sudo cat /etc/hosts > hosts.txt
sudo hostname > hostname.txt
sudo kubectl cluster-info dump >dump.txt
sudo kubectl top node >top_node.txt
sudo firewall-cmd --zone=trusted --list-interfaces > interfaces.txt
sudo firewall-cmd --list-ports --zone=trusted > zone_trusted.txt
sudo firewall-cmd --list-ports --zone=public > zone_public.txt
Process for v1.13.1.582 and later:
The App Host MustGather tool, apphostmg.sh, is used to collect data from the app host system for analysis and troubleshooting issues.
During the data collection, by default, the user is offered the option to manually edit and desensitize all of the app.config files BEFORE they are stored for analysis.
Steps
1. Using SSH from a client machine, log in to the App Host machine by using the appadmin ID.
2. Locate the apphostmg.sh script under the /usr/bin directory:
which apphostmg.sh
Note: if you are not seeing it you might not be using apphost 1.13.1.582 or later.
3. Run apphostmg.sh:
sudo apphostmg.sh
Note: The --skip-desens option can be passed to the apphostmg.sh command to bypass the stage where files can be desensitized:
sudo apphostmg.sh --skip-desens
The following shows an example of the output with desensitization:
APP HOST MUSTGATHER TOOL
========================
=== NOTE! DURING THE STEP WHERE app.config FILES ARE COLLECTED YOU WILL BE PROMPTED TO
=== TAKE THE OPPORTUNITY TO EDIT AND REMOVE ANY SENSITIVE TEXT AND RE-SAVE THE FILES.
Press any key to continue...
====== END OF LOG CAPTURE FOR APP HOST .... ======
Collecting pod info for ....
Collecting pod info for ....
Collecting pod info for ....
Collecting pod info for ....
====== END OF POD INFO CAPTURE ======
=== NOTE! - THE FOLLOWING STEP WILL OPEN EACH app.config IN A TEXT EDITOR. ====
==== YOU CAN EDIT AND SAVE ANY OF THE FILES TO REMOVE ANY SENSITIVE TEXT. ====
Press any key to continue...
4. Follow the instructions shown when prompted. You are given the opportunity to censor any sensitive data within the app.config files.
The following shows an example app.config file.
[fn_someapp]
api_token = ****************
polling_interval_sec = 60
max_polling_wait_sec = 600
# uncomment proxies needed to access the someapp
#http_proxy=192.x.x.x
#https_proxy=10.x.x.x
The following example shows console output:
Running AppHost connection checks...
Collecting OS and cluster data...
./
./apphostlogs/
...
./check_connection.txt
./system_info.txt
./os-release.txt
./cpuinfo.txt
./meminfo.txt
./diskusage.txt
./appHostPods.txt
./appHostDeploy.txt
./appHostNodes.txt
./appHostEvents.txt
./appHostCoreDNS.txt
./pods.txt
./top-node.txt
./top-pods.txt
./nodes-yaml.txt
./cluster-dump.txt
./k3s_proxy_env.txt
./proxy_env.txt
./redhat_default_env.txt
./netstat.txt
./network_services.txt
./hosts.txt
./sysconfig_network.txt
./hostname.txt
./resolv.conf
./firewall_interfaces.txt
./firewall_zone_trusted.txt
./firewall_zone_public.txt
./firewall_all_rules.txt
./journalctl.txt
====== FILE apphostmustgather-20230509-093728.tgz CREATED FOR ANALYSIS ======
5. The apphostmustgather-<YEAR><MONTH><DATE>-<HOUR><MINUTE><SECOND>.tgz file is created and can be provided to IBM Support for analysis.
Note: to provide logs to IBM for a particular issue, open a case.
Collecting logs from the SOAR Server
$ sudo resPackageLogs -l <num_of_days>
Related Information
Document Location
Worldwide
[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"ARM Category":[{"code":"a8m0z0000001jTpAAI","label":"Integrations-\u003EAppHost"}],"ARM Case Number":"TS004246044","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
21 December 2023
UID
ibm16338707