IBM Support

QRadar: Applications display offline mode or can fail to connect to external URLs due to an iptables rule

Troubleshooting


Problem

Administrators may experience connection issues with apps that need to communicate with external resources. This can lead to problems where these apps fail to function as intended or they may show stale information.  These apps include, but not limited to, QRadar Assistant, Threat Intelligence and Watson Advisor

Symptom

Administrators may see any of the following symptoms:
  1. Apps like QRadar Assistant, Watson, or Threat Intelligence may display a banner indicating the app is having problems connecting to X-force® Exchange.
    image 9767
  2. The app container logs indicate a problem with name resolution to an external resource.
    021-03-31 06:21:46,151 [abstract_qpylib.log] [MainThread] [INFO] - 127.0.0.1 [APP_ID/2651]
    [NOT:0000006000] Proxy.py: make_request: ERROR: HTTPSConnectionPool(host='api.xforce.ibmcloud.com', 
    port=443): Max retries exceeded with url: /lang (Caused by NewConnectionError('<urllib3.connection.
    VerifiedHTTPSConnection object at 0x7fee7da45510>: 
    Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
  3. Apps that pull data based on a regular interval, show no recent data being populated. For example. Threat Intel shows no new IOCs being updated, due to the fact that the app can not reach the intended URL.
  4. Output of  /opt/qradar/support/recon ps will show "-" against columns I,J & K.
    image 9749

Cause

The cause for the issue is that the iptables or ip6tables service on the console or the app host is down. When iptables or ip6tables is not running, this results in the necessary rules that allow apps to connect externally to become unavailable and hence the apps cannot establish a connection.

Environment

QRadar 7.3.2+

Diagnosing The Problem

  1. Check the output of the recon script to see if columns I,J & K are reporting problems. The script should be run on the box hosting the apps.
    ​#/opt/qradar/support/recon ps
    ​
  2. Using the same recon script connect to the docker container of the problematic app and try to curl to the URL that the app is meant to access (In this example xforce).
    ​[root@QradarConsole ~]# /opt/qradar/support/recon connect <app_id>
    
    sh-4.1# curl -v https://api.xforce.ibmcloud.com
    * getaddrinfo(3) failed for api.xforce.ibmcloud.com:443
    * Couldn't resolve host 'api.xforce.ibmcloud.com'
    * Closing connection #0
    curl: (6) Couldn't resolve host 'api.xforce.ibmcloud.com'
    ​
  3. Ensure you're able to resolve this URL from the console or App host, if one exists
    ​[root@QradarConsole ~]# nslookup api.xforce.ibmcloud.com
    ​
  4. Check to the status of both itpables and ip6tables service on the console and the App host, if one exists
    ​#systemctl status ip6tables
    
    #systemctl status iptables
    ​

    image 9753

Resolving The Problem

To resolve this issue, administrators need to make sure that iptables or ip6tables are running on both the console and the App host. In most cases, the service fails to start due to a bad entry as a result of adding a customized rule. When checking the status of iptables/ip6tables, it will indicate why the service failed to start and  in the event that there is a bad entry in the configuration file, it'll print out the problematic line number. From the above example, we can see that there's an invalid entry in line 23 of the iptables configuration file.
  1. Review any custom iptable entries in the following files:
    /opt/qradar/conf/iptables.pre
    /opt/qradar/conf/ip6tables.pre
    
    /opt/qradar/conf/iptables.post
    /opt/qradar/conf/ip6tables.post
  2. If the bad entry in the above mention files has been identified, correct it or comment it out and then run the following command to reload iptables
    ​#/opt/qradar/bin/iptables_update.pl
  3. Check to the status of both itpables and ip6tables service and ensure they're both running
    #systemctl status ip6tables
     ● ip6tables.service - IPv6 firewall with ip6tables
       Loaded: loaded (/usr/lib/systemd/system/ip6tables.service; enabled; vendor preset: disabled)
       Active: active (exited) since Tue 2021-05-04 10:37:18 EDT; 2h 20min ago
       Main PID: 1047 (code=exited, status=0/SUCCESS)
        Tasks: 0
       Memory: 0B
       CGroup: /system.slice/ip6tables.service
    
    #systemctl status iptables
    ● iptables.service - IPv4 firewall with iptables
       Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/iptables.service.d
               └─start_post_override.conf
       Active: active (exited) since Tue 2021-05-04 10:37:17 EDT; 2h 19min ago
       Main PID: 882 (code=exited, status=0/SUCCESS)
    

    For additional information on how to properly add custom iptable rules, see Configuring iptables rules
     

Document Location

Worldwide

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
18 May 2021

UID

ibm16430697