IBM Support

QRadar: Managed Host connectivity fails due to an unknown network device translating the connection

Troubleshooting


Problem

A Managed Host connection fails to be established from the Console due to a NAT configuration translating the connection and no NAT Group is configured. The addition process and tunnel connection may fail in certain scenarios.

Symptom

The symptoms may vary depending on which host originates the connection and which way (inbound or outbound) the network device is doing the translation. Some scenarios and their symptoms are:
Scenario #1 - Console connection to the Managed Host being translated.
A Managed host can't be added to the deployment. The following error message appears in the /var/log/qradar.error file.
 
[hostcontext.hostcontext] com.q1labs.configservices.capabilities.AddHost: [ERROR][-/- -]SSH connection or SSH command execution failed. The ip of the host is: <Managed Host IP>
This also affects the tunnel creation fails when an attached managed host gets encryption enabled.
Scenario #2 - Managed Host connection to the Console being translated.
Managed hosts already part of the deployment report issues when connecting back to the Console in some services.

Example of ecs-ec to ecs-ep affectation:
 
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [WARN] [NOT:0000004000][-/- -]IO Error
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] java.io.IOException: Broken pipe
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [WARN] [NOT:0000004000][-/- -]Unable to connect to server

Example of replication affectation:
 
<Managed Host IP> replication[49309]: Response is empty:  .
<Managed Host IP> replication[49309]: Could not download database updates: 0
<Managed Host IP> replication[49309]: Will attempt with the next address.
<Managed Host IP> replication[49309]: Response is empty: 0 .
<Managed Host IP> replication[49309]: Failed to download updates.  Is the console online?

Cause

An unknown network device to the administrator is doing bi-directional NAT translating between the connections to the Console and the Managed Host. This IP address is not configured in the deployment, therefore, gets blocked by the iptables service in the remote host.

Environment

  1. Console and Managed Host on different subnets causing the connection to pass throughout network devices doing NAT.
  2. No QRadar NAT Group configured in the deployment. 

Diagnosing The Problem

Note: Some of the steps include stopping the iptables service on the Console or Managed Host. Stopping the iptables service does not cause downtime to QRadar®.
Scenario #1 - Console connection to the Managed Host being translated.
  1. Use an SSH session to log in to the Console as the root user.
  2. Use an SSH session to connect to the affected Managed Host.
    Note: If an SSH can't be established then connect to the Integrated Management Module (IMM) or XClarity Controller (XCC) WebUI and start a connection to the appliance.
  3. Stop the iptables service in the Managed Host. 
    systemctl stop iptables
  4. SSH from the Console to the Managed Host.
  5. Run the "who" command to verify the connections established. 
    who
root     pts/13       2021-02-26 08:45 (<IP translated>)
  6. Start the iptables service back. 
    systemctl start iptables
Scenario #2 - Managed Host connection to the Console being translated.
  1. Log in to the Console host as the root user.
    Note: If an SSH can't be established then connect to the Integrated Management Module (IMM) or XClarity Controller (XCC) WebUI and start a connection to the appliance.
     
  2. Stop the iptables service in the Console. 
    systemctl stop iptables
  3. SSH from the Managed Host to the Console.
  4. Run the "who" command to verify the connections established.
      
    who
root     pts/13       2021-02-26 08:45 (<IP translated>)
  5. Start the iptables service in the Console.
      
    systemctl start iptables

Resolving The Problem

To resolve this issue, the administrator must reach out to their Network team and ensure the connection being translated is intended or is caused by a misconfiguration in the network device.
If the translation is intended, then QRadar® NAT Groups must be configured to match the NAT configuration. The following technotes explain how this implementation works in QRadar®:

QRadar: Network Address Translation (NAT) in QRadar deployments

QRadar: Understanding NAT Groups and implementation scenarios

As a temporary measure, if the Managed Host uses encryption, the administrator can add the translated IP address to the iptables service in the remote host by following the next steps:

  1. Log in to the remote host blocking the translated IP address.
  2. Edit the /opt/qradar/conf/iptables.pre file and allow that IP address:

    1. Make a backup of the current file:

      mkdir -p /store/IBM_Support
cp -pfv /opt/qradar/conf/iptables.pre /store/IBM_Support

    2. Append the entry changing <ipaddress> with the translated IP address obtained from the "Diagnosing the Problem" section:

      echo "-A INPUT -p tcp -s <ipaddress> --dport 22 -j ACCEPT" >> /opt/qradar/conf/iptables.pre
    3. Reload the iptables service: 
      /opt/qradar/bin/iptables_update.pl
Once the connection is restored by either the required network changes or the QRadar® NAT Groups configuration, the previously appended entry can be removed from the /opt/qradar/conf/iptables.pre file.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"TS004773797","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
25 October 2023

UID

ibm16403123

Page Feedback