Qradar: Fix time synchronization error "Address already in use socat failed to initialize"

Troubleshooting

Problem

In Qradar, Chrony is used to perform time synchronization between the managed and the console. The service uses port 123 and cannot function properly when that port is used by another service.

Symptom

In the Qradar system notifications you can see:

Time Synchronization to Console has failed - socat failed to initialize.

In the Qradar logs (/var/log/qradar.log):

DATE TIME hostname [time_sync]: [ERROR] [NOT:0150003100] Time Synchronization to Console has failed - socat failed to initialize.
DATE TIME hostname [time_sync]: [ERROR] [NOT:0150003100][10.105.112.77] Time Synchronization to Console has failed - chrony error

Cause

This issue often occurs when the ntpd service has been manually started (it is stopped by default). Ntpd should be in inactive state in all managed hosts and the console to avoid the issue described above.

Default status:

[root@managedhost ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Environment

Qradar version: 7.5.0

Diagnosing The Problem

Ssh to the Qradar console
Run the command below to display the status of ntpd in all managed hosts and the console:
```
/opt/qradar/support/all_servers.sh -Ck 'systemctl status ntpd'
```
Note: If the output above is too long, you can alternatively use the command below to only display the managed host where ntpd is in active state:
```
/opt/qradar/support/all_servers.sh -Ck 'systemctl status ntpd' |grep 'Active: active' -B 6 -A 2
```

If there is any managed host with ntpd running it should look similar to the output below:

10.X.X.X -> managedhost.local
Appliance Type: 4000    Product Version: 2021.6.6.20230519190832
 12:09:55 up 45 days, 19:05,  0 users,  load average: 1.98, 1.98, 2.19
------------------------------------------------------------------------
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-10-23 12:09:35 ADT; 20s ago
  Process: 15295 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 15299 (ntpd)
    Tasks: 1
   Memory: 644.0K
   CGroup: /system.slice/ntpd.service
           └─15299 /usr/sbin/ntpd -u ntp:ntp -g
Oct 23 12:09:35  ntpd[15299]: Listen normally on 14 veth7c1b76c fe80:::6612 UDP 123

Note the IP address(es) of the concerned managed host(s) for the next troubleshooting steps

Resolving The Problem

Ssh to the problematic managed host(s)

Confirm that ntpd is running with the command:

systemctl status ntpd

Expected output:

[root@managedhost ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-10-23 12:09:35 ADT; 18min ago
 Main PID: 15299 (ntpd)
    Tasks: 1
   Memory: 644.0K
   CGroup: /system.slice/ntpd.service
           └─15299 /usr/sbin/ntpd -u ntp:ntp -g
Oct 23 12:09:35 managedhostlocal ntpd[15299]: Listen normally on 16 vethd51daf7 fe80::UDP 123

Stop ntpd:
```
systemctl stop ntpd
```
Disable ntpd:
```
systemctl disable ntpd
```

Run the manual time synchronization script to confirm if the issue is resolved:

[root@managedhost ~]# /opt/qradar/bin/time_sync.sh
2023-10-23T15:33:21Z chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
2023-10-23T15:33:25Z System clock wrong by -0.000014 seconds (step)
2023-10-23T15:33:25Z chronyd exiting

Results:
The time synchronization should work correctly if there is no underlying issues and the system notifications Time Synchronization to Console has failed should stop

Related Information

INTERMITTENT QRADAR SYSTEM NOTIFICATIONS 'TIME SYNCRONIZATION HAS FAILED - SOCA…

Troubleshooting chrony errors

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.5.0"}]

Tips