Troubleshooting
Problem
The managed host cannot be added to the deployment after the add host process fails in step 10 with the error:
![Step10-Edit2](/support/pages/system/files/inline-images/Step10-Edit2.png)
![Error1](/support/pages/system/files/inline-images/Picture2_5.png)
On the Console, the following error appears in /var/log/qradar.log:
[hostcontext.hostcontext] com.q1labs.configservices.capabilities.AddHost: [ERROR] [-/- -]Failed to add host. Output: 'Done Presence Script', data:'Modifying nva.conf [hostcontext.hostcontext] com.q1labs.configservices.capabilities.AddHost: [ERROR][-/- -]Failed to read output from ssh connection on host <managedhost_ip> [hostcontext.hostcontext] com.q1labs.configservices.common.ConfigServicesException: Failed to read output from ssh connection on host <managedhost_ip> [hostcontext.hostcontext] com.q1labs.configservices.common.ConfigServicesException: Failed to add host. Installation problem on the host.
Cause
This issue is caused by a leftover lock file in /tmp that causes hostservices to fail to start postgresql-qrd service.
Environment
QRadar® 7.3.x and later.
Diagnosing The Problem
After the add host process fails, the following errors can be seen on the affected managed host:
On /var/log/qradar.error:
init_script[16177]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 9 more times.
init_script[16206]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 8 more times.
init_script[16235]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 7 more times.
init_script[16270]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 6 more times.
init_script[16532]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 5 more times.
init_script[16555]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 4 more times.
init_script[16596]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 3 more times.
init_script[16625]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 2 more times.
init_script[16758]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 1 more times.
init_script[16798]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Errors were encountered starting the database. Rebuilding.
init_script[18984]: [/opt/qradar/systemd/bin/hostservices.sh] [ERROR] Could not rebuild the database. Please contact customer support.
On /var/log/qradar-sql.log:
postgres[28905]: [2-1] LOG: database system is shut down
postgres[29260]: [1-1] FATAL: lock file "/tmp/.s.PGSQL.5432.lock" is empty
postgres[29260]: [1-2] HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
The postgresql-qrd service fails to start:
# systemctl status postgresql-qrd
● postgresql-qrd.service - PostgreSQL 9.6 database server
Loaded: loaded (/etc/systemd/system/postgresql-qrd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-09-04 20:24:00 EDT; 2min 38s ago
Docs: https://www.postgresql.org/docs/9.6/static/
Process: 29260 ExecStart=/usr/pgsql-9.6/bin/postmaster -D ${PGDATA} (code=exited, status=1/FAILURE)
Process: 29253 ExecStartPre=/usr/pgsql-9.6/bin/postgresql96-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
Main PID: 29260 (code=exited, status=1/FAILURE)
Sep 04 20:24:00 hostname systemd[1]: Starting PostgreSQL 9.6 database server...
Sep 04 20:24:00 hostname postmaster[29260]: FATAL: lock file "/tmp/.s.PGSQL.5432.lock" is empty
Sep 04 20:24:00 hostname systemd[1]: postgresql-qrd.service: main process exited, code=exited, status=1/FAILURE
Sep 04 20:24:00 hostname systemd[1]: Failed to start PostgreSQL 9.6 database server.
Sep 04 20:24:00 hostname systemd[1]: Unit postgresql-qrd.service entered failed state.
Sep 04 20:24:00 hostname systemd[1]: postgresql-qrd.service failed.
Resolving The Problem
- Log in to the affected managed host.
- Stop hostservices service.
systemctl stop hostservices
- Remove the temporary lock file from /tmp.
Note: The lock file number can differ on different managed hosts. Move or remove the one reported by the logs file or command output as outlined in the "Diagnosing the Problem" section.
Move the file with:mv /tmp/.s.PGSQL.5432.lock /root/s.PGSQL.5432.lock.bck
rm -fv /tmp/.s.PGSQL.5432.lock
- Start hostservices service.
systemctl start hostservices
- Retry to add the managed host.
Document Location
Worldwide
[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"TS003975932","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
29 September 2020
UID
ibm16326949