IBM Support

QRadar: Unable to add a managed host to deployment due to error “Failed to add host. Installation problem on the host.”

Troubleshooting


Problem

The managed host cannot be added to the deployment after the add host process fails in step 10 with the error:
Step10-Edit2
Error1
On the Console, the following error appears in /var/log/qradar.log:
  [hostcontext.hostcontext] com.q1labs.configservices.capabilities.AddHost: [ERROR]  [-/- -]Failed to add host. Output: 'Done Presence Script', data:'Modifying nva.conf  [hostcontext.hostcontext] com.q1labs.configservices.capabilities.AddHost: [ERROR][-/- -]Failed to read output from ssh connection on host <managedhost_ip>  [hostcontext.hostcontext] com.q1labs.configservices.common.ConfigServicesException: Failed to read output from ssh connection on host <managedhost_ip>  [hostcontext.hostcontext] com.q1labs.configservices.common.ConfigServicesException: Failed to add host. Installation problem on the host.

Cause

This issue is caused by a leftover lock file in /tmp that causes hostservices to fail to start postgresql-qrd service.

Environment

QRadar® 7.3.x and later.

Diagnosing The Problem

After the add host process fails, the following errors can be seen on the affected managed host:
On /var/log/qradar.error:
init_script[16177]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 9 more times.
init_script[16206]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 8 more times.
init_script[16235]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 7 more times.
init_script[16270]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 6 more times.
init_script[16532]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 5 more times.
init_script[16555]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 4 more times.
init_script[16596]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 3 more times.
init_script[16625]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 2 more times.
init_script[16758]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Postgres failed to start. Will try 1 more times.
init_script[16798]: [/opt/qradar/systemd/bin/hostservices.sh] [WARN] Errors were encountered starting the database.  Rebuilding.
init_script[18984]: [/opt/qradar/systemd/bin/hostservices.sh] [ERROR] Could not rebuild the database.  Please contact customer support.
On /var/log/qradar-sql.log:
postgres[28905]: [2-1] LOG:  database system is shut down
postgres[29260]: [1-1] FATAL:  lock file "/tmp/.s.PGSQL.5432.lock" is empty
postgres[29260]: [1-2] HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
The postgresql-qrd service fails to start:
# systemctl status postgresql-qrd
● postgresql-qrd.service - PostgreSQL 9.6 database server
   Loaded: loaded (/etc/systemd/system/postgresql-qrd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-09-04 20:24:00 EDT; 2min 38s ago
     Docs: https://www.postgresql.org/docs/9.6/static/
  Process: 29260 ExecStart=/usr/pgsql-9.6/bin/postmaster -D ${PGDATA} (code=exited, status=1/FAILURE)
  Process: 29253 ExecStartPre=/usr/pgsql-9.6/bin/postgresql96-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 29260 (code=exited, status=1/FAILURE)

Sep 04 20:24:00 hostname systemd[1]: Starting PostgreSQL 9.6 database server...
Sep 04 20:24:00 hostname postmaster[29260]: FATAL:  lock file "/tmp/.s.PGSQL.5432.lock" is empty
Sep 04 20:24:00 hostname systemd[1]: postgresql-qrd.service: main process exited, code=exited, status=1/FAILURE
Sep 04 20:24:00 hostname systemd[1]: Failed to start PostgreSQL 9.6 database server.
Sep 04 20:24:00 hostname systemd[1]: Unit postgresql-qrd.service entered failed state.
Sep 04 20:24:00 hostname systemd[1]: postgresql-qrd.service failed.

Resolving The Problem

  1. Log in to the affected managed host.
  2. Stop hostservices service.
      
    systemctl stop hostservices
  3. Remove the temporary lock file from /tmp.
    Note: The lock file number can differ on different managed hosts. Move or remove the one reported by the logs file or command output as outlined in the "Diagnosing the Problem" section.

    Move the file with: 
    mv /tmp/.s.PGSQL.5432.lock /root/s.PGSQL.5432.lock.bck
    Remove the file with: 
    rm -fv /tmp/.s.PGSQL.5432.lock
  4. Start hostservices service.
      
    systemctl start hostservices
  5. Retry to add the managed host.
​

Document Location

Worldwide

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"TS003975932","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
29 September 2020

UID

ibm16326949

Page Feedback