IBM Support

QRadar: Deploy times out due to missing or mismatched tokens

Troubleshooting


Problem

The QRadar Console is responsible for replicating its database and also pushing deployment configuration to all managed hosts in the deployment. Occasionally, one or more hosts might timeout during the Deploy Changes process. The Console and all managed hosts in the deployment must have matching tokens in /opt/qradar/conf/host_tokens.masterlist and /opt/qradar/conf/host.token files to avoid deploying changes communication issues.

Symptom

After an administrator attempts to deploy a change, the Console or the managed hosts timeout and display messages similar to the following in /var/log/qradar.log. Note the IP address of the appliance that timed out from the user interface or the logs. 

Host token issues can be reported with the following log messages:
  • Unable to retrieve authentication token for RPC call
  • Host token invalid. Unable to download database updates
  • Unable to decrypt the host token from: /opt/qradar/conf/host.token
  • Failed Read Host Token File: host.token
Example on 7.5.0+ Console
[ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [INFO] [NOT:0000006000][X.X.X.X/- -] [-/- -]Following message suppressed 39 times in 300000 milliseconds
[ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [ERROR] [NOT:0000003000][X.X.X.X/- -] [-/- -]Unable to retrieve authentication token for RPC call
[ConfigChangeObserver Timer[1]] com.q1labs.frameworks.crypto.DecryptException: com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
[ConfigChangeObserver Timer[1]]    at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:56)
[ConfigChangeObserver Timer[1]]    at com.q1labs.core.shared.jsonrpc.RPC.readAuthenticationToken(RPC.java:291)
[ConfigChangeObserver Timer[1]]    at com.q1labs.core.shared.jsonrpc.RPC.executeMethodWithTimeout(RPC.java:213)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.getActionRequest(ConfigChangeObserver.java:426)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.timeExpired(ConfigChangeObserver.java:401)
[ConfigChangeObserver Timer[1]]    at com.q1labs.hostcontext.configuration.ConfigChangeObserver$ConfigChangeObserverTask.run(ConfigChangeObserver.java:662)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[ConfigChangeObserver Timer[1]]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[ConfigChangeObserver Timer[1]]    at java.lang.Thread.run(Thread.java:822)
[ConfigChangeObserver Timer[1]] Caused by: 
[ConfigChangeObserver Timer[1]] com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:390)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:373)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.Crypto.decrypt(Crypto.java:73)
[ConfigChangeObserver Timer[1]]    at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:53)
[ConfigChangeObserver Timer[1]]    ... 12 more
[ConfigChangeObserver Timer[1]] Caused by: 
[ConfigChangeObserver Timer[1]] javax.crypto.BadPaddingException: Given final block not properly padded
[ConfigChangeObserver Timer[1]]    at com.ibm.crypto.provider.AbstractBufferingCipher.a(Unknown Source)
[ConfigChangeObserver Timer[1]]    at com.ibm.crypto.provider.AbstractBufferingCipher.engineDoFinal(Unknown Source)
[ConfigChangeObserver Timer[1]]    at javax.crypto.Cipher.doFinal(Unknown Source)
[ConfigChangeObserver Timer[1]]    at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:387)
[ConfigChangeObserver Timer[1]]    ... 15 more
Example on 7.5.0+ managed host
managed-host.local systemd[1]: Starting hostcontext daemon...
managed-host.local systemd[1]: Started hostcontext daemon.
managed-host.local python[12887]: detected unhandled Python exception in '/opt/qradar/lib/python/qradar/mks.py'
managed-host.local replication[12496]: Host token invalid. Unable to download database updates.
managed-host.local hostcontext[11107]: com.q1labs.hostcontext.lifecycle.LifeCycleException: Unable to reset running lock
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5349)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.start0(HostContext.java:733)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.access$700(HostContext.java:98)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext$5.run(HostContext.java:915)
managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: unable to release running lock, future actions will not run until this lock is released
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1726)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5337)
managed-host.local hostcontext[11107]: ... 3 more
managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: Unable to determine if backup already running
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:556)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1722)
managed-host.local hostcontext[11107]: ... 4 more
managed-host.local hostcontext[11107]: Caused by: java.lang.Exception: Tomcat is not running. Unable to update a backup running lock (key:BACKUP_RUNNING_105, jsonObject:null)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.core.BackupUtils.setBackupRunningLock(BackupUtils.java:2221)
managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:552)
managed-host.local hostcontext[11107]: ... 5 more
managed-host.local systemd[1]: hostcontext.service: main process exited, code=exited, status=1/FAILURE
managed-host.local systemd[1]: Unit hostcontext.service entered failed state.
managed-host.local systemd[1]: hostcontext.service failed.
Example on 7.4.3 and older on the Console or managed host:
[hostcontext.hostcontext] [ConfigChangeObserver Timer[1]] com.q1labs.configservices.hostcontext.exception.HostContextException: Failed to execute url https://127.0.0.1/console/fetchConfig/globalset_list.xml HTTP/1.1 400 Bad Request

[hostcontext.hostcontext] [main] java.lang.Exception: Unable to decrypt the host token from: /opt/qradar/conf/host.token

[hostcontext.hostcontext] [main] com.q1labs.configservices.common.ConfigServicesException: Failed Read Host Token File: host.token

Cause

A common cause for deployment changes timeouts is missing or mismatched tokens between the Console and the managed host that failed to deploy. The hostcontext service uses the following tokens to deploy changes: 
 
  • Console tokens are found in /opt/qradar/conf/host_tokens.masterlist on the Console appliance.
  • Managed host tokens are found in /opt/qradar/conf/host.token on the remote managed host.

Diagnosing The Problem

Before you begin
This procedure requires an administrator to have permissions to the command line for the Console and managed host that failed to deploy. The administrator must also have the IP address of the managed host that did not complete a deployment changes.
Procedure
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Check the sums of the tokens on the Console by using md5sum and review the output to confirm the MD5 sums match each other. 
    updatedb;md5sum $(locate host_tokens.masterlist)

Example of matching output:

    9c42f5150dd2a3e3923a9bfb35d15a22  /opt/qradar/conf/host_tokens.masterlist
9c42f5150dd2a3e3923a9bfb35d15a22  /store/configservices/host_tokens.masterlist
9c42f5150dd2a3e3923a9bfb35d15a22  /store/configservices/backup/deployed/GLOBALSET/host_tokens.masterlist
9c42f5150dd2a3e3923a9bfb35d15a22  /store/configservices/backup/deployed/LOCALSET/host_tokens.masterlist
9c42f5150dd2a3e3923a9bfb35d15a22  /store/configservices/deployed/GLOBALSET/host_tokens.masterlist
9c42f5150dd2a3e3923a9bfb35d15a22  /store/configservices/deployed/LOCALSET/host_tokens.masterlist
  3. From the Console, record the host token entry for the managed host that failed to deploy by using the managed host's IP. 
    grep x.x.x.x /opt/qradar/conf/host_tokens.masterlist
    Note: On QRadar 7.4.3, a new hash was implemented to reinforce security. The entries in the file are expected to be longer than in previous versions.

    Example token on 7.4.2 and older versions.     
    x.x.x.x=55fe8879-f67a-422f-9dec-d8061e62dab0
    Example token on 7.4.3+ 
    x.x.x.x=AQAAAAAAAAABBcctA0NsEeX96c01QFrwId0F8xSr7LHMwvDLEbzdEjEeB2gpzERLuCjGIMHwyN6sASAR8ZNBtRUJbeR+BY6prw==
  4. Open an SSH session to the managed host experiencing a deployment issue.
  5. Confirm that the host token on the managed host in /opt/qradar/conf/host.token matches the token recorded in /opt/qradar/conf/host_tokens.masterlist  from the Console. 
    cat /opt/qradar/conf/host.token
    Example token on 7.4.2 and older versions. 
    x.x.x.x=55fe8879-f67a-422f-9dec-d8061e62dab0
    Example token on 7.4.3+ 
    x.x.x.x=AQAAAAAAAAABBcctA0NsEeX96c01QFrwId0F8xSr7LHMwvDLEbzdEjEeB2gpzERLuCjGIMHwyN6sASAR8ZNBtRUJbeR+BY6prw==
    Note: There is no newline after the CLI output on a valid token.
  6. Compare the hash between the files with the md5sum command on the managed host. 
    [root@hostname-managed_host ~]# updatedb; md5sum $(locate host.token)
829268b46f16e4c4ca3bcf7d53e05aae  /opt/qradar/conf/host.token
829268b46f16e4c4ca3bcf7d53e05aae  /store/configservices/deployed/LOCALSET/host.token

    Results
    If the tokens on the managed host and Console do not match, review the Resolving The Problem section of this technical note. If the hashes do not match, the restarting and redeploying sections steps in the resolution resolve that issue. If the tokens do match between the Console and managed host, review your network, bandwidth, or review QRadar Deploy Changes 101.

Resolving The Problem

Administrators must manually update the /opt/qradar/conf/host.token on the affected managed host and match it with the entry in /opt/qradar/conf/host_tokens.masterlist.
Procedure
  1. SSH into the Console as the root user.
  2. Gather the token associated with the affected managed host. 
    grep <Managed host IP> /opt/qradar/conf/host_tokens.masterlist
x.x.x.x=arOnjSosaAtTqFgx1111111111111111111111111111111111111V2
  3. SSH into the QRadar managed host with a mismatched token as the root user.
  4. Back up the original token. 
    mkdir -pv /store/ibm_support
cp -fv /opt/qradar/conf/host.token /store/ibm_support/host.token.backup-$(date +%F)
  5. With the echo command, update the /opt/qradar/conf/host.token file and do not insert a carriage return 
    echo -n "arOnjSosaAtTqFgx1111111111111111111111111111111111111V2" > /opt/qradar/conf/host.token
    Example output on 7.4.2 and older: 
    echo -n '55fe8879-f67a-422f-9dec-d8061e62dab0' > /opt/qradar/conf/host.token
    Example output on 7.4.3 and later:
    echo -n 'AQAAAAAAAAABBcctA0NsEeX96c01QFrwId0F8xSr7LHMwvDLEbzdEjEeB2gpzERLuCjGIMHwyN6sASAR8ZNBtRUJbeR+BY6prw==' > /opt/qradar/conf/host.token
  6. Confirm the correct value: 
    cat /opt/qradar/conf/host.token
    The output must display a managed host token that matches the master token from the Console. In our example, it was: arOnjSosaAtTqFgx1111111111111111111111111111111111111V2
  7. Restart the hostcontext service and wait at least 2 minutes.
    Note: The restart of the hostcontext service affects other functions such as correlations, searches, offenses creation, and other functions. See QRadar: Core services and the impact of restarting services. Administrators are advised to run the following steps during scheduled maintenance to avoid undesired affectation. 
    systemctl restart hostcontext
  8. Confirm hostcontext service is running and remains active for at least 5 minutes. 
    systemctl status hostcontext
  9. Log in to the QRadar Console as an administrator.
  10. Click the Admin tab.
  11. Click Deploy Changes.

    Results
    Wait for the deployment to replicate changes to all managed host. The host with the updated token is expected to deploy successfully. If other managed hosts in the deployment fail to deploy successfully, you can repeat this procedure to confirm tokens match. If you continue to experience issues with the managed hosts, you can review QRadar Deploy Changes 101 or contact QRadar Support for further assistance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.0;7.4.1;7.4.2;7.4.3;7.5.0"}]

Document Information

Modified date:
21 July 2022

UID

ibm10961320

Page Feedback