IBM Support

QRadar: 31 December License and event processing issue report (APAR IJ30161)

Question & Answer


Question

This technical note is intended to provide more context and information about the 31 December 2020 license issue (APAR IJ30161) and address frequently asked questions for administrators. 

Cause

The IBM QRadar® team became aware of an issue impacting the event correlation system on 31 December 2020. An analysis of the issue from incoming support cases determined a 3rd party license file check timeout was occurring during a service restart. When the 3rd party license file timed out, event correlation and processing services did not start as expected, generating a "Waiting on license..." message in the logs. The license issue affected services for customers who restarted their appliances or restarted services through general administration, such as a deploy changes.

Answer

Timeline
  • 30 December 2020 - Support cases reported from GMT+13 time zones where appliances were not processing events after a deploy changes. Support and development investigated the logs and identified the "Waiting for license..." service messages in the logs.
  • 31 December 2020 - Investigation continued by development and temporary jar files were supplied to users with open cases.
  • 31 December 2020 - A flash notice was issued to QRadar users to not deploy changes in order to prevent unnecessary service restarts.
  • 31 December 2020 - An second flash notice was issued to users with instructions for a one-line Console command to more easily resolve the issue. 
  • 1 January 2021 - Customer communications were updated for a new one-line command for older QRadar versions, such as 7.2.8, 7.3.0, and 7.3.1.
  • 1 January 2021 - Customer communications were updated to include instructions for Disconnected Log Collectors (DLC).
  • 1 January 2021 - An Auto Update flash notice was issued to alert users that a fix can be installed using QRadar Auto Update. 
  • 4 January 2021 - Updated communications to alert users that upgrading their QRadar software required the license fix to be applied post-upgrade.
  • 6 January 2021 - APAR IJ30161 issued to officially track the service issue. Development continues to work and test pending software releases (fix packs) for QRadar 7.3.3 and 7.4.2.
  • 12 January 2021 - QRadar® software released to address this issue for administrators long-term. QRadar 7.3.3 Fix Pack 7 and 7.4.2 Fix Pack 1, are available on IBM® Fix Central.
  • 20 January 2021 - Added a section about new appliance purchases to the FAQ.
Frequently asked questions
 

What is the long-term resolution from QRadar development for IJ30161?

Administrators can install fix packs for QRadar 7.3.3 and QRadar 7.4.2 versions to resolve the 3rd party license issue when available. Development has 3rd party license changes in upcoming fix pack releases to ensure that users do not experience the event processing issue in the future. Quality assurance (QA) teams are validating the fix packs now. Release schedule, pending QA approval:

How do I verify my QRadar appliances have the fix from the auto update?

Starting on 2 January 2021, a special auto update was released to include the one-line fix for the license issue. Administrators can verify their Console received the license fix for this issue by reviewing their auto update. A minor update with the name of patch_2021_1.sh was issues to apply the one-line fix to all Consoles at 7.3.0 or later. Administrators can confirm the patch_2021_1.sh file was applied from the Admin tab > Auto Update user interface. 
image 7730
Note: The auto update system attempts to apply the change daily for all users. However, subsequent auto updates do not display the attempt to install the patch_2021_1.sh file as it was previously applied to the Console. 

For users on QRadar 7.2.8 or air-gapped networks, you can run the one-line command as documented in the Auto Update flash notice or apply the latest Auto Update file from IBM Fix Central, which also includes the patch_2021_1.sh file in QRADAR-QRAUTO-1609491687 or later.

I'm a QRadar on Cloud administrator, what do I need to know?

The one-line command was applied to QRadar on Cloud appliances when available from QRadar Development. There is no action required by QRadar on Cloud administrators. When an upgrade for your QRadar on Cloud appliance is available, your DevOps team will contact your site lead to inform them of the pending upgrade to resolve APAR IJ30161.

What about my high availability pair and fail overs?

The one-line command issued by your QRadar administrators or through QRadar auto update were applied to high availability primary and secondary appliances. If an appliance fails over, you are not required to apply the one-line fix again.

I purchased new appliances, what do I need to do?

Appliances purchased in the 4th quarter of 2020 are impacted by APAR IJ30161 unless they are installed with QRadar 7.3.3 Fix Pack 7 or QRadar 7.4.2 Fix Pack 1. Administrators can complete the following steps to ensure the workaround for IJ30161 is applied to your new appliance.

Procedure

  1. Rack the appliance and complete any installation steps.
  2. Add the appliance to the QRadar deployment.
  3. Select one of the following steps to resolve IJ30161:
    1. Run the one-line command from the flash notice for IJ30161 on your QRadar Console (Recommended).
    2. Run a QRadar auto update check. A check for updates runs support utilities and can apply the fix for IJ30161 to your new appliance.
  4. Log in to the QRadar Console as a user.
  5. Click the Log Activity tab.
  6. In the Quick Filter bar, type the address or hostname of the new appliance.
  7. Verify events are received.

    Results
    If you have multiple appliances to rack or add to your deployment, you can repeat this procedure at any time and run the one-line command to ensure that the new appliance has the workaround for IJ30161. Optionally, administrators can upgrade their QRadar deployment to a version where IJ30161 is resolved. If you continue to experience issues with APAR IJ30161 Waiting for license errors, contact QRadar Support for assistance. 

 

What if I restore a configuration backup to my Console?

The license from APAR IJ30161 does not impact the configuration backups or restores. There is no need to apply the license fix if you restore a configuration backup as the license files exist outside of the files included in a configuration backup or restore.

Can I upgrade my QRadar Console?

Yes, you upgrade your QRadar® version before the 7.3.3 Fix Pack 7 or 7.4.2 Fix Pack 1 updates are available on IBM® Fix Central. However, if you upgrade to a software version before 7.3.3 Fix Pack 7 or 7.4.2 Fix Pack 1 is released to IBM Fix Central, you must apply the one-line fix to your QRadar Console. All appliances must be upgraded and online for the one-line command to apply the fix to QRadar appliances using the all_servers command from the flash notice.

After 7.3.3 Fix Pack 7 or 7.4.2 Fix Pack 1 releases are available, administrators can upgrade to these versions without having to apply the one-line fix for the 3rd party license issue. 

Examples

Current version Upgrading to IJ30161 workaround required?
7.3.x 7.3.3 Fix Pack 6 Yes, any upgrades prior to the release of QRadar 7.3.3 Fix Pack 7 must have the one-line command applied from the flash notice.
7.4.x 7.4.2 GA Yes, any upgrades prior to the release of QRadar 7.4.2 Fix Pack 1 must have the one-line command applied from the flash notice.

What about Disconnected Log Collectors (DLC)?

Disconnected Log Collectors (DLCs) were initially thought to be affected by the 3rd party license issue. Upon further investigation, it was determined that Disconnected Log Collector installations do not require the one-line fix as they do not reference the 3rd party license file when services restart. The procedure documented to apply the one-line fix for DLC installs is removed from the flash notice.

Does this license change impact my EPS/FPM license?

No, the issue identified as IJ30161 was caused by a 3rd party license issue. The 3rd party license is separate from your QRadar Event Per Second (EPS) or Flows per Minute (FPM) licenses. If you received updated license files from IBM or q1pd@us.ibm.com, you can upload and allocate your licenses as normal.

I use QRadar Community Edition, what do I do?

QRadar Community Edition users were affected by this issue. If you are a QRadar Community Edition administrator, you must complete the one-line command as documented in the flash notice.

Are QRadar Trial appliances affected?

Yes, QRadar SIEM Trial installations were impacted by this issue. Users on a 14-day free trial reported issues that event processing stopped for replayed events. An update is being prepared for QRadar Trial appliances to resolve this issue. If you experience issues with your QRadar Trial, email us at qroctry@us.ibm.com.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Document Information

Modified date:
29 January 2021

UID

ibm16398674

Page Feedback