IBM Support

QRadar: How to pull AWS CloudTrail logs from a user specified point.

Troubleshooting


Problem

Creating a new Amazon AWS CloudTrail log source to monitor a trail with a large amount of historical log data can result in performance and disk space issues.

Cause

Creating and deploying a new Amazon AWS CloudTral log source results in the log source attempting to retrieve all the compressed log data starting from the oldest timestamp to the most recent. This causes the log data to be retrieved and temporarily stored in the /store directory prior to QRadar attempting to extract and process this data. This can result in performance and disk space issues if a large number of historical log files are found.

Resolving The Problem

To resolve the problem, the persisted session properties for the AWS CloudTrail log source can be manually updated to set the starting point for log data retrieval.
Note: Only log files with the default CloudTrail log file name format can be collected. The filename format is <AccountID>_CloudTrail_<RegionName>_<YYYYMMDDTHHmm>Z_UniqueString.<FileNameFormat>. For example, 111122223333_CloudTrail_us-east-2_20150801T0210Z_Mu0KsOhtH1ar15ZZ.json.gz.

Procedure to set a Starting Marker.

  1. Disable the AWS CloudTrail Log source.
  2. Using SSH, log in to the QRadar Console.
  3. If the log source is configured for an Event Processor other than the Console, SSH from the Console to the appropriate Event Processor.
  4. Change Directories to /store/ec/amazonaws
    cd /store/ec/amazonaws
  5. Using vi editor, open the AWS CloudTrail session properties file for the log source of interest.
  6. Modify the marker property with the name of the log file you would like to have the log source start retrieving data from.
    #Amazon AWS REST API compare list
    #Wed Aug 02 12:02:49 ADT 2017
    marker=AWSLogs/379708147527/CloudTrail/us-east-1/2017/08/02/379708147527_CloudTrail_us-east-1_20170802T0015Z_IJjTQMuj4iA5COc1.json.gz
    lastPoll=1501686169203
  7. Type esc :wq to save the changes.
  8. Enable the AWS CloudTrail log source.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
28 May 2024

UID

swg22006878

Page Feedback