Troubleshooting
Problem
When the /storetmp partition in QRadar does not have enough space, it can affect the regular functioning of QRadar® SIEM. The purpose of this article is to help the administrator with the removal of files and directories when the /storetmp partition has not enough available disk space.
Symptom
Lack of available space in the /storetmp partition can cause the following issues:
- Alerts about "Process monitor application failed to start multiple times".
- Searches reporting I/O errors.
- Services not starting.
- Configuration deployment changes due to critical disk space.
[tomcat.tomcat] /console/JSON-RPC/QRadar.scheduleDeployment QRadar.scheduleDeployment] com.q1labs.configservices.util.ConfigServicesUtil: [INFO] [-/--] Deployment is blocked due to critical disk space issue - Failed disk space checks when a software update runs.
=-= DiskSpace Report for Mountpoint '/storetmp' =-= =-= Available: 996688 Kb, Required: 1571635.2 KB =-= =-= Directories over 1G on mountpoint /storetmp to a depth of 3: /storetmp =-= Size (MB) Directory 14339 /storetmp 14337 /storetmp/test =-= Files on mountpoint /storetmp over 1G =-= 15G /storetmp/test/14GBfile =-= Disk Space Report Complete for '/storetmp' <Hostname> : patch test failed.
Cause
By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the partitions. When a partition goes beyond the critical warning threshold, an alert is triggered for administrators to investigate.
Diagnosing The Problem
If the appliance has a disk allocation that meets the storage requirements and the /storetmp is mounted as a separate partition, the administrators can identify the largest directories and files by following the steps in Troubleshooting disk space usage problems. Once identified, compare them with the following list.
/storetmp/status
/storetmp/reporting
/storetmp/vis
/storetmp/cmt
/storetmp/backup
/storetmp/ecs-ec
/storetmp/ecs-ec-ingress
/storetmp/ecs-ep
/storetmp/tempdownload
/storetmp/offline_forwarder
/storetmp/hostcontext
/storetmp/configset
/storetmp/patches
/storetmp/qvm
/storetmp/qvmprocessor
/storetmp/ariel_proxy_server
The following example shows the /storetmp/backup20220929 using 15GB. This directory is not in the list, therefore, it's likely a directory that can be deleted.
[root@qradar ~]# du -xch -d 1 /storetmp/ | sort -h | tail -n 5
36K /storetmp/status
2.4M /storetmp/cmt
15G /storetmp/backup20220929
Once these large directories are identified, follow the instructions in Resolving the Problem to remove them.
Resolving The Problem
Use the following instructions to identify safe to remove files and regain space.
Depending on the directory reported during diagnosis, follow the suggestions provided. You might follow some or all of the suggestions, depending on your needs.
- Files excluded by diskmaintd.pl.
- Ensure the directory reported is not excluded by disk maintenance. For more information about this procedure, see: QRadar: Files in /storetmp are removed daily by disk maintenance.
- Move or remove the directory.
Note: The following are examples that use /storetmp/backup20220929. Administrators must change this directory according to their needs.
To move:
To remove:mkdir -p /store/IBM_Support/ mv -v /storetmp/backup20220929 /store/IBM_Support/rm -rfv /storetmp/backup20220929
- Certificate revocation list (CRL) files.
- Move or remove the files.
To move:
To remove:mkdir -p /store/IBM_Support/ mv -v /storetmp/crl_download* /store/IBM_Support/rm -fv /storetmp/crl_download*
- Move or remove the files.
- Certain Log sources. Administrators might take the following actions:
Note: Example of log sources that saved data temporarily in /storetmp are AWS REST API in /storetmp/53/ and Bluecoat in /storetmp/bluecoatwss.- Validate whether the amount of data coming in is valid for the period.
- If the data being pulled is valid, the administrator might consider to reconfigure the log source and send the events to a different target event collector.
- Alternatively, the following steps can be performed to alleviate the disk space issues in /storetmp while the required actions from previous steps are arranged:
- Disable the Log Source in the Log Source Management App.
- Take a backup of the current directory filling /storetmp.
Note: In this example, /storetmp/53 is used as the conflicting directory.mkdir -pv /store/IBM_Support/ tar -zcvf /store/IBM_Support/storetmp-53-bck-$(date +%F).tar.gz /storetmp/53/ - Moved the directory and link it to a larger partition with more space. In this article, /store is used.
mv -v /storetmp/53/ /store/ ln -s /store/53/ /storetmp/ - Verify the directory is now linked to /storetmp. Note the "/storetmp/53 -> /store/53/" section that indicates the files written to /storetmp/53 are stored in /store/53 instead.
ls -lad /storetmp/53 lrwxrwxrwx 1 root root 10 Sep 30 18:52 /storetmp/53 -> /store/53/ - Enable the Log Source in the Log Source Management App.
- Verify the /storetmp partition is back to normal values and the events of the conflicting Log Source are received.
- Validate whether the amount of data coming in is valid for the period.
Result
The /storetmp partition no longer has disk space constraints. If the partition reached the point of critical services stop, restart the services in the proper order and wait 5 mins with the following commands:
IMPORTANT: When the QRadar core service restart, the QRadar UI, event processing, and database are not available to all users. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
systemctl stop hostcontext
systemctl stop tomcat
systemctl restart hostservices
systemctl start tomcat
systemctl start hostcontext
If the partition does not decrease its usage or the services do not start properly, contact QRadar Support for assistance.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
30 September 2022
UID
ibm16825477