IBM Support

QRadar: Error "User limit of inotify instances reached" when using the changePasswd.sh script caused by max_user_instances limit

Troubleshooting


Problem

This article explains how to identify, understand and troubleshoot the error message:
java.io.IOException: User limit of inotify instances reached or too many open files.

Symptom

When an administrator tries to change the password with the changePasswd.sh script, they see the following error:
[root@primary ~]# /opt/qradar/support/changePasswd.sh -a -V
Please enter the new admin password.
Password:
Confirm password:
Altering admin user password ...
-----------------------------------------
Altering admin user password using PasswordCommandLineClient.
Exception thrown while initializing Frameworks.
com.q1labs.frameworks.exceptions.FrameworksException: Exception thrown while initializing Frameworks.
        at com.q1labs.frameworks.core.FrameworksContext.initFrameworks(FrameworksContext.java:272)
  com.ibm.si.security_model.authentication.internal.PasswordCommandLineClient.main(PasswordCommandLineClient.java:230)
Caused by: com.q1labs.frameworks.exceptions.FrameworksInitException: Unable to register custom X.509 certificate manager
        at com.q1labs.frameworks.core.FrameworksContext.postInitialization(FrameworksContext.java:827)
Caused by: com.q1labs.frameworks.crypto.trustmanager.exceptions.Q1TrustStoreException: Error starting watch service at /opt/qradar/conf//trusted_certificates

<...>

Caused by: java.io.IOException: User limit of inotify instances reached or too many open files

<...>

ERROR: Unable to set password

Cause

The files fs.inotify.max_user_instances define the user limits on the number of inotify resources.
If these limits are reached, you may experience processes failing with error messages similar to the previously mentioned:
User limit of inotify instances reached or too many open files.

Diagnosing The Problem

You are able to check the current max instances configuration by checking the file named located on /proc/sys/fs/inotify/, run the following command:
cat /proc/sys/fs/inotify/max_user_instances
Output example:
cat /proc/sys/fs/inotify/max_user_instances
2048
Please take note of the current configuration in case you need to change it back after the troubleshooting.

Resolving The Problem

The max_user_instances needs to be modified, run the following command to set the max user instances to a bigger amount:
sysctl fs.inotify.max_user_instances=8192
After this, the administrator can try to change the password with the changePasswd.sh script without any error.
It is important to mention that this change is not persistent after a reboot, so it is recommended to change back to the original max instances configuration:
  
sysctl fs.inotify.max_user_instances=2048

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSV4BL","label":"IBM QRadar"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
19 June 2024

UID

ibm17156944