IBM Support

"Too Many Open files" found in Resilient Circuits

Troubleshooting


Problem

"Too Many Open Files" errors can be found in the app.log of Resilient Circuits.

Symptom

Resilient Circuits stops working so that functions and applications that leverage the framework fail.
You might see a similar error in the app.log of Resilient Circuits
2019-12-13 15:05:22,838 ERROR [incident_snow_actions] Traceback (most recent call last): File "/home/integration/Resilient/components/incident_snow_actions.py", line 231, in create_ticket File "/usr/lib/python2.7/site-packages/rc_servicenow/lib/snow_template_functions.py", line 66, in load_template_by_name IOError: [Errno 24] Too many open files: u'/home/integration/Resilient/snowtemplates/snow_create_srq.jinja'

Cause

When the "Too Many Open Files" error message is written to the logs, it indicates that all available file handles for the process were used (includes sockets as well).
By default, the ova has 1024 open files set. A busy system, with a high work load or many incoming requests might not close open sockets quick enough. It is easy to reach the limit of the default file descriptor setting.

Resolving The Problem

Is it possible to display the current ulimit settings?
The command ulimit -Xa displays all current settings that are set for the current login session, where X represents the hard (H) or soft (S) limits to be displayed. By default, if no value is supplied for X, soft limits are displayed.

However, if a process is already running, it might have a different ulimit configuration than the current shell that you run the ulimit -a command in.

Is it possible to find out the ulimit settings for Resilient Circuits?

Find out the PID xxx of the running Resilient Circuits
ps aux | grep resilient-circuit
Run following commands to check limits:
cat /proc/XXX/limits
How are ulimts set?
Soft versus Hard limits

Soft limits are the current setting for a particular limit. They can be increased only to the current hard limit setting.

Hard limits are the maximum limit that can be configured. Any changes to these require root access.

Temporary Settings (for current session)

ulimit -[H|S]limit_name limit_value

Temporary settings are set via the command line that uses the ulimit command. The limit_name must be present in the command (see the chart). These settings are only temporary for the current session and might be lost once the session is over or the terminal window is closed.

This syntax can be used within shell scripts. Any processes created from these shell scripts have the temporary settings for the lifetime they are running.

By default, the soft limit is changed (as changing the hard limit needs root access). Use -H (for hard limits) or -S (for soft limits) to change specific hard or soft limits.

Global Settings

Any changes to these global configuration limits files must be performed by your system administrator. Instructions on how to edit the files can be found on the man page or in the commented section of the configuration file, as they can vary between versions of UNIX or Linux.

The format of the file, which differs slightly on versions Linux, allows individual limits per users or groups, as well as set hard and soft limits. The limits defined in this configuration file are set globally.

Changes made to any limits file take effect when launching a new login shell (such as bash -l), or if the user logs out and logs back in to the system.

 

Specifics for each Operating System:

The most up-to-date information on the limits can be found in the man page for "ulimit".
Linux

Global settings are set in the file /etc/security/limits.conf.

The format for setting each limit is as follows:
<domain> <type> <item> <value>

<domain> controls which users or groups have these limits

<type> is either the string "soft" or "hard" limits.
The hyphen "-" can also be used which represents both soft and hard limits

<item> and <value> are shown in the table. Most values can also use the string unlimited.

This table might not represent all ulimit items as they can vary between Linux variants. See the ulimit man page or the limits.conf file for further information.

 
Command Line limit_name
/etc/security/limits.conf <item>
Description
ulimit -c [value] core Max core dump file size (KB)
ulimit -d [value] data Max data size (KB)
ulimit -f [value] fsize Max file size (KB)
ulimit -l [value] memlock Max size of address space that can be locked into memory (KB)
ulimit -n [value] nofile Max number of open files/sockets per process**
ulimit -u [value] nproc Max number of processes for a single user
ulimit -m [value] rss Max resident set size (KB)
ulimit -s [value] stack The maximum stack size (KB)
ulimit -t [value] cpu Max CPU time (minutes or seconds)
** There is a restriction that occurs on Red Hat Enterprise Linux that this value cannot be set to unlimited
Starting Resilient Circuits manually
If Resilient Circuits is started manually (resilient-circuits run) as opposed to using systemd, then values in /etc/security/limits.conf can be used, for example, 
  * soft nofile 2048  * hard nofile 4096
Starting Resilient Circuits by using systemd
If you start Resilient Circuits using systemd (sudo systemctl start resilient_circuits), then systemd does not follow the same rules set in /etc/security/limits.conf.
In /etc/systemd/system/resilient_circuits.service set LimitNOFILE=4096 to increase the hard and soft limits to 4096.
  [Service]  LimitNOFILE=4096
Restart Resilient Circuits
sudo systemctl daemon-reload
sudo systemctl restart resilient_circuits
Check the change was successful by following the steps from "Is it possible to find out the ulimit settings for Resilient Circuits?"

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"Component":"Resilient Circuits","Platform":[{"code":"PF006","label":"CentOS"},{"code":"PF043","label":"Red Hat"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
19 April 2021

UID

ibm12801397

Page Feedback