IBM Support

Server hang with "unable to create a pipe" error message

Troubleshooting


Problem

A Tivoli Storage Manager server may become unresponsive and stop processing tape mounts to an external library. When this situation occurs, the server will log messages about being unable to create a pipe.

Cause

The maximum number of file descriptors in use has been reached.

Diagnosing The Problem

Examine the Tivoli Storage Manager server activity log for error messages such as the one listed below.

ANR9999D_0307426690 ExtCreatePipe(psext.c:1023) Thread<n>: unable to create a pipe
ANR9999D Thread<n> issued message 9999 from:
ANR9999D Thread<n> 0x00000001017CA630 OutDiagToCons
ANR9999D Thread<n> 0x00000001017E0124 outDiagfExt
ANR9999D Thread<n> 0x0000000101AC72C0 ExtCreatePipe
ANR9999D Thread<n> 0x0000000101AC5730 psExtLibProcessInitialize
ANR9999D Thread<n> 0x0000000101A921B0 ExtLibProcessInitialize
ANR9999D Thread<n> 0x0000000101A926C8 ExtMount
ANR9999D Thread<n> 0x0000000101A8E540 ExtMountVolume
ANR9999D Thread<n> 0x0000000100F44808 MmsMountVolume
ANR9999D Thread<n> 0x00000001019D6A7C PvrGtsOpen
ANR9999D Thread<n> 0x0000000101A6535C ECartOpen
ANR9999D Thread<n> 0x0000000100EAC108 AgentThread
ANR9999D Thread<n> 0x00000001019442C0 StartThread
ANR9999D Thread<n> 0xFFFFFFFF7DECD880 *UNKNOWN*
ANR1227E BACKUP STGPOOL: Process X terminated - internal server error detected.
ANR9999D Thread<n> issued message 1227 from:
ANR9999D Thread<n> 0x00000001017CB0B0 outRptf
ANR9999D Thread<n> 0x0000000100572B24 EndBackupProc
ANR9999D Thread<n> 0x000000010056AFC8 DfBackupPoolThread
ANR9999D Thread<n> 0x00000001019442C0 StartThread
ANR9999D Thread<n> 0xFFFFFFFF7DECD880 *UNKNOWN*

Resolving The Problem

The error indicates that the operating system maximum number of file descriptors has been reached. This may happen if there is a lot of concurrent work and the maximum number of file descriptors value is too low. The value needs to be increased.

Increasing the maximum number of file descriptors on AIX
On AIX, the "ulimit -a" command may be used to find out more about the file descriptors. This command will report the system limits for memory, stack,....etc, as well as number of file descriptors. The default maximum number of file descriptors is 2000. For example :
$ ulimit -a
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
To temporarily change the maximum number of file descriptors to "unlimited", the following AIX ulimit command may be used : ulimit -n unlimited
Alternatively, to set this permanently, the default section of the /etc/security/limits file may be updated to set nofiles to -1 (unlimited). For example :
default:
fsize = -1
core = -1
cpu = -1
data = -1
rss = -1
stack = -1
nofiles = -1
After updating the /etc/security/limits file, it is necessary to re-login for the new values to take effect.

Increasing the maximum number of file descriptors on HP-UX
On HP-UX start SAM and select:
-1- Kernel Configuration
-2- Configurable Parameters
HP has an on-line manual with more information titled "Tunable Kernel Parameters" located here :
http://docs.hp.com/hpux/onlinedocs/TKP-90202/TKP-90202.html
Review the maxfiles, maxfiles_lim and nfile parameters

Increasing the maximum number of file descriptors on Linux
On Linux review the file "/proc/sys/fs/file-max". The "maximum number of open files" value cannot be larger than the value specified in file-max. The current "maximum number of open files" value may be obtained by examining the value for the "open files" field in the ouput the "ulimit -a" command.
The "maximum number of open files" value may be changed by editing the /etc/security/limits.conf file. For example, to increase the value to 2048, add the following line :
root - nofile 2048
The value may also be changed dynamically from a unix shell with the following command
ulimit -n 2048
The Tivoli Storage Manager server must then be started from that same shell to get the new value for open files.

Increasing the maximum number of file descriptors on Sun Solaris
On Sun Solaris, issue the ulimit command to view/set the value for maximum number of file descriptors: ( nofiles or open files ). For example :
ulimit -a ....to view the nofiles or open files value ( Solaris default value is 256 )
ulimit -n 2048 ....to set the nofiles value to 2048
ulimit -n unlimited ....to set the nofiles value to unlimited
This will change the value for the current session. To make the system wide change, edit the edit /etc/system file. For example :

rlim_fd_max= 65536 // hard limit
rlim_fd_cur= 1024 // soft limit

In summary, the solution is to :

1) increase the OS nofiles hardlimit (to unlimited for example)
--or--
2) decrease the amount of concurrent work to prevent the nofiles maximum value from being reached.

Once this is corrected, the Tivoli Storage Manager server needs to be stopped and restarted from the same shell with the new nofiles value.

[{"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
23 June 2018

UID

swg21404917