Direct links to fixes
APAR status
Closed as program error.
Error description
If a Tivoli Storage Manager server is configured with very large random access (disk) storage pools, a server start might show a delay. Under the APAR condition, the call stack for the admStartServer thread seems to be hung in bfInit(). pstack coomand output collected several times will look similar to: ffffffff7c9dcf48 semsys (2, 29000079, ffffffff7fffbde4, 1, 2e8c8) ffffffff7aec7b9c __1cMsqlccipcrecv6FpnRSQdDLCC_COMHANDLE_T_pnMSQdDLCC_COND_T__h_ (ffffffff76026ce0, ffffffff76026ba0, 19580000, 0, 0, ffffffff76027164) + 6fc ffffffff7aecd6fc sqlccrecv (ffffffff76026ce0, ffffffff76027100, 10004, ffffffff7fffc1ec, 6, 90) + 314 ffffffff7b12e370 __1cMsqljcReceive6FpnKsqljCmnMgr__i_ (ffffffff76026978, 0, 0, f000, ffffffff7fffc1e0, 19b00000) + 130 ffffffff7b198ec4 __1cPsqljrDrdaArOpen6FpnOdb2UCinterface_pnPdb2UCCursorInfo__i_ (ffffffff74600090, ffffffff19800620, 0, 0, 0, ffffffff7c451778) + 28c ffffffff7ab134ec __1cHcsmOpen6FpnOdb2UCinterface_pnPdb2UCCursorInfo__i_ (ffffffff74600090, ffffffff7b198c38, 0, 10070dc, 0, 19f00000) + 9c ffffffff7a7ee074 __1cLCLI_sqlOpen6FpnRCLI_STATEMENTINFO_pnTCLI_ERRORHEADERINFO__h _ (ffffffff19800080, ffffffff19800090, 1, ffffffff19800620, 3, 0) + ad4 ffffffff7a8eaa9c __1cLSQdDLExecute26FpnRCLI_STATEMENTINFO_pnTCLI_ERRORHEADERINFO_ _h_ (ffffffff19800080, ffffffff19800090, 2ce0, 1800, 1000, 2c00) + 18d4 ffffffff7a8db834 SQLExecute (1000000, 2c00, 19530000, 0, 0, 2cd4) + 11d4 0000000101833c14 tbRegExecEx (104ceec48, 0, ffffffff7fffcdf0, ffffffff7fffcddc, ffffffff7fffcde8, ffffffff7fffcdf8) + 17e4 0000000101da618c dsObtainFreeListForVol (104904968, 17096, c34fff, 104ac1ac4, 104ac1ad0, 103a40988) + 804 000000010185f988 ObtainFreeList (104904968, 104ac1a88, 0, 3, 7fffffff, e) + 418 000000010185bd74 bvOpen (104904968, 17096, ffffffff7fffd290, 1, 10370df30, e) + 5d4 0000000101dc6a20 DsGetPoolStatus (4, ffffffff7fffd798, ffffffff7fffd7a0, ffffffff7fffd7b8, ffffffff7fffd7b0, 0) + bd0 0000000101e29998 ssGetNextPool (ffffffff7fffd5e0, ffffffff7fffd5e0, 1, ffffffff7fffd5e0, 0, 1047373a8) + 12b0 000000010090d258 BfInitDedup (104bebd98, 306, 104bebd98, 104be3258, 0, 104be3258) + 488 000000010070d590 bfInit (1, 0, 173a, 4753495a4500, ff0000, 8080808080808080) + 1020 00000001001f558c StartServer (0, 0, ffffffff7fffeeac, ffffffff7fffeea8, 102e1c718, 1046fc8a8) + 2354 00000001001e564c admStartServer (0, 0, 527, 102623cb0, 104698699, 104698298) + 1dc 000000010007c8d0 main (1, ffffffff7ffff888, ffffffff7ffff898, ffffffff7c94c920, ffffffff79300b40, ffffffff79700200) + 27a0 00000001000779dc _start (0, 0, 0, 0, 0, 0) + 17c A trace with traceflags PVR MMS will show the delay as with the following example: 15:51:18.781 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed out waiting for session manager initialization. 16:16:18.841 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed out waiting for session manager initialization. 16:41:18.902 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed out waiting for session manager initialization. 16:51:17.170 [1][pvrclass.c][5644] [pvrIsFileDevClass]:pvrIsFileDevClass: device class <SOMEDEVICECLASSis not File The trace above documnets the initialization of a disk storage pool exceeding 20TB in size. Tivoli Storage Manager Versions Affected: all supported versions Initial Impact: Medium Additional Keywords: bitvector bfo hang stgpool zz62 zz63 zz64 zz71
Local fix
reduce the size of the disk stoarge pool or use file device storage pool When you look for a recommendation on how much to lower the disk storage pool to, as a rule of thumb a disk storage pool should only be large enough to contain one nights worth of backups and then migrate off to sequential media. In addition, larger backups (for example, TDP type backups or system image type backups) should go direct to sequential media skipping the disk storage pool.
Problem summary
**************************************************************** * USERS AFFECTED: * * All Tivoli Storage Manager server users. * **************************************************************** * PROBLEM DESCRIPTION: * * See error description. * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in level 7.1.5. Note that this is * * subject to change at the discretion of IBM. * * * * New FFDC entries are being created to help determine how * * long it takes to build the bitvector for each of the DISK * * storage pool volumes. This processing only happens once on * * server startup or when a new volume is created. * * * * The following are the sample entries created for each * * volume: * * * * [01-29-2016 14:47:11.297][ FFDC_GENERAL_SERVER_ERROR ]: * * (dsalloc.c:574) Building freelist for volid 110 * * [01-29-2016 14:47:11.430][ FFDC_GENERAL_SERVER_ERROR ]: * * (dsalloc.c:869) Built freelist for volid 110 from 10 * * database rows. * * * * A large number of entries on a volume could indicate * * fragmented disk storage, client underallocation into the * * storage pool or client data that is not aggregated such as * * API applications that use single file transactions. For * * this type of API data a file pool or a container pool should * * be used instead of a disk storage pool. * * * * The following can be used to get the volume name that * * matches the volume id from a db2 command line prompt: * * * * db2 connect to tsmdb1 * * C:\Program Files\Tivoli\TSM\server>db2 -x "select volname * * from tsmdb1.ss_volume_names where volid=110" * * * * Sample output: * * C:\PROGRAM FILES\TIVOLI\TSM\SERVER\DISKVOL000 * ****************************************************************
Problem conclusion
This problem was fixed. Affected platforms: AIX, HP-UX, Solaris, Linux, and Windows.
Temporary fix
Comments
APAR Information
APAR number
IT04997
Reported component name
TSM SERVER
Reported component ID
5698ISMSV
Reported release
63W
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2014-10-17
Closed date
2016-02-01
Last modified date
2016-02-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
TSM SERVER
Fixed component ID
5698ISMSV
Applicable component levels
[{"Business Unit":{"code":"BU029","label":"Software"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"63W"}]
Document Information
Modified date:
28 August 2023