IT46601: DELETION OF EXPIRED CHUNKS DELAYS DUE TO DAMAGED CHUNKS IN CONTAINER POOL.

8.1.24.000-IBM-SPSRV-WindowsX64-262
8.1.24.000-IBM-SPSRV-Linuxx86_64-262
8.1.24.000-IBM-SPSRV-Linuxs390x-262
8.1.24.000-IBM-SPSRV-Linuxppc64le-262
8.1.24.000-IBM-SPSRV-AIX-262

APAR status

• Closed as program error.

Error description

• The problem is if the server has huge data in cloud pool and
  directory container pool, and there is some damaged chunks in
  any container pool. Those expired chunks in directory container
  pool may be delayed to delete.
  
  show CHUNKDELETIONSTATUS on directory container pool will output
  large number of chunks eligible for deletion, for example:
  
  Protect: SERVER1>show CHUNKDELETIONSTATUS CONTAINER_POOL
  
  Number of chunks not referenced: 26307464
  Number of chunks eligible for deletion: 26245926
  
  Current reuse delay setting: 0 minutes
  
  In show thread output, we can see SdRunChunkDeletion thread was
  always busy running in SdRunChunkDeletionOnPoolType:
  
  Thread 2575, Parent 64: SdRunChunkDeletion, Storage 5712,
  AllocCnt 56881868 HighWaterAmt 1771440
  tid=8169, ptid=6491, det=1, zomb=0, join=0, result=0, sess=0,
  procToken=0, sessToken=0
  Stack trace:
  0x090000000021d4a0 semop
  0x090000000bf990d0 sqloSSemV
  0x090000000cbaf978
  sqlccipcsend__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
  0x090000000bf935e8 sqlccsend
  0x090000000c573930 sqljcSend__FP10sqljCmnMgrb
  0x090000000c5c67f8
  sqljrDrdaArOpen__FP14db2UCinterfaceP15db2UCCursorInfo
  0x090000000c80a7bc csmOpen__FP14db2UCinterfaceP15db2UCCursorInfo
  
  0x090000000c722cec
  CLI_sqlOpen__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
  0x090000000c7a94b0
  SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
  0x090000000c7a3628 SQLExecute
  0x00000001001654bc tbRegExecEx2
  0x000000010075c114 SdRunChunkDeletionOnPoolType
  0x000000010075a998 SdRunChunkDeletion
  0x00000001007d755c CallbackWorker
  0x0000000100011a70 StartThread
  
  If enable the SD trace for an hour, search the
  SdRunChunkDeletionOnPoolType, then we can see that it was busy
  on processing cloud pool for delete damaged chunk check, there
  are huge records of adding all cloud pool containers into the
  damaged deletion mode list:
  
  15:09:43.118 [41426][sddelete.c][4276][SdRunChunkDeletionOnPool
  Type]:Processingpool ID=72.
  15:09:43.118
  [41426][sddelete.c][4344][SdRunChunkDeletionOnPoolType]:For
  SdChunkTypeDedup, ItemsQueued: 27028, ItemsDone: 27028
  15:09:43.118
  [41426][sddelete.c][4366][SdRunChunkDeletionOnPoolType]:Previous
  batch of 27028 items in pool 72 is complete
  15:09:43.118 [41426][sddelete.c][4397][SdRunChunkDeletionOnPool
  Type]:Examiningpool 72 for chunks of type SdChunkTypeDedup
  15:09:44.695 [41426][sddelete.c][4532][SdRunChunkDeletionOnPool
  Type]:Container(846509) will be added since this damaged deletio
  15:09:44.696 [41426][sddelete.c][4569][SdRunChunkDeletionOnPool
  Type]:ContainerId(846509)has deletion-eligible chunks
  15:09:44.701
  [41426][sddelete.c][4607][SdRunChunkDeletionOnPoolType]:Chunk
  deletion request 1 queued for cntrId 846509, rc 0
  15:09:44.702 [41426][sddelete.c][4532][SdRunChunkDeletionOnPool
  Type]:Container(846517) will be added since this damaged deletio
  15:09:44.702 [41426][sddelete.c][4569][SdRunChunkDeletionOnPool
  Type]:ContainerId(846517)has deletion-eligible chunks
  15:09:44.705
  [41426][sddelete.c][4607][SdRunChunkDeletionOnPoolType]:Chunk
  deletion request 2 queued for cntrId 846517, rc 0
  15:09:44.706 [41426][sddelete.c][4532][SdRunChunkDeletionOnPool
  Type]:Container(846519) will be added since this damaged deletio
  15:09:44.706 [41426][sddelete.c][4569][SdRunChunkDeletionOnPool
  Type]:ContainerId(846519)has deletion-eligible chunks
  15:09:44.708
  [41426][sddelete.c][4607][SdRunChunkDeletionOnPoolType]:Chunk
  deletion request 3 queued for cntrId 846519, rc 0
  15:09:44.708 [41426][sddelete.c][4532][SdRunChunkDeletionOnPool
  Type]:Container(846524) will be added since this damaged deletio
  15:09:44.708 [41426][sddelete.c][4569][SdRunChunkDeletionOnPool
  Type]:ContainerId(846524)has deletion-eligible chunks
  15:09:44.737
  [41426][sddelete.c][4607][SdRunChunkDeletionOnPoolType]:Chunk
  deletion request 4 queued for cntrId 846524, rc 0
  15:09:44.738 [41426][sddelete.c][4532][SdRunChunkDeletionOnPool
  Type]:Container(846525) will be added since this damaged deletio
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Storage Protect server users.                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available.                           *
  * This problem is currently projected to be fixed in level     *
  * 8.1.24.                                                      *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release: AIX, Linux, and
  Windows.
  Platforms fixed: AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels