IT46213: REPLICATION OF OPEN SNAP STORE MANAGER (OSSM) DATA FAILS WITH ANR9999D_3640283921

8.1.24.000-IBM-SPSRV-WindowsX64-262
8.1.24.000-IBM-SPSRV-Linuxx86_64-262
8.1.24.000-IBM-SPSRV-Linuxs390x-262
8.1.24.000-IBM-SPSRV-Linuxppc64le-262
8.1.24.000-IBM-SPSRV-AIX-262

APAR status

• Closed as program error.

Error description

• IBM Storage Protect Plus can backup data to an IBM Storage
  Protect server using the Open Snap Store Manager (OSSM).
  When IBM Storage Protect Plus is configured to replicate this
  data between IBM Storage Protect servers, storage rules are
  configured between the IBM Storage Protect servers.
  
  These storage rules can fail with following messages in the
  actlog on the source server:
  
  ANR9999D_3640283921 smReplSnapshotObjs(smrepl.c:11677)
      Thread<1184498>: One of handle ((nil)) or sessP ((nil)) is
                       NULL
  ANR9999D Thread<1184498> issued message 9999 from:
  ..      0x1486140 /opt/tivoli/tsm/server/bin/dsmserv
                    (OutDiagToCons+0xb0)
  ..      0x148e660 /opt/tivoli/tsm/server/bin/dsmserv
                    (outDiagfExt+0x1f0)
  ..      0x12d07f5 /opt/tivoli/tsm/server/bin/dsmserv
                    (smReplSnapshotObjs+0x- 175)
  ..       0xca0102 /opt/tivoli/tsm/server/bin/dsmserv
                    (paReplicateSnapshot+0- x152)
  ..      0x14955ec /opt/tivoli/tsm/server/bin/dsmserv
                    (PcConsumerThread+0x77- c)
  ..       0xb4c0b7 /opt/tivoli/tsm/server/bin/dsmserv
                    (paReplicateSnapshotSt- ackTrackEntry+0x17)
  ..      0x152b8a3 /opt/tivoli/tsm/server/bin/dsmserv
                    (StartThread+0x93)
  .. 0x7ffff78341ca /lib64/libpthread.so.0(*UNKNOWN*)
  .. 0x7ffff3b7de73 /lib64/libc.so.6(clone+0x43)
  
  This happens only if the SPP backup policy contains multiple
  VM backups and from the IBM Storage Protect server side there
  are two conditions to be met for this APAR to apply.
  
  1. The activity log shows the following synchronize commands
     after the backup completed:
     ANR2017I Administrator IBM-OSSM-ADMIN issued command:
              SYNCHRONIZE VOLUME nodeid=XX fsid=YY  (SESSION: 500)
     The synchronization command is an expected behavior.
  2. The output of the following select command from the Db2
     prompt being the IBM Storage Protect Server instance user is
     not empty :
     "select pasi.nodeid, pasi.fsid FROM tsmdb1.pa_snapshot_info
  pasi WHERE pasi.nodeid=<node_id> and NOT EXISTS(select 1 from
  tsmdb1.pa_snapshot_objects paso where pasi.nodeid=paso.nodeid
  and pasi.fsid=paso.fsid and paso.snapid=0) group by pasi.nodeid,
  pasi.fsid order by pasi.nodeid, pasi.fsid"
  
     If the output returns rows like ...
        NODEID   FSID
        -------  ------
        35       101
        35       102
        35       103
        3 record(s) selected.
     then this APAR applies.
  
  IBM Storage Protect Versions Affected:
  IBM Storage Protect server 8.1.16 and later on all supported
  platforms
  
  Additional Keywords: Spectrum Protect, TSM, Tivoli, TS015626827,
                       SPP
  

Local fix

• When no VMware guest backup using OSSM-enabled vADPs is active,
  Run the following Db2 commands being the instance user on the
  source IBM Storage Protect server:
     db2 connect to tsmdb1
     db2 set schema tsmdb1
     db2 "delete from tsmdb1.pa_snapshot_info pasi where
  pasi.nodeid=<node_id> and not exists (select 1 from
  tsmdb1.pa_snapshot_objects paso where pasi.nodeid=paso.nodeid
  AND pasi.fsid=paso.fsid AND pasi.snapid=paso.snapid)"
  
  New incorrect entries can be created in the
  IBM Storage Protect Server after applying this local fix.
  Running this command again may be needed until a fixing version
  is installed.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Storage Protect server users of OSSM.                *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available.                           *
  * This problem is currently projected to be fixed in level     *
  * 8.1.24.                                                      *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release: AIX, Linux.
  Platforms fixed: AIX, Linux.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels