IT23387: Queue manager ends unexpectedly with error AMQ7047 and FDC with probe ID AO063006 due to object catalog corruption

APAR status

• Closed as program error.

Error description

• Queue manager ended unexpectedly with the following errors.
  Although  customer is using linear logging, an attempt to
  restore the catalog also failed.
  
  A failure data capture (FDC) record similar to the following is
  generated:-
  
  Operating System  :- Linux 4.4.74-92.38-default
  PIDS              :- 5724H7251
  LVLS              :- 9.0.0.1
  Product Long Name :- IBM MQ for Linux (x86-64 platform)
  Vendor            :- IBM
  Probe Id          :- AO063006
  Application Name  :- MQM
  Component         :- aocLoadCatalogue
  Effective UserID  :- 32089 (mqm)
  Real UserID       :- 32089 (mqm)
  Program Name      :- amqzxma0
  Arguments         :- -m XYZ -u mqm
  Addressing mode   :- 64-bit
  Major Errorcode   :- arcE_OBJECT_NOT_FOUND
  Minor Errorcode   :- OK
  Probe Type        :- INCORROUT
  Probe Description :- AMQ6125: An internal IBM MQ error has
  occurred.
  Comment2          :- qmgr
  
  MQM Function Stack
  amqzxma0
  zxcExecutionController
  zxcStartupLogic
  kpiStartup
  apiStartup
  aocLoadCatalogue
  xcsFFST
  
  amqzxma trace shows that it attempts to look up a null object
  name in the object catalog and fails.
  
  09:23:48.246906    11327.1  :
  OSName(/var/mqm/qmgrs/XYZ/queues/TEST2)
  09:23:48.246910    11327.1  :       ---------}  adiCheckFile
  rc=OK
  09:23:48.246912    11327.1  :       --------}  adhCheckFile
  rc=OK
  09:23:48.246913    11327.1  :       -------}  aodCheckFile
  rc=OK
  09:23:48.246918    11327.1  :       -------{  adhClose
  09:23:48.246921    11327.1  :       --------{ aduReleaseFileCtl
  09:23:48.246923    11327.1  :       ---------{  adiCloseFile
  09:23:48.246925    11327.1  :
  OSName(/var/mqm/qmgrs/X/qmanager/QMQMOBJCAT)
  ...
  09:23:48.246960    11327.1  :       --------{
  aotLocateEntryByName
  09:23:48.246962    11327.1  :            ObjType: 4, Name: ''
  09:23:48.246963    11327.1  :       --------}!
  aotLocateEntryByName rc=arcE_OBJECT_NOT_FOUND
  09:23:48.246964    11327.1  :       -------}! aouLocateEntry
  rc=arcE_OBJECT_NOT_FOUND
  09:23:48.246966    11327.1  :       -------{ aouUnlockSynChain
  09:23:48.246967    11327.1  :       --------{  xlsReleaseMutex
  09:23:48.246968    11327.1  :            MtxName:
  hmtxSynChain_0_0 Id: 631
  09:23:48.246970    11327.1  :       --------} xlsReleaseMutex
  rc=OK FunctionTime=3
  09:23:48.246971    11327.1  :       -------} aouUnlockSynChain
  rc=OK FunctionTime=5
  09:23:48.246972    11327.1  :       ------}! aocEnquireObject
  rc=arcE_OBJECT_NOT_FOUND FunctionTime=30
  09:23:48.246974    11327.1  :            Queue Manager object
  missing or damaged, (null)
  09:23:48.246981    11327.1  :       ------{  xcsFFST
  09:23:48.246986    11327.1  :            ErrorCode :- 20800825
  Numeric Insert1 :- 00000000 (0) Numeric Insert2 :- 00000000 (0)
  09:23:48.246988    11327.1  :              String Insert  :-
  '(null)'
  09:23:48.246989    11327.1  :              String Insert  :-
  'qmgr'
  09:23:48.246992    11327.1  :       -------{  xcsGetMessage
  09:23:48.246995    11327.1  :            msgid:20006125
  a1:00000000 a2:00000000 c1:(null) c2:qmgr c3:(null)
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  All MQ users
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  The MQ Object catalog had been damaged such that the queue
  manager object did not appear to be in the catalog, but the
  catalog itself was found and otherwise valid.  Incorrect
  handling of this error condition during the queue manager start
  caused MQ to fail to recover the object catalog from the media
  recovery log.
  

Problem conclusion

• The MQ queue manager code has been modified to include number of
  checks to detect or to prevent the object catalog being damaged
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v8.0       8.0.0.11
  v9.0 LTS   9.0.0.4
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  IBM MQ BASE M/P

• Fixed component ID

  5724H7261

Applicable component levels