A fix is available
APAR status
Closed as program error.
Error description
A deadlock in the Control Region causes the Servant to timeout waiting on a response from the Adjunct during transaction recovery of IN-COMMIT transactions during server startup. A javacore of the Servant Region shows a thread with the following stack: 4XESTACKTRACE at java/lang/Object.wait(Native Method) 4XESTACKTRACE at java/lang/Object.wait(Object.java:199) 4XESTACKTRACE at com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo mplete(ExchangeReceiveListener.java:235) 4XESTACKTRACE at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver sationImpl.java:918) 4XESTACKTRACE at com/ibm/ws/sib/comms/common/JFAPCommunicator.JFAPExchange(JFAPCo mmunicator.java:382) 4XESTACKTRACE at com/ibm/ws/sib/comms/client/BaseSIXAResourceProxy.internalRecove r(BaseSIXAResourceProxy.java:518) 4XESTACKTRACE at com/ibm/ws/sib/comms/client/OptimizedSIXAResourceProxy.recover(O ptimizedSIXAResourceProxy.java:883) 4XESTACKTRACE at com/ibm/ws/sib/ra/recovery/impl/SibRaRecoveryXaResource.recover( SibRaRecoveryXaResource.java:223) 4XESTACKTRACE at com/ibm/ws/Transaction/JTA/XARminst.recover(XARminst.java:138) 4XESTACKTRACE at com/ibm/ws390/tx/XARecoveryAgentImpl.commit(XARecoveryAgentImpl. java:965) 4XESTACKTRACE at com/ibm/ws390/tx/XARecoveryAgentImpl$XARecoveryAgentThread.run(X ARecoveryAgentImpl.java:358) The resource that is being committed is an MQ Server resource.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V7.0 MQ Server Bus Members * **************************************************************** * PROBLEM DESCRIPTION: Deadlock in Control Region leading to * * ABEND when attempting to commit MQ * * Server IN-COMMIT transaction. * **************************************************************** * RECOMMENDATION: * **************************************************************** MQ Server uses JCA 1.5 transaction in-flow interfaces as part of its normal operation. These interfaces enable MQ Server Bus members to coordinate WMQ and Service Integration Bus resources using WebSphere Application Server transactions. During commit of a transaction at recovery time, xa_recover is called to ensure some older DB2 drivers do not return XAER_NOTA even if the the transaction does exist when xa_commit or xa_rollback is issued. The deadlock ensues after the xa_recover call is issued against an MQ Server resource at recovery time. This is because the MQ Server resource calls back into the Control Region and attempts to lock all known transactions to generate a list of transactions needed to respond to the xa_recover call. The IN-COMMIT transaction being committed is already locked so the xa_recover call hangs waiting to lock the IN-COMMIT transaction which will not be unlocked until the transaction commits.
Problem conclusion
This APAR introduces a new custom property of the transaction service: ZOS_RECOVER_BEFORE_COMMIT (default true) The default value "true" has no effect. When set to "false" this property prevents the xa_recover call prior to xa_commit or xa_rollback during the processing of transactions at recovery time. This avoids the deadlock described by this APAR. To set this property: 1. Login to the Administrative Console. 2. Select Servers -> Application Servers -> <server> -> Container Services -> Transaction Service. 3. Select the Configuration tab. 4. Select Custom Properties. 5. Click new. 6. Enter ZOS_RECOVER_BEFORE_COMMIT as the name, and "false" as the value. 7. Click Ok. 8. Save changes. 9. Restart the server. All up to date DB2 drivers have an auto-recover feature which makes xa_recover prior to xa_commit or xa_rollback unnecessary. If unsure, or it is not clear from the documentation for the version of the DB2 driver used, consult IBM DB2 support for final clarification on whether setting this property is safe for servers hosting applications that access DB2. APAR PM88418 requires changes to documentation. NOTE: Periodically, we refresh the documentation on our Web site, so the changes might have been made before you read this text. To access the latest on-line documentation, go to the product library page at: http://www.ibm.com/software/webservers/appserv/library The following changes to the WebSphere Application Server Version 6.1 Information Center will be made available in June 2013. The topic "Transaction service custom properties" will be updated to include the following description of the ZOS_RECOVER_BEFORE_COMMIT custom property: ZOS_RECOVER_BEFORE_COMMIT Specifying this property prevents a deadlock from occurring after an xa_recover call is issued against an MQ Server resource at recovery time. The MQ Server uses JCA 1.5 transaction in-flow interfaces as part of its normal operation. These interfaces enable MQ Server Bus members to coordinate WMQ and Service Integration Bus resources using WebSphere Application Server transactions. During the commit of any transaction at recovery time, xa_recover is called to ensure some older DB2 drivers do not return XAER_NOTA even if the transaction exists when the xa_commit or xa_rollback call is issued. This deadlock occurs because the MQ Server resource issues calls back to the controller and attempts to lock all known transactions so that the MQ Server resource can generate a list of transactions that need to respond to the xa_recover call. However, because the IN-COMMIT transaction being committed is already locked, the xa_recover call waits indefinitely to lock the IN-COMMIT transaction because that transaction will not be unlocked until the transaction commits. Setting this property to FALSE ensures that during the processing of transactions at recovery time, the xa_recover call is not issued before an xa_commit or xa_rollback call. Avoid Trouble: All currently supported DB2 drivers have an auto-recover feature which makes it unnecessary to issue an xa_recover calls prior to issuing an xa_commit or xa_rollback call. If, after reading the documentation for your DB2 driver, you are not sure whether your DB2 driver includes the auto-recover feature, contact IBM DB2 support for final clarification on whether setting this property is safe to use for servers hosting applications that access DB2. Table 8: ZOS_RECOVER_BEFORE_COMMIT custom property Data Type Boolean Accepted Values TRUE, FALSE Default TRUE APAR PM88418 is currently targeted for inclusion in Fix Pack 7.0.0.29 of WebSphere Application Server V7.0. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM88418
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2013-05-03
Closed date
2013-05-03
Last modified date
2013-07-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R700 PSY UK94926
UP13/06/20 P F306
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
10 February 2022