Fixes are available
8.5.5.4: WebSphere Application Server V8.5.5 Fix Pack 4
PI17533;8.5.5: jms message producer and consumer applications fail to produce an
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.5.5.5: WebSphere Application Server V8.5.5 Fix Pack 5
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
JMS application is configured to send messages to Websphere Application Server v7.0.x Service Integration Bus destination and an MDB is configured to consume messages from the destination.The message producer and consumer application suddenly stop producing and consuming messages from the destination.Javacores collected from the server hosting the applications and the server hosting the messaging engine show the threads in blocked state with the following stack: On the server hosting the messaging engine,7 out of 8 SpillDispatcher threads were hung while adding messages to the filestore: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/ws/objectManager/AbstractSingleFileObjectStore.reserve(A bstractSingleFileObjectStore.java:718(Compiled Code)) at com/ibm/ws/objectManager/ManagedObject.reserveSpaceInStore(Manag edObject.java:663(Compiled Code)) at com/ibm/ws/objectManager/ManagedObject.getSerializedBytes(Manage dObject.java:637(Compiled Code)) at com/ibm/ws/objectManager/InternalTransaction.add(InternalTransac tion.java:780(Compiled Code)) at com/ibm/ws/objectManager/Transaction.add(Transaction.java:243(Co mpiled Code)) at com/ibm/ws/sib/msgstore/persistence/objectManager/PersistableImp l.addToStore(PersistableImpl.java:454(Compiled Code)) at com/ibm/ws/sib/msgstore/persistence/objectManager/BatchingContex tImpl.insert(BatchingContextImpl.java:205(Compiled Code)) at com/ibm/ws/sib/msgstore/task/AddTask.persist(AddTask.java:371(Co mpiled Code)) at com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S pillDispatcherThread.writeBatch(SpillDispatcher.java:1669(Compil ed Code)) at com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S pillDispatcherThread.run(SpillDispatcher.java:711(Compiled Code)) at java/lang/Thread.run(Thread.java:738(Compiled Code)) On the server hosting the message producer and consumer applications 90+ threads were found with the following stack: Consumer threads: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo mplete(ExchangeReceiveListener.java:227(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver sationImpl.java:988(Compiled Code)) at com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange(JFAPCo mmunicator.java:386(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy.performReceive( ConsumerSessionProxy.java:914(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy._receiveWithWai t(ConsumerSessionProxy.java:825(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy.receiveWithWait (ConsumerSessionProxy.java:733(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receiveInboundMes sage(JmsMsgConsumerImpl.java:1114(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receive(JmsMsgCon sumerImpl.java:481(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueReceiverImpl.receive(JmsQueu eReceiverImpl.java(Compiled Code)) Producer threads: at com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi on(ConnectionProxy.java:1007(Compiled Code)) at com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio n(ConnectionProxy.java:932(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.<init>(JmsMsgProd ucerImpl.java:369(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueSenderImpl.<init>(JmsQueueSe nderImpl.java:60(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueSessionImpl.instantiateProdu cer(JmsQueueSessionImpl.java:224(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer(JmsSes sionImpl.java:866(Compiled Code))
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: Users of the default messaging provider * * for IBM WebSphere Application Server V7.0, * * V8.0 and V8.5 * **************************************************************** * PROBLEM DESCRIPTION: Message producer application suddenly * * stops producing messages to the * * destination because dispacher threads * * are hung. * **************************************************************** * RECOMMENDATION: * **************************************************************** When persisting messages into the filestore, an object manager needs to ensure free space in the filestore to store the messages. So the "SpillDispatcher" thread is indefinitely waiting in AbstractSingleFileObjectStore.reserve(). Another thread notifies the waiting thread based on a condition which is controlled by a boolan variable which is not thread safe. So the waiting "SpillDispatcher" thread may not get notified and waits forever and this may eventually happen to all the dispacther threads. When all the dispacther threads are hung, the message producer hangs as there are no dispacther threads to persist messages. Consumer application waits on receiveWithWait() because there are no messages in the destination. This is a rare problem due to timing especially when there are more CPUs. This problem is applicable if file store is being used as a message store.
Problem conclusion
Source code has been modified to ensure that the boolean variable is thread safe. The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.37, 8.0.0.10, and 8.5.5.4. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI17533
Reported component name
WAS SIB & SIBWS
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2014-05-08
Closed date
2014-08-18
Last modified date
2014-08-18
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WAS SIB & SIBWS
Fixed component ID
620800101
Applicable component levels
R800 PSY
UP
R850 PSY
UP
Document Information
Modified date:
28 April 2022