PI17533;8.5.5: jms message producer and consumer applications fail to produce an

Download Description

PI17533 resolves the following problem:

ERROR DESCRIPTION:
JMS application is configured to send messages to Websphere Application Server v7.0.x Service Integration Bus destination and an MDB is configured to consume messages from the destination.The message producer and consumer application suddenly stop producing and consuming messages from the destination.Javacores collected from the server hosting the applications and the server hosting the messaging engine show the threads in blocked state with the following stack:

On the server hosting the messaging engine,7 out of 8 SpillDispatcher threads were hung while adding messages to the filestore:

at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at
com/ibm/ws/objectManager/AbstractSingleFileObjectStore.reserve(A
bstractSingleFileObjectStore.java:718(Compiled Code))
at
com/ibm/ws/objectManager/ManagedObject.reserveSpaceInStore(Manag
edObject.java:663(Compiled Code))
at
com/ibm/ws/objectManager/ManagedObject.getSerializedBytes(Manage
dObject.java:637(Compiled Code))
at
com/ibm/ws/objectManager/InternalTransaction.add(InternalTransac
tion.java:780(Compiled Code))
at
com/ibm/ws/objectManager/Transaction.add(Transaction.java:243(Co
mpiled Code))
at
com/ibm/ws/sib/msgstore/persistence/objectManager/PersistableImp
l.addToStore(PersistableImpl.java:454(Compiled Code))
at
com/ibm/ws/sib/msgstore/persistence/objectManager/BatchingContex
tImpl.insert(BatchingContextImpl.java:205(Compiled Code))
at
com/ibm/ws/sib/msgstore/task/AddTask.persist(AddTask.java:371(Co
mpiled Code))
at
com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S
pillDispatcherThread.writeBatch(SpillDispatcher.java:1669(Compil
ed Code))
at
com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S
pillDispatcherThread.run(SpillDispatcher.java:711(Compiled
Code))
at java/lang/Thread.run(Thread.java:738(Compiled Code))


On the server hosting the message producer and consumer applications 90+ threads were found with the following stack:

Consumer threads:

at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at
com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo
mplete(ExchangeReceiveListener.java:227(Compiled Code))
at
com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver
sationImpl.java:988(Compiled Code))
at
com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange(JFAPCo
mmunicator.java:386(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy.performReceive(
ConsumerSessionProxy.java:914(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy._receiveWithWai
t(ConsumerSessionProxy.java:825(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy.receiveWithWait
(ConsumerSessionProxy.java:733(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receiveInboundMes
sage(JmsMsgConsumerImpl.java:1114(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receive(JmsMsgCon
sumerImpl.java:481(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueReceiverImpl.receive(JmsQueu
eReceiverImpl.java(Compiled Code))

Producer threads:

at
com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi
on(ConnectionProxy.java:1007(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio
n(ConnectionProxy.java:932(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.<init>(JmsMsgProd
ucerImpl.java:369(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueSenderImpl.<init>(JmsQueueSe
nderImpl.java:60(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueSessionImpl.instantiateProdu
cer(JmsQueueSessionImpl.java:224(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer(JmsSes
sionImpl.java:866(Compiled Code))

LOCAL FIX:
None

PROBLEM SUMMARY

USERS AFFECTED:
Users of the default messaging provider for IBM WebSphere Application Server V7.0,
V8.0 and V8.5

PROBLEM DESCRIPTION:
Message producer application suddenly stops producing messages to the destination because dispacher threads are hung.

RECOMMENDATION:
None

When persisting messages into the filestore, an object manager needs to ensure free space in the filestore to store the messages. So the "SpillDispatcher" thread is indefinitely waiting in AbstractSingleFileObjectStore.reserve(). Another thread notifies the waiting thread based on a condition which is controlled by a boolean variable which is not thread safe. So the waiting "SpillDispatcher" thread may not get notified and waits forever and this may eventually happen to all the dispacther threads.
When all the dispacther threads are hung, the message producer hangs as there are no dispacther threads to persist messages. Consumer application waits on receiveWithWait() because there are no messages in the destination. This is a rare problem due to timing especially when there are more CPUs. This problem is applicable if file store is being used as a message store.

PROBLEM CONCLUSION:
Source code has been modified to ensure that the boolean variable is thread safe.

The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.37, 8.0.0.10, and 8.5.5.4. Please refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980