APAR status
Closed as program error.
Error description
When using the Default JMS Provider, JMS messages may remain stuck on the queues in either committing or removing state. The messages remain in these states even after expiry of TTL. The SystemOut.log of the JVM hosting the messaging engine shows errors like the following: CWSIC2009E: There was a failure to send a message: exception com.ibm.ws.sib.processor.exceptions.SIMPErrorException: com.ibm.websphere.sib.exception.SIErrorException: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: java.lang.RuntimeException: Found invalid balance factor: -1. CWSIP0002E: An internal messaging error occurred in com.ibm.ws.sib.processor.impl.ConsumerDispatcher, 1:786:1.280.7.4, com.ibm.ws.sib.msgstore.TransactionException: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: java.lang.RuntimeException: Found invalid balance factor: -1 CWSIP0002E: An internal messaging error occurred in com.ibm.ws.sib.processor.impl.JSLockedMessageEnumeration, 1:272:1.8.1.10, com.ibm.ws.sib.msgstore.TransactionException: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: java.lang.NullPointerException CWSIP0003E: An internal messaging error occurred JMS/OperationFeedbackCmOpenApi in com.ibm.ws.sib.processor.impl.destination.PubSubRealization, 1:2646:1.35.2.5, com.ibm.ws.sib.msgstore.TransactionException: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.gbs.GBSTreeException: optimisticDelete, v1 = 5384170 FFDC is produced with the following stack: Exception:com.ibm.ws.sib.msgstore.MessageStoreRuntimeException SourceId:com.ibm.ejs.util.am._Alarm.run ProbeId:95 Reporter:com.ibm.ejs.util.am._Alarm@2620b3b3 com.ibm.ws.sib.msgstore.MessageStoreRuntimeException: CWSIS2001E: The expiry daemon ended abnormally due to an unexpected exception: com.ibm.ws.sib.msgstore.gbs.OptimisticDepthException: maxDepth (47) exceeded. Depth is 48. at com.ibm.ws.sib.msgstore.expiry.Expirer.alarm(Expirer.java:598) at com.ibm.ejs.util.am._Alarm.runImpl(_Alarm.java:151) at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:136) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909) Caused by: com.ibm.ws.sib.msgstore.gbs.OptimisticDepthException: maxDepth (47) exceeded. Depth is 48. at com.ibm.ws.sib.msgstore.gbs.NodeStack.push(NodeStack.java:155) at com.ibm.ws.sib.msgstore.gbs.NodeStack.push(NodeStack.java:143) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.nextNode(GBSIterator.jav a:704) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.internalGetNext(GBSItera tor.java:656) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.pessimisticGetNext(GBSIt erator.java:613) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.findNextBeforeEof(GBSIte rator.java:411) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.findNext(GBSIterator.jav a:381) at com.ibm.ws.sib.msgstore.gbs.GBSIterator.next(GBSIterator.java:75 1) at com.ibm.ws.sib.msgstore.expiry.ExpiryIndex.next(ExpiryIndex.java :151) at com.ibm.ws.sib.msgstore.expiry.Expirer.alarm(Expirer.java:526)
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: WebSphere Application Server users of the * * Default JMS Provider (SIBus) * **************************************************************** * PROBLEM DESCRIPTION: For systems using message expiry under * * heavy loads, processing of expired * * messages may fail and leave messages * * locked. * **************************************************************** * RECOMMENDATION: * **************************************************************** Messages seen stuck in committing or removing state.
Problem conclusion
The SIBus message store component was modified to fix an internal concurrency error. The fix for this APAR is targeted for inclusion in fix packs 8.5.5.19 and 9.0.5.7. For more information, see 'Recommended Updates for WebSphere Application Server': https://www.ibm.com/support/pages/node/715553
Temporary fix
Restarting the messaging engine will restore normal operation.
Comments
APAR Information
APAR number
PH28719
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-08-21
Closed date
2020-12-01
Last modified date
2020-12-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R850 PSY
UP
R900 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
01 November 2021