APAR status
Closed as program error.
Error description
The MQ 9.3 LTS Telemetry service is installed onto a Linux system. After running for a number of months, the /var/mqm/errors directory contains approximately 100,000 files called MQXR<number>.FDC. Looking at the files, it can be seen that they are generated by the MQ Telemetry service and fall into two main categories: Category 1: --------------- FDCs containing probe identifier XR071024 and an "Exception Cause:" reporting a java.io.IOException. An example of one of these is shown below: +========================================FFDC Start================================================+ | Product :- IBM MQ Telemetry First Failure Symptom Report | ...... | Operating System :- Linux | | Process Id :- 25182 | | Product Long Name :- IBM MQ Telemetry classes for Java | | Vendor :- IBM Corporation | | Program Name :- java | | UserID :- mqm | | Java Version :- 1.8.0_361 | | Thread :- name=XBOB0021.TT.PLAIN ServerWorker2 priority=5 group=main ccl=sun.misc.Lau | | :- ncher$AppClassLoader@c58611e4 | | Source Class :- com.ibm.mq.MQXRService.MQTTServerSessionV311 | | Source Method :- doAsynchronousWork | | Probe Id :- XR071024 | | FileInformation :- /var/mqm/errors/MQXR0.FDC | ........ |--------------------------------------------------------------- -----------------------------------| No MQM Function Stack Available :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:59) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:105) at sun.nio.ch.IOUtil.write(IOUtil.java:77) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:485) at com.ibm.mq.communications.NonBlockingConnection.send(NonBlocking Connection.java:230) at com.ibm.mq.MQXRService.MQTTServerSessionV311.doAsynchronousWork( MQTTServerSessionV311.java:1689) at com.ibm.mq.MQXRService.MQTTServerContext.doAsynchronousWork(MQTT ServerContext.java:197) at com.ibm.mq.communications.NonBlockingWorker.synchronousWorkCompl eted(NonBlockingWorker.java:631) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:422) at java.lang.Thread.run(Thread.java:825) Category 2: --------------- FDCs containing probe identifier XR22122014 and an "Exception Cause" reporting TransactionFullExceptions, as shown in this example: +========================================FFDC Start================================================+ | Product :- IBM MQ Telemetry First Failure Symptom Report | ...... | Operating System :- Linux | | Process Id :- 31646 | | Product Long Name :- IBM MQ Telemetry classes for Java | | Vendor :- IBM Corporation | | Program Name :- java | | UserID :- mqm | | Java Version :- 1.8.0_361 | | Thread :- name=XBOB0021.TT.PLAIN ServerWorker4 priority=5 group=main ccl=sun.misc.Lau | | :- ncher$AppClassLoader@796ceb3c | | Source Class :- com.ibm.mq.MQXRService.MQTTServerSession | | Source Method :- prepareAsynchronousWork | | Probe Id :- XR22122014 | | FileInformation :- /var/mqm/errors/MQXR21558.FDC | ........ |--------------------------------------------------------------- -----------------------------------| No MQM Function Stack Available :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.MQXRService.TransactionFullException: TransactionFullException_info at com.ibm.mq.MQXRService.MQTTServerSession.clearDurableSubs(MQTTSe rverSession.java:676) at com.ibm.mq.MQXRService.MQTTServerSession.purge(MQTTServerSession .java:491) at com.ibm.mq.MQXRService.MQTTServerSession.purge(MQTTServerSession .java:452) at com.ibm.mq.MQXRService.MQTTServerSessionV311.completeConnection( MQTTServerSessionV311.java:613) at com.ibm.mq.MQXRService.MQTTServerSession.prepareAsynchronousWork (MQTTServerSession.java:1050) at com.ibm.mq.MQXRService.TempMQTTContext.prepareAsynchronousWork(T empMQTTContext.java:60) at com.ibm.mq.communications.NonBlockingWorker.synchronousWorkCompl eted(NonBlockingWorker.java:622) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:422) at java.lang.Thread.run(Thread.java:825)
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the MQ Telemetry service. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: The MQ Telemetry service used to generate FDC files in the following scenarios: Scenario 1: ----------------- The MQ 9,.3 Telemetry service would create an FDC containing probe identifier XR071024 and an "Exception Cause" reporting an IOException if an MQTT client disconnected unexpectedly when service was trying to send some data to it Scenario 2: ----------------- If an MQTT client sent an MQTT unsubscribe packet containing an invalid topic filter to the MQ 9.3 Telemetry service, then the service would generate an FDC containing probe identifier XR071003. Scenario 3: ----------------- The MQ 9.3 and 9.4 Telemetry services would also generate an FDC containing probe identifier XR22122014 and an "Exception Cause:" reporting a TransactionFullException if they detected that an internal transaction used to perform MQ API calls had filled up. As these conditions were not serious issues, the MQ Telemetry service should not have generated FDCs for them.
Problem conclusion
To resolve this issue, the following changes have been made: Change 1: --------------------- Firstly, the MQ 9.3 Telemetry service now writes the message: AMQXR2123E: Exception ''{0}'' occurred and the connection to the client ''{1}'' at network address ''{2}'' connected to channel ''{3}'' was closed. to its log file (mqxr_<number>.log) if an MQTT client disconnects unexpectedly, rather than generating an FDC containing probe identifier XR071024. Change 2: --------------------- If the MQ 9.3 Telemetry service receives an MQTT unsubscribe packet containing an invalid topic filter from an MQTT client, it will now write the message: AMQXR2122E: An unsubscribe request from client ''{1}({2})'' failed on channel ''{0}'' because the client specified an invalid Topic Filter ''{3}''. to its log file. Change 3: --------------------- The last change affects both the MQ 9.3 and 9.4 Telemetry services. Rather than generating FDCs containing probe identifier XR22122014 if an internal TransactionFullException occurs, the services will instead write the message: AMQXR2124W: Thread ''{0}'' was unable to add ''{1}'' operations for client ''{2}'' to its internal transaction. [Current operations: ''{3}''. Maximum number of operations: ''{4}''. Current subscription operations: ''{5}''. Maximum number of subscription operations: ''{6}'']. to their log files. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.3 LTS 9.3.0.25 v9.4 LTS 9.4.0.5 v9.x CD 9.4.1 The latest available maintenance can be obtained from 'IBM MQ Recommended Fixes' https://www.ibm.com/support/pages/recommended-fixes-ibm-mq If the maintenance level is not yet available information on its planned availability can be found in 'IBM MQ Planned Maintenance Release Dates' https://ibm.biz/mqplannedmaintenance ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT45631
Reported component name
MQ BASE V9.3
Reported component ID
5724H7291
Reported release
930
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2024-03-04
Closed date
2024-07-04
Last modified date
2024-07-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
MQ BASE V9.3
Fixed component ID
5724H7291
Applicable component levels
[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.3","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"}}]
Document Information
Modified date:
04 July 2024