APAR status
Closed as program error.
Error description
When IBM MQ v8 JMS application creates/closes many connections to a queue manager, there is gradual CPU usage increase over time, and an increase in the JVM memory usage. This causes serious impact on the JVM and GC. Over 30 day period the system/CPU load (as reported by Linux) is increasing, from around 0.1-0.5 to 3-5%. Memory allocation increase, Every time full GC is performed, memory goes down, but memory allocation increases, and as a consequence, frequency of (full) GCs is increasing. Once the system is under increased load (even though it's idle really) performing a particular user action/request can push the system over the line and it then does not recover.
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The IBM MQ V8 allclient JAR file. - The IBM MQ V8 classes for JMS. - The IBM MQ V8 classes for JMS OSGi bundle. - The IBM MQ V8 resource adapter. who have JMS applications that: - Connect to a queue manager using either IBM MQ messaging provider normal mode, or IBM MQ messaging provider normal mode with restrictions, using the CLIENT transport. - And connect using an IBM MQ channel instance that has the SHARECNV attribute set to a value greater than 1. - And create lots of short-lived JMS connections, JMS sessions or JMSContexts. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: When an IBM MQ classes for JMS application connects to a queue manager in either IBM MQ messaging provider normal mode, or IBM MQ messaging provider normal mode with restrictions, using the CLIENT transport, it can make use of the "conversation sharing" functionality provided by the queue manager. This allows multiple conversations between the application and the queue manager to use the same underlying channel instance (or TCP/IP connection, if you prefer to think of it in those terms). The maximum number of conversations that can take place over a single channel instance is specified by the IBM MQ channel property SHARECNV. The default value is 10. For JMS applications, every JMS Connection starts a conversation with the queue manager. Similarly, every JMS Session created by an application also starts a conversation with the queue manager. Each JMSContext starts two conversations with a queue manager. Every conversation that is started over a channel instance has a unique number (or identifier), which increments by 2 - the first conversation over a specified channel instance is given the number 1, then second conversation over the same channel instance has the number 3, and so on. These conversation identifiers are generated by the queue manager. If the WebSphere MQ classes for JMS are required to start a new conversation on a new channel instance, then that conversation will have the identifier 1. The IBM MQ classes for JMS maintain internal lists containing information about the conversations that are currently active over a channel instance. When a new conversation is started, details of that conversation are added to the internal list for the appropriate channel instance. When a conversation ends, the entry for that conversation is removed from the relevant list. These lists were defined as arrays, which was keyed off of the conversation identifier. When a new conversation was created on the channel instance, some information about that conversation was put into the array at a position specified by the identifier for that conversation. If an application created lots of short lived conversations over a single channel instance, then the following behaviour occurred: - A JMS Connection was created. This resulted in a new channel instance being created, as well as a new conversation being started. This conversation had identifier 1, and so went into position 1 in the array for the channel instance. - A short-lived JMS Session was created. A new conversation was started with the queue manager, which had conversation identifier 3. The array was resized to contain 3 elements, and details about this conversation went into position 3 in the array. - When the JMS Session was closed, the conversation was ended and details about it were removed from position 3 in the array. - A second short-lived JMS Session was created and a new conversation was started with the queue manager. This had conversation identifier 5. The array was resized to contain 5 elements, and the details about the new conversation went into position 5 in the array. - When this JMS Session was closed, the conversation was ended and details about it were removed from position 5 in the array. If we move forward in time a bit now, the following behaviour was seen: - A short-lived JMS Session was created from the original JMS connection (which had conversation identifier 1). A new conversation was started with the queue manager, which had conversation identifier 505. The array was resized to contain 505 elements, and information about this conversation went into position 505 in the array. - When the JMS Session was closed, the conversation was ended and details about it were removed from position 505 in the array. And moving forward a bit further in time: - A short-lived JMS Session was created from the original JMS connection (which had conversation identifier 1) and a new conversation was started with the queue manager. This had conversation identifier 5005. The array was resized to contain 5005 elements, and details about this conversation went into position 5005 in the array. - When this JMS Session was closed, the conversation was ended and details about it were removed from position 5005 in the array. And again: - A short-lived JMS Session was created from the original JMS connection (which had conversation identifier 1). A new conversation was started with the queue manager, which had conversation identifier 50005. The array was resized to contain 50005 elements, and information about the new conversation went added to the array in position 50005. - When the JMS Session was closed, the conversation was ended and details about it were removed from position 50005 in the array. This resulted in the Java Runtime Environment creating a bigger array each time a new conversation was started, even though only a few elements in the array were actually being used...... Because of the size of the array, the memory usage of the Java Runtime Environment that the application was running in kept growing and shrinking, which resulted in high CPU usage as the Java Runtime Environment had to perform lots of garbage collection to free up enough memory to hold the array.
Problem conclusion
The IBM MQ classes for JMS have been updated to use sets to store information about the conversations associated with the object, rather than an array. When a new conversation is started, an entry is added to the set. The entry will be removed (and the size of the set decremented) when the conversation finishes. This ensures that the size of the set will never be larger than the maximum number of conversations that can take place over the channel instance (as defined by the IBM MQ channel attribute SHARECNV). --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.4 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT09442
Reported component name
WMQ BASE MULTIP
Reported component ID
5724H7251
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2015-06-12
Closed date
2015-09-14
Last modified date
2015-09-14
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ BASE MULTIP
Fixed component ID
5724H7251
Applicable component levels
R800 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
14 September 2015