PH40286: CICS Liberty JVM Server - Threadlocal leaks with JCICS late binding

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• This is an out of memory error in the JVM.
  .
  JVMDUMP039I Processing dump event "systhrow", detail
  "java/lang/OutOfMemoryError"
                                                                 .
  From the dump, there were a couple of ThreadLocal leaks
  occurring in the JVM for JVM Server.
                                                                 .
  Leak 1, the TASK_CONTEXT_STACK Stack.
  Example from the dump:
  Below the java.util.Stack had 19,125 objects of type
  com.ibm.cics.wlp.impl.CICSTaskWrapper.
                                                                 .
  Class Name
                                    | Shallow Heap | Retained Heap
  ----------------------------------------------------------------
  [15] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x161219bb8
                                    |           32 |    30,831,752
  |- <class> class java.lang.ThreadLocal$ThreadLocalMap$Entry @
  0x156e0a308 System Class          |          585 |           585
  |- referent java.lang.ThreadLocal @ 0x1578a5968
                                    |           16 |            16
  |- value java.util.Stack @ 0x161219bd8
                                    |           24 |    30,831,720
  ----------------------------------------------------------------
                                                                 .
  Leak 2; the taskListeners ArrayList.
  Example from the dump.
  Below the java.util.ArrayList had 19,150 objects of type
  com.ibm.cics.wlp.jta.impl.CICSUOWEventListener$1
                                                                 .
  Class Name
                                    | Shallow Heap | Retained Heap
  ----------------------------------------------------------------
  [24] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x162303590
                                    |           32 |    49,534,096
  |- <class> class java.lang.ThreadLocal$ThreadLocalMap$Entry @
  @ 0x156e0a308 System Class        |          585 |           585
  |- referent java.lang.ThreadLocal @ 0x1578a37d0
                                    |           16 |            16
  |- value java.util.ArrayList @ 0x162303a98
                                    |           24 |    49,534,064
  ----------------------------------------------------------------
  
  Note that the report fragments above can be generated by
  consuming the system dump in the Eclipse MAT tool, which can
  be downloaded from:
  https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-d
  tfj-and-ibm-extensions
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All CICS Users.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: JVMDUMP039I OutOfMemoryError in a       *
  *                      Liberty JVM server.                     *
  ****************************************************************
  CICS is running with a Liberty JVM server.
  During a request execution for an installed application it
  becomes necessary to create a CICS Task.
  This dynamic creation of the structures needed to support the
  CICS Task at the point it is required is referred to as
  'Late Binding'.
  
  (1) Leak 1
  
  An object of Type CICSTaskContext is pushed onto a Threadlocal
  based stack during execution. This object is used (for example)
  to decide if CICS API calls are allowable on this Thread.
  The object is popped off the Stack after the request completes.
  
  - Code changes were made with APAR PH17250 to correctly clean
  up the task state after a CJSP task has completed, these
  included an additional push of a CICSTaskContext object on to
  the Stack, which was not subsequently popped off.
  
  - If enough transactions that use late binding and require a
  CICS Task are run, excessive memory can be reachable via
  this Stack of CICSTaskContext objects
  
  (2) Leak 2
  
  - When a transaction under Liberty starts a Listener object of
  type com.ibm.cics.wlp.jta.impl.CICSUOWEventListener$1 is added
  to a ThreadLocal based ArrayList. The Listener is notified if
  a CICS Task is created and is able to enlist CICS in the
  transaction, so that any CICS resources can be committed or
  rolled back as appropriate when the transaction ends.
  This Listener object was not removed from the ArrayList.
  
  - If enough transactions that use late binding and require a
  CICS Task are run, excessive memory can be reachable via
  this ArrayList of CICSUOWEventListener$1 objects
  
  Since the reachability of the individual objects is small, to
  encounter this problem you would need to run a high number
  of transactions where late binding occurs in a modestly sized
  Java heap.
  

Problem conclusion

• CICS code has been modified to prevent the memory leaks.
  
  The additional push of a CICSTaskContext object is now only done
  for system Tasks.
  
  The Listener object is now added into a Map for a specific
  transaction and is removed when the transaction completes.
  
  Keywords: OOM OutOfMemoryError JVMDUMP039I CICSTaskContext
  CICSUOWEventListener Liberty ThreadLocal
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI79778 UI79779

Modules/Macros

• DFJ@H351 DFJ@H387 DFJ@H415
  

Fix information

• Fixed component name

  CICS TS Z/OS V5

• Fixed component ID

  5655Y0400

Applicable component levels

• R200 PSY UI79779

     UP22/03/19 P F203

• R300 PSY UI79778

     UP22/03/19 P F203

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.