Threads, Java heap, MEMLIMIT, REGION

How to measure your workload

Before you can calculate your Java heap size and MEMLIMIT, you need to understand your workload, and how to obtain performance measurements.

Some workloads are steady and consistent, whereas others might have peaks and troughs throughout a period.

When you prepare your test workload, have realistic targets, such as the minimum number of transactions per second (TPS), or an average response time per request. You should include a think time in your workload driver to simulate a realistic incoming workload. The think time value represents the delay between a simulated client receiving a response from IBM z/OS Connect and sending it the next request. A large think time reduces the transaction rate, whereas a small think time increases the transaction rate. Do not use a think time of zero as this is not only unrealistic, but risks overwhelming the IBM z/OS Connect server causing threads to queue and potentially time out.

Warming up the IBM z/OS Connect server

Your IBM z/OS Connect server runs in a JVM. A running JVM must perform several critical functions. The most significant of these functions are just-in-time (JIT) compilation and garbage collection (GC). JIT and GC can use significant amounts of CPU in the JVM. Therefore, before you take any performance measurements, it is important to warm up the JIT so it is fully optimized by running several hundred thousand requests through your IBM z/OS Connect server. The first time a transaction is run in Java, the z/Architecture instructions that are produced by the JIT compiler are at a low optimization level, which results in a relatively high CPU cost to run the Java methods. As more transactions are run, the Java method invocation counts are increased. Therefore, the JIT recompiles a Java method to a more aggressive level of optimization. This greater level of optimization results in a Java method that requires less CPU to run than before the recompilation took place. As a result, the CPU that is required to run the transaction reduces. This process is repeated several times during the lifetime of the JVM. The warm-up period completes at the point at which the CPU cost per transaction ceases to show any improvements. After the workload is running in a steady-state for the warm-up period, it is assumed that the JIT compiler will not optimize the IBM z/OS Connect server further, and CPU measurements can be taken.

Shutting down an IBM z/OS Connect server (and thus the JVM it runs in) discards the JIT-compiled native code. Therefore, the iterative process of optimization begins again when the JVM is restarted.

When your workload is running and your IBM z/OS Connect server is warmed up, you can start monitoring thread usage, the Java heap, and native storage usage.

Threads

IBM z/OS Connect uses native threads and Java threads to process requests. For each Java thread, the JVM asks the operating system to create a native thread in the LE native stack storage that the operating system controls.

Native threads

Each native thread requires a minimum of 3 M of LE native stack storage (1M for the JVM and 2M for LE). This storage is allocated above-the-bar. Therefore, the value set for MEMLIMIT must allow enough storage for these threads. The supplied JCL for IBM z/OS Connect sets this value to 8G. For more information, see Calculating the MEMLIMIT.

To observe the number of native threads for the IBM z/OS Connect address space, you can issue MVS commands from SDSF. You need the process ID (PID) number for the IBM z/OS Connect server. You can find the PID from the messages.log file, or by issuing the following MVS command from SDSF: /D OMVS,ASID=<hexadecimal address space id for your server>.

For example, /D OMVS,ASID=B4.

When you have the PID number, you can issue the following MVS command from SDSF: /D OMVS,LIMITS,PID=<server pid>. This command displays the current usage and high water usage values for MAXTHREADS and MAXTHREADTASKS, and the PROCESS LIMIT for your IBM z/OS Connect server. The limit for MAXTHREADS must be greater than or equal to MAXTHREADTASKS to ensure that enough threads are available to handle the tasks currently being processed, as shown in the following output.

OMVS          0011 ACTIVE          OMVS=(22,DB,JV,00,US)                
USER          JOBNAME  ASID        PID       PPID STATE     START      CT_SECS
ZOSPERF       ZOSCONN  00B4        16843242  66037 HK------ 11.12.37    85.8
 LATCHWAITPID=         0 CMD=/java/J64/bin/java -javaagent:/u/zosperf
PROCESS LIMITS:       LIMMSG=NONE                                    
                  CURRENT  HIGHWATER    PROCESS                       
                    USAGE      USAGE      LIMIT                                                                                            
...                  
MAXTHREADS             78         81      15000         <----  MaxThreads must be >= MaxThreadTasks               
MAXTHREADTASKS         79         81       6000 

Java threads: Overview

Java threads are used for many different purposes, such as JIT compilation, GC, and handling IBM z/OS Connect requests. Most Java threads that are used for handling IBM z/OS Connect workloads are executor threads.

Each Java thread requires a minimum of 1.6 K of Java heap storage. You can use a z/OS monitor, such as the Java Health Center, to show the number of active Java threads that are used by IBM z/OS Connect.

Java threads: maxOpenConnections

When the Liberty AsyncIO feature is in use, requests can be accepted from TCP/IP on nonexecutor threads and queued within the server waiting for an executor thread to become available. This can result in the queued requests consuming memory and threads in the servers JVM, especially in the event of a slow down in response from a SOR.

It is possible to limit the incoming work to the server to prevent this queuing of work waiting for executor threads by setting the maximum number of connections that are allowed to be open on the defined endpoints. This limit is implemented by specifying a value for the maxOpenConnections attribute on the tcpOptions subelement of the httpEndpoint element. When the maxOpenConnections limit is reached, no more connections are accepted, preventing the server from accepting any further requests.

Set maxOpenConnections to a value that should allow the server to accept the peak rate that it is expected to handle. By default, maxOpenConnections is set to 128000.

Java threads: Executor threads

Executor threads are Java threads that are created by Liberty and used to handle IBM z/OS Connect requests. The Liberty default executor is self-tuning and adapts to the current workload by dynamically adding or removing threads. For most workloads, the executor does not require any tuning, and you are advised not to change any settings of the executor unless you encounter specific problems with thread creation. For more information about tuning the default executor element, see Tuning Liberty in the WebSphere® Application Server for z/OS Liberty documentation.

You can see how many executor threads are active in the Java Health Center and the state of each. For example, WAITING, TIMED_WAITING, RUNNABLE, and so on. On the Threads tab, filter on Exec*. This filter uses case-sensitive values.

When all executor threads are busy, incoming requests are queued. The request will only timeout if an associated timeout value is exceeded. For example, asyncRequestTimout, or waiting for a socket, session, or connection.

If the server is configured with multiple httpEndpoints that can accept requests, and even if maxOpenConnections is being used to limit the requests that can be accepted for each port, it may be appropriate to consider setting maxThreads for the executor thread group if you have memory usage concerns. By default, maxThreads is unlimited so consider adding this element to your server.xml and setting a value by monitoring your incoming workload. To limit the number of executor threads add the following to the server.xml:

<executor maxThreads="300"/>

Calculating and monitoring the Java heap size

The Java heap is a runtime data area from which memory is allocated for all class instances and arrays. IBM z/OS Connect loads all the objects it needs into the Java heap, including features, APIs, services, API requesters, policies, and security certificates. The Java heap is managed in 64-bit storage by the JVM and uses some of the storage that is allocated by the MEMLIMIT setting.

Every workload that runs through an IBM z/OS Connect server is different. It is very important to understand the peaks and troughs of your workload over time, and continually monitor the Java heap to ensure optimum performance.

When an IBM z/OS Connect server starts up, it loads all the required objects into the Java heap. When the server is running normally, further Java objects are required to manage the incoming requests, connections, and session pools. The Java heap memory is also used for parsing and transforming API requests and responses. This heap is application-specific and the amount that is required for transformation depends on the size and complexity of the payload. This payload might include nested arrays, long field names, and so on. For more information, see How payload size and transformation affect performance for the API provider.

As requests start and finish at various times, it can be difficult to establish how much heap is required at any one time. Adjustments to the Java heap size need to be made for inconsistent workloads where peaks and troughs might occur throughout the day. Further allowances need to be made for requests that take longer than anticipated due to delays caused by other products in the workflow. It is important not to overwhelm the IBM z/OS Connect server with work and to configure enough heap for these situations.

Therefore, it is important to constantly monitor your Java heap to ensure that there are enough resources to allow for such circumstances.

Maximum heap size (-Xmx)

The maximum heap size is set by the JVM property -Xmx. By default, the IBM z/OS Connect server has a maximum Java heap size of 512 M. However, this value might not be large enough for your expected workloads. Java properties cannot be changed for an active server, so unless you use the default -Xmx, you must set this before starting your server.

Calculating the amount of heap your server requires is not straightforward and might require running several iterations of your workload to obtain the optimum heap size.

As a rough guide, when you set your maximum heap size you should allow a minimum of the following values:

• 60M for a IBM z/OS Connect server to start.
• An extra 8 M for every 100 archive files that are to be loaded, either API, service, or API requester. For example, if you have 200 APIs and 200 services, add 32M. Do not waste heap space by installing archive files that are no longer required, such as old versions.
• For a single connection to CICS over IPIC, allow 300K per session.
  • The number of sessions on an IPIC connection between IBM z/OS Connect and a CICS region is negotiated during the establishment of the connection. The lower of the two values, sendSessions on the zosconnect_cicsIpicConnection definition in server.xml and Receivecount on the IPCONN CICS definition (if configured), is used. The default value is 100, taken from the default sendSessions value. For more information about configuring your IPIC connection, see IP interconnectivity (IPIC) overview.
  • When the connection is established, heap storage is allocated for all the sessions. For example, if the negotiated number of sessions is 100, then 100 sessions would require 30 M (100 x 300 K per session).
  • Use CICS statistics to monitor the usage of each IPIC connection while running a workload and adjust the value of sendSessions to avoid wasting Java heap.
• For each connection to IMS, allow 1.6 K .
  • Each IMS service request obtains a connection from the resource adapter connection pool. The default value for maxPoolSize on the connection factory element is 50. This value is the maximum number of physical connections for a pool and is usually sufficient for most installations. Therefore, if a workload requires 50 simultaneous requests to IMS, then 80 K of heap storage would be required at that point in time, and also while the connections are held in the connection pool. For more information, see Configuration parameters.
  • To aid performance, unused connections remain in the pool ready to be reused, and are only removed when agedTimeout expires. The agedTimeout attribute is set on the connectionManager element for a connection factory. The default is -1, which means that there is no timeout and the connections remain in the pool.
• Allow room for your Java heap to grow. You can calculate how much extra heap is required only by monitoring your heap over a long period (hours or days) as this very much depends on the type and speed of your workload.

As a suggestion, start with the default of 512 M and then use a Java monitor such as Omegamon for JVM or the Java Health Center to see how much heap is being used. The Working example describes a suggested approach.

If the maximum heap size set for your IBM z/OS Connect server is too small, the JVM might be unable to reclaim enough storage during a garbage collection cycle. This causes the JVM and IBM z/OS Connect to terminate, typically with one of the following errors:

• java/lang/OutOfMemoryError
  • errno 112: suggests a thread limit was reached.
  • errno 132: suggests MEMLIMIT or maximum heap size was reached.
• Java/lang/StackOverflowError
• An abend, SIGSEGV, or General Protection Fault

However, if the maximum heap size is set too large, your IBM z/OS Connect might exhibit inconsistent performance. A large heap can increase the length of time that is required for the GC to pause the processing of requests while it manages the objects in memory by freeing up unused memory objects and compacting areas of the heap to reduce wasted space. For workloads that perform consistently with few peaks and troughs, it is easier to tune your Java heap, but for erratic workloads it is safer to specify enough memory to cater for the workload peaks and possibly compromise performance.

Initial heap size (-Xms)

In addition to setting the maximum heap size (-Xmx), you can optionally set the initial heap size (-Xms). For workloads that run through IBM z/OS Connect, the main priority is to avoid long GC pauses. Therefore, the default setting of 8 M is used to allow the GC to run frequently and efficiently with minimal GC pauses.

Setting the initial and maximum heap size to the same value is typically not a good idea for IBM z/OS Connect workloads because garbage collection is delayed until the heap is full and then a long GC pause occurs, which might impact response times. This configuration also uses a large amount of RAM, which leaves less RAM available for other applications.

GC policy

Java offers various GC policies that can be set for your IBM z/OS Connect server. For most users, the default policy gencon, tends to provide the best and consistent performance for IBM z/OS Connect.

When you use the gencon GC policy, the Java heap uses new and old (nursery and tenure) areas within the Java heap to provide smoother GCs. The garbage collection algorithms are very good at automatically tuning the sizes of different areas in the Java heap, so typically you do not need to set Xmns, Xmnx, Xmos, or Xmox. In fact, poor setting of these parameters can have a detrimental impact to your server's performance. For most users, the following default settings of the gencon GC policy are sufficient.

• -Xmns - initial size of new (nursery) area is 25% of -Xms
• -Xmnx - maximum size of new (nursery) area is 25% of -Xmx
• -Xmos - initial size of old (tenure) area is 75% of -Xms
• -Xmox - maximum size of old (tenure) area is approximately -Xmx minus -Xmns

For more information about these parameters, see the Java Documentation.

Monitoring the Java heap

To observe the Java heap values that are used by your IBM z/OS Connect server at initialization, specify the -verbose:sizes JVM option for your server. For more information, see Specifying JVM options. The STDERR for your server shows the heap values, as shown in the following example where -Xmx was set to 512 M, and -Xms set to 64 M.

 -Xmns16M              initial new space size
 -Xmnx128M             maximum new space size
 -Xms64M               initial memory size
 -Xmos48M              initial old space size
 -Xmox496M             maximum old space size
 -Xmx512M              memory maximum

The Java heap that is used by your IBM z/OS Connect server can be monitored by using tools such as IBM Omegamon for JVM or the Java Health Centre. The latter is one of the IBM Monitoring and Diagnostic Tools.

The Garbage Collector (GC) adapts the heap size to keep occupancy between 40% and 70% for the following reasons:

• A heap occupancy greater than 70% causes more frequent GC cycles, which can reduce performance.
• A heap occupancy of less than 40% means infrequent GC cycles. However, these cycles are longer than necessary, causing longer pause times, which can reduce performance.

Therefore, to optimize application performance and keep within the 40-70% range, the maximum heap size setting should be at least 43% larger than the maximum occupancy of the IBM z/OS Connect server. For example, if your IBM z/OS Connect server has a maximum occupancy of 600 M, the maximum heap size is calculated as follows:

600 * 1.43 = 858 M

If the heap is fully expanded and the occupancy level is greater than 70%, increase the -Xmx value so that the heap is not more than 70% occupied. Accordingly, you might then need to increase the MEMLIMIT size.

For the best performance, the maximum heap size should, if possible, be contained in your machine's physical memory to avoid paging. The physical memory of your machine can be seen at the top of the verbose GC log. For example,

<system> <attribute name="physicalMemory" value="33617920000" />

This value is in bytes, so the RAM for the LPAR in this example is approximately 32G. The heap for your IBM z/OS Connect servers share the LPAR's RAM with other applications. Therefore, if you notice on the SDSF DA panels your IBM z/OS Connect servers paging, allocate more RAM to the LPAR or reduce the number of processes that compete for the RAM in the LPAR.

The following screen capture is taken from the Java Health Center while a IBM z/OS Connect ran a consistent workload. You can see how the GCs fall within the ideal 40-70% range of the heap size currently used. When the new (nursery) area of the heap is filled up, the JVM pauses to clear the old (tenure) area, and moves the objects from the new area to the old area. During this JVM pause, no other work is processed. Therefore, the JVM pauses should be kept to a minimum.

In the following Summary screen capture for the GC, also taken from the Java Health Center, the two key metrics are:

1. The "Proportion of time spent in garbage collection pauses (%)". A value of 2% or less indicates good performance.
2. The "Proportion of time spent unpaused (%)". A value of 98% or more indicates good performance.

The other values that are shown depend on the type of workload that is run. For instance, whether it is a heavy workload, or a light workload. Also, mixed workloads that require new Java objects to be created in the heap can lead to more frequent GCs. This is not an issue when the goals for the two key metrics that are mentioned before are maintained.

Pause-less garbage collection

The Java option, concurrentScavenge, is supported by all generations of IBM Z hardware to enable pause-less GC with two modes of operation: hardware-based and software-based operations. IBM z13™ and earlier hardware operates in software-based pause-less GC mode; IBM z14™ and later hardware (with supported software) operates in hardware-based mode. It is only supported when using the gencon GC policy and is enabled by specifying -Xgc:concurrentScavenge as a Java option.

Despite the name of this feature, GCs are not completely free of pauses, but pause-times might be shorter for some workloads that throttle peak throughput to meet response time Service Level Agreements (SLAs). For more information, see concurrentScavenge in the IBM SDK, Java Technology Edition 8 documentation.

Specifying optional environment variables

Specifying JVM options

JVM_OPTIONS=-Xoptionsfile=<WLP_USER_DIR>/servers/<name>.options

Calculating the MEMLIMIT

MEMLIMIT is a fixed memory size that you can set in the JCL for IBM z/OS Connect. It controls the amount of virtual storage above-the-bar that IBM z/OS Connect can access in its address space. However, it is a soft cap and not pre-allocated as the storage is only used if required. Sample JCL supplied with IBM z/OS Connect sets this value to 8G, which should be ample for most users. However, if the maximum Java heap is set to 5.5G or more, then MEMLIMIT should be increased to be 50% larger than the Java heap. For example, if maximum Java heap size is 6G, then MEMLIMIT = 6G + (50% of 6G) which is 9G. The extra virtual storage is to allow for native thread stacks, the JIT data cache, and other JVM requirements.

If you choose to set the maximum number of executor threads (see #heap_memlimit_region_threads__java_executor_threads) you can alternatively calculate MEMLIMIT use the following formula

To check the MEMLIMIT value that is configured for your IBM z/OS Connect server, you can either issue the following MVS command from SDSF /D OMVS,LIMITS,PID=<server pid> and observe the MAXMEMLIMIT value in the PROCESS LIMIT column, or look in the messages.log file for message CWWKB0126I. The value for MEMLIMIT in the messages.log file is shown as a hexadecimal value. For example, CWWKB0126I: MEMLIMIT=2000, which equates to 8G.

From SDSF, you can issue the MVS command /D OMVS,LIMITS,PID=<server pid> to see how much virtual storage is currently being used by your IBM z/OS Connect server by observing the CURRENT USAGE and HIGHWATER USAGE values. For example, in Figure 7, the current usage is 1870 M. The amount of virtual storage peaked at 1990M, well within the 8192M configured.

OMVS    0011 ACTIVE          OMVS=(22,DB,JV,00,US)                
USER          JOBNAME  ASID        PID           PPID STATE     START     CT_SECS
ZOSPERF       ZOSCONN  00B4        16843242      66037 HK------ 11.12.37    85.8
 LATCHWAITPID=         0 CMD=/java/J64/bin/java -javaagent:/u/zosperf
PROCESS LIMITS:       LIMMSG=NONE                                    
                    CURRENT  HIGHWATER    PROCESS                       
                      USAGE      USAGE      LIMIT                       
...                                                                                                                                        
...                                     
MAXMEMLIMIT          1870M       1990M      8192M         <---- current and highwater usage of MEMLIMIT   

Setting the REGION size

You can set the REGION value in the JCL for IBM z/OS Connect. z/OS uses the region size to determine the amount of 31-bit storage available to running programs (that is, above-the-line and below-the-bar). Do not restrict the region size, but allow the Java runtime environment to use what is necessary. Restricting the region size might cause failures with storage-related error messages or abends such as 878-10.

Therefore, leave the default setting of REGION=0M, so the JVM uses what it needs.

Handling out-of-memory errors

The -XX:+CrashOnOutOfMemoryError option is not supported by the IBM JVM. When an OutOfMemoryError occurs in the IBM z/OS Connect JVM, the server continues running, but stops processing work.

Instead, use the Java option -XX:OnOutOfMemoryError="<command_string>", where <command_string> is a command or list of commands to run when a java.lang.OutOfMemoryError occurs. Multiple commands must be separated by a semicolon (;).

-XX:OnOutOfMemoryError='"/bin/date; /bin/echo MSG1234 Restart;kill -9 %p

Working example

Before a workload was started, the Java heap for the started server was observed to use approximately 80M-90M.

A workload was then started with 100 clients sending 100-byte JSON requests to the server, with each client receiving a 16 K response. After several hundred thousand requests had run to allow the JIT to warm up, the Java heap size was observed by using the Java Health Center. As the number of clients was gradually increased to 500 clients (500 being the maximum number of clients for this scenario), it was clear from the Java Health Center that the Java heap was outside the optimum 40-70% range at 34%. This indicated that the maximum heap size was too large for this scenario and might be reduced to the heap size observed, plus 43%. Not only does this reduction save on RAM but makes the GC more efficient and fewer pause times are observed.

The IBM z/OS Connect server was restarted and reconfigured with:

• Maximum heap size of 328 M (Xmx)
• Initial heap size of 64 M (Xms)
• MEMLIMIT = 8G
• REGION size = 0 M
• One IPIC connection with 100 sessions.

Running with 500 clients the Java heap was observed to be running more efficiently within the optimum 40-70% range at 50%.

The two key metrics, Proportion of time spent in garbage collection pauses (%) and Proportion of time spent unpaused (%) were checked in the Summary pane of the Java Health Center and indicated good performance.

The output from the MVS command /D OMVS,LIMITS,PID=<server pid> showed that the number of native threads that are used for the workload was acceptable and did not increase dramatically as the workload increased. The virtual storage required, 1990 M, which was well within the MEMLIMIT of 8G.

OMVS          0011 ACTIVE             OMVS=(22,DB,JV,00,US)                
USER          JOBNAME  ASID        PID          PPID STATE     START     CT_SECS
ZOSPERF       ZOSCONN  00B4       16843242      66037 HK------ 11.12.37    85.8
 LATCHWAITPID=         0 CMD=/java/J64/bin/java -javaagent:/u/zosperf
PROCESS LIMITS:        LIMMSG=NONE                                    
                     CURRENT  HIGHWATER    PROCESS                       
                       USAGE      USAGE      LIMIT                       
...                                                                       
...                  
MAXTHREADS               78          83      15000         <----  MaxThreads must be >= MaxThreadTasks               
MAXTHREADTASKS           79          83       6000                       
...                                                                       
...                                     
MAXMEMLIMIT           1270M       1289M      8192M         <---- current and highwater usage of MEMLIMIT   

In addition to monitoring the Java heap and thread usage, check the CPU usage for any contention through collecting and viewing SMF records. For more information, see GCPs and zIIPs.