IBM Support

WebSphere Native OutOfMemory - Linux ONLY

Troubleshooting


Problem

Unable to start a WebSphere Java process:
JVMJ9VM015W Initialization error for library j9gc29(5): Failed to allocate default memory space
Encountering native OutOfMemory errors on a running WebSphere Java process:
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted"

Symptom

WebSphere Native OutOfMemory

Cause

Incorrect memory configuration on Linux OS can cause native OOM errors for a WebSphere Java process.

Diagnosing The Problem

Check the output of "sysctl -a" (Example: sysctl -a > out_sysctl). It will provide values for:
  • vm.overcommit_ratio: 50 (default) 
  • vm.overcommit_memory: 2 
If the value of vm.overcommit_memory is 2 then proceed further (any other value this technote does not apply)
Check the output of "/proc/meminfo" (Example: /proc/meminfo > out_meminfo). It will provide values for:
  • CommitLimit: xxx KB
  • Committed_AS: yyy KB
What is:
  1. CommitLimit: This is the total amount of memory currently available to be allocated on the system. It is based on the overcommit ratio (vm.overcommit_ratio). This limit is only adhered to if strict overcommit accounting is enabled (mode 2 in 'vm.overcommit_memory')
  2. Committed_AS: The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes running on the system, even if it has not been "used" by them as of yet.
  3. vm.overcommit_ratio:  By default this value is set to 50, which means the system would allocate up to 50% of your memory and make it available for processes running on the system. For example if the memory on the system is 96G and vm.overcommit_ratio is set to 50 then the total MAXIMUM memory made available to ALL processes by the OS cannot exceed 48G. 
  4. vm.overcommit_memory: This value contains a flag that enables memory overcommitment. When this flag is 2, the kernel uses a "never overcommit" policy that attempts to prevent any overcommit of memory (default setting is zero).
If /proc/meminfo output shows value of CommitLimit close to the value of Committed_AS then most of the memory that can be allocated is used up. If future allocation request will push the value of Committed_AS above the value for CommitLimit then the request will fail with native OOM. 
It is important to consult with your system administrator to understand the memory requirements for your application topology and tune the vm.overcommit_ratio accordingly or increase the memory on the system if vm.overcommit_ratio can't be tuned.
 
The value of vm.overcommit_ratio can be changed with below command:
sudo sysctl -w vm.overcommit_ratio=<value>

Example:
Total memory is 96G (96000000 KB)
The output of "/proc/meminfo" shows:
CommitLimit:  48000000 kB
Committed_AS: 45000000 kB

The output of "sysctl -a" shows:
vm.overcommit_ratio: 50 (48G available for use)

If next allocation request exceeds 3G request then a native OOM will be thrown.

Resolving The Problem

To fix the problem:
Kindly work with the system administrator to understand the memory requirements for your application topology and  tune the overcommit_ratio accordingly or increase the memory on the system if this can't be tuned. 
For example, vm.overcommit_ratio can be tuned to 75 with the following: 

sudo sysctl -w vm.overcommit_ratio=75
that will allow 72G memory available (75% of 96G = 72G). 

Document Location

Worldwide

[{"Type":"SW","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdB6AAK","label":"OutOfMemory->Native->System resources (physical RAM)"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
09 June 2021

UID

ibm16440559

Page Feedback