Troubleshooting
Problem
I have already enlarged PLC maximum heap size, but sometimes I am still seeing a OOM error in one of the PLC loader logs. How can I verify if it’s still a heap sizing issue or a true memory leak?
Symptom
You see from one of PLC loader log a OutOfMemoryError thrown or a failure message on thread creation, For example:
ERROR com.platform.perf.dataloader.DataLoaderJob.execute(DataLoaderJob.java:132)
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
Or
You see a javacore.<timestamp>.txt generated (javacore.txt file only generated by IBM JDK)
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
Diagnosing The Problem
First if you see OOM with "Failed to create a thread" error from javacore.txt or PLC loader log. This means a native out of memory issue has been encountered. For most cases, it is caused by an insufficient nproc setting, because on Linux nproc value counts the number of threads for all processes within a user.
In some other cases, the default value for nproc will be set to 1024, which may not allow to create enough threads in all processes；therefore, you have to check the ulimit setting on Linux.
The PID of PLC (symphony processes) need to be confirmed. After getting the PID, check the /proc/<pid>/limits file or confirm ulimit setting from javacore.txt.
The contents of /proc/<pid>/limits file is similar to the "ulimit -a" output.
For more how to troubleshoot native memory issues, please refer to Troubleshooting native memory issues.
Resolving The Problem
IBM Platform Symphony recommends setting the nproc to a value of 30720 when Symphony running on Linux. For more information, please refer to: System and cluster parameter tuning for support large scale Spectrum Symphony cluster.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
isg3T1026451