Troubleshooting
Problem
java.lang.OutOfMemoryError: unable to create new native thread
Symptom
In some of the instances we have seen an native OutOfMemoryError accompanied
by a large number of concurrent "RMI RenewClean" threads(was seen in the thread
dumps) when cache updates are happening with a very high frequency.
For
more information and bugs related to "RMI RenewClean" threads please refer the
following link,
http://bugs.su
n.com/bugdatabase/view_bug.do?bug_id=5053529
<Aug 11, 2009 5:13:40 PM PDT> <Error> <HTTP> <BEA-101017>
<[weblogic.servlet.internal.WebAppServletC
ception.
java.lang.OutOfMemoryError: unable to create new native
thread
at java.lang.Thread.start0(Native Method)
at
java.lang.Thread.start(Thread.java:574)
at
sun.rmi.transport.DGCClient$EndpointEntry.<init>(DGCClient.java:223)
at
sun.rmi.transport.DGCClient$EndpointEntry.lookup(DGCClient.java:193)
at
sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:111)
Truncated.
see log file for complete stacktrace
>
Exception in thread "RMI
RenewClean-[10.16.212.172:52809,com.sterlingcommerce.woodstock.util.frame.j
ndi.AddrClientFactory@6b32a7]" java.lang.OutOfMemoryError: unable to create
new native thread
at java.lang.Thread.start0(Native Method)
at
java.lang.Thread.start(Thread.java:574)
at
sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
at
sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
at
sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
at
sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at
sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
at
sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:54
0)
at java.lang.Thread.run(Thread.java:595)
Exception in thread
"RMI
RenewClean-[10.16.212.177:53061,com.sterlingcommerce.woodstock.util.frame.j
ndi.AddrClientFactory@cb136d]" java.lang.OutOfMemoryError: unable to create
new native thread
at java.lang.Thread.start0(Native Method)
at
java.lang.Thread.start(Thread.java:574)
at
sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
at
sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
at
sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
at
sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at
sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
at
sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:54
0)
Resolving The Problem
How is RMI RenewClean and SSCAP related?
Each JVM including app server
will have a JNDI server [with AddrClientFactory as the socket factory] with
registered RMI port(each of these LiveReference will have a RMIRenewClean
Thread). When performing lookup for JNDI context, system ends up creating a
socket .That being the case, system will end up creating one socket per
broadcast.
How can it cause Native OutOfTheMemory?
Even though
system issues a close on the context, the system/jvm will GC these references
only when the DGC interval is elapsed. This means that there will be lot of
live references(per cache broadcast) at a given point of time and also lots of
RMI RenewClean Threads.
Each of RMI RenewClean will occupy
memory(outside the heap memory) depending on the -Xss value. So when the number
of RMI RenewClean threads increases(which is proportional to the number of
cache broadcasts), there are chances that system can run out of resources
resulting in Native OutOfTheMemory error.
What is the resolution to this
issue?
As suggested in SSCAP Performance Management Guide, any entity
which can receive bulk/frequent updates should not be cached. For example,
Item table should be taken out of Cache if there is an Integration Service in
the implementation which downloads items every minute.
We should keep in
mind that more the cache updates more the RMI RenewClean threads which can
ultimately cause this error.
Please refer the section "Performance Feature - Reference Data Caching" in our Performance Mangement guide before enabling cache for any database entities.
Historical Number
NFX7906
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21554222