Troubleshooting
Problem
Your server crashes causing applications to stop responding.
Symptom
The file from the crash, hs_err_pid, shows:
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xffffffff7e701a80, pid=22523, tid=106
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode solaris-sparc compressed oops)
# Problematic frame:
# C [libc_psr.so.1+0x1a80] __align_cpy_1+0xf64
#
...
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
....
...
Stack: [0xfffffffee2d00000,0xfffffffee2e00000], sp=0xfffffffee2df9050,
free space=996k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
C [libc_psr.so.1+0x1a80] __align_cpy_1+0xf64
C [libzip.so+0x2920] ZIP_GetEntry+0xdc
C [libzip.so+0x2ddc] Java_java_util_zip_ZipFile_getEntry+0xc8
J java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
J
java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;
j
com.ibm.ws.classloader.SinglePathClassProvider$2.run()Ljava/lang/Object;+56
J
com.ibm.ws.classloader.CompoundClassLoader.localFindClass(Ljava/lang/String;)Ljava/lang/Class;
j
com.ibm.ws.classloader.CompoundClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+217
....
Cause
JVM crash.
Environment
Running in HotSpot JVM.
Diagnosing The Problem
Resolving The Problem
Most likely causes for this kind of crash:
- Insufficient swap space on the machine.
- a JAR file that was modified or overwritten while the JVM was running, resulting in a corrupted jar file.
To ascertain which is the case:
- Check/monitor the output of command vmstat -s and check for swap space before crash occurence.
- Look for any any activity in jar files prior to the crash that may have resulted in a JAR file being modified underneath the JVM.
The second scenario is the most common. The reason why JAR files cannot be modified underneath the JVM is because the JVM caches each JAR file's central directory structure (using mmap) for performance reasons. The central directory is the part of the JAR file that describes its contents - i.e. entry locations, sizes and so on. Caching it in memory means that the JVM does not have to read it from the disk every time an entry in that JAR file needs to be accessed.
If the JAR file is modified after this region has been cached, then the JVM's version of the central directory will no longer be an accurate description of the JAR file as it exists on disk and the end result is a SIGBUS or SIGSEGV when the JVM attempts to load an entry from the modified JAR file
(usually during class loading).
Note: This has been a well known limitation of the HotSpot JVM for a long time but Oracle did recently introduce an option to work around the issue. From 1.6.0_23 onwards, the following option can be added to the command line to completely disable memory mapping for ZIP/JAR files:
-Dsun.zip.disableMemoryMapping=true
Enabling this option will impact performance because the VM will have to conduct some additional disk IO every time it reads an entry from a JAR file (to interrogate the central directory structure).
As a result, you should verify whether any jar files were modified some time prior to the crashes starting, while the JVM was still running.
Related Information
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21625858