Troubleshooting
Problem
When using InfoSphere Master Data Management (MDM) customers will see coredumps being created when an application server is being started or stopped in an environment with multiple application servers.
Symptom
There are several use cases that can lead to this behavior, all involve a WebSphere Application Server topology of more than one JVM in a clustered or non-clustered environment.
Some examples below:
- JVMs recycle by themselves no apparent cause
- JVM1 is up and running, JVM2 is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, runbatch.sh process is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, if a user attempts to stop JVM1 or JVM2, the JVM that is being stopped crashes
Cause
This issue has been identified by IBM WebSphere Support as a defect APAR PI64662. Please subscribe to APAR to monitor for a fix.
Diagnosing The Problem
The javacore and heapdumps will have two things in common:
1. The crash occurring in the ld-linux-x86-64.so.2 library (Red Hat Linux) which is called from dlsym() call in the JVM
2. A process tries to access the libMAD.so library (MDM) from an existing JVM causing it to crash
Contact IBM Support if you suspect you are experiencing this issue. Ensure the MustGather: Crash on Linux is followed to collect all required information and that the core has been processed through jextract.
Resolving The Problem
In order to resolve the issue you can use workaround #1 for immediate relief or plan to install the WebSphere Application Server iFix in #2.
1. Make the libMAD.so file immutable
Follow steps below:
i. Stop any JVM that has the core MDM application installed
ii. Locate the libMAD.so library and set attribute to IMMUTABLE
Example: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/expandedBundles/com.ibm.mdm.mds.jni.app_11.4.0.FP00IF000_20140924-0832/com.ibm.mdm.mds.jni_11.4.0.FP00IF000_20140924-0832.jar/linux64
Ensure to locate the library in expandedBundles directory.
iii. As root, run the command below:
chattr +i /some/path/libMAD.so
You may want to run the command lsattr libMAD.so to ensure the property is set. If it is the output will be:
[root@localhost linux64]# lsattr libMAD.so
----i--------e- libMAD.so
Note:
chattr (Change Attribute) is a command line Linux utility that is used to set/unset certain attributes to a file in Linux system to secure accidental deletion or modification of important files and folders.
The defect seems to be opening/closing the libMAD.so, setting this property will prevent such activity, which will prevent the crash.
iv. Restart JVMs and test
Note: If application of an MDM fix or a new CBA is deployed ensure the file immutable properties are removed. If the file is designed to not be updated your upgrade or new deployment will fail.
2. Apply WebSphere iFix
Apply the attached iFix to your environment. For details on how to apply WebSphere iFixes refer to Installing and uninstalling interim fixes and fix packs on distributed operating systems.
a. iFix for 8.5.5.0-8.5.5.9
8.5.5.0-WS-WASProd-IFPI64662.zipb. iFix for 9.0.0.0
9.0.0.0-WS-WASProd-IFPI64662.zip
Related Information
Product Synonym
PME;MDM;MDM AE;MDM SE;MDM Server;InfoSphere MDM Server;Master Data Management;Probabilistic Matching Engine;Master Data Management Standard Edition;Master Data Management Advanced Edition;InfoSphere Master Data Management Server;IBM InfoSphere Master Data Management Server
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21981455