Troubleshooting
Problem
WebSphere Application Server (WAS) Service Integration Bus (SIB) Messaging Engine (ME) fails to initialize its file store and therefore fails to start.
Symptom
Messaging Engine fails to start because it is not able to acquire a lock on it's file store files. The files are:
/log
/permanentStore
/temporaryStore
Their default path is:
${USER_INSTALL_ROOT}/filestores/com.ibm.ws.sib/<me_name>-<me_uuid>/
A typical example of the failure is:
2/22/14 0:34:46:513 IST] 000000b5 SibMessage I
[ConnectionsBus:Communities_Cluster.000-ConnectionsBus] CWSIS1581I: The file store is attempting to initalise its log file:
/usr/lc/cstore/messageStores/Communities_Cluster/log/Log
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:515)
at com.ibm.ws.objectManager.utils.Utils.getImpl(Utils.java:44)
at com.ibm.ws.objectManager.utils.FileLock.getFileLock(FileLock.java:41)
at com.ibm.ws.objectManager.ObjectManager.createObjectManagerState(ObjectManager.java:293)
at com.ibm.ws.objectManager.ObjectManager.initialise(ObjectManager.java:237 )
at com.ibm.ws.objectManager.ObjectManager.<init>(ObjectManager.java:197)
at com.ibm.ws.sib.msgstore.persistence.objectManager.PersistentMessageStore Impl.start(PersistentMessageStoreImpl.java:356)
at com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.start(MessageStoreImpl.java:1524)
at com.ibm.ws.sib.admin.impl.JsMessagingEngineImpl.start(JsMessagingEngineImpl.java:609)
at com.ibm.ws.sib.admin.impl.HAManagerMessagingEngineImpl.activate(HAManagerMessagingEngineImpl.java:1009)
at com.ibm.ws.sib.admin.impl.JsActivationThread.run(JsActivationThread.java:92)
Caused by: java.io.IOException: Invalid argument
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:914)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:973)
at com.ibm.ws.objectManager.utils.FileLockImpl.<init>(FileLockImpl.java:58)
Cause
You are using NFS V3 for the file system that hosts the file store files. This version of NFS is known to have file locking problems.
If a messaging engine is running normally but then becomes unexpectedly disconnected from it's file store files the lock on the files should be automatically released. However, this may not happen if you are using NFS V3. As a result, messaging (JMS) High Availability will fail when the messaging engine restarts because it will not be able to get a fresh lock on the files. This is because the old lock is still there. The system administrator will need to manually release the lock.
The solution to this problem is NFS V4. NFS V4 uses a lease based locking system that is specifically designed to address this kind of problem. If you are using NFS V3 it is strongly recommended that you upgrade to version 4 to avoid this problem.
This issue is discussed in the WAS Information center here: "File Store High Availability":
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-iseries&topic=cjm1460
This link includes a link to the "IBM File System Locking Protocol Test for WebSphere Application Server", which can be used to help you test your file system to ensure it can perform file locking properly and therefore support failover.
Environment
AIX, Solaris, Linux, Windows
Diagnosing The Problem
Check to see if there is a lock on the file store files when the messaging engine is NOT running. There should be NO LOCKS if the messaging engine is not running.
On Unix use the 'lsof' command to check for locks on files. On Windows the Microsoft Handle program can be used to check for file locks.
Resolving The Problem
Manually unlock the file if at all possible and upgrade to NSF 4.
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21671633