Technical Blog Post
Abstract
10 things every WebSphere Service Integration Bus user must be aware of
Body
A service integration bus is a group of one or more application servers or server clusters in a WebSphere Application Server cell that cooperate to provide asynchronous messaging services. The application servers or server clusters in a bus are known as bus members. In the simplest case, a service integration bus consists of a single bus member, which is one application server. While administering SIBus, you may have come across some problems and may have had questions related to these problems. In this BLOG entry, I have put together some of the most frequently asked questions related to SIBUS.
1. Why is my messaging engine holding an exclusive lock on the SIBOWNER table indefinitely?
The exclusive lock on the SIBOWNER table is by design and remains for the lifetime of the messaging engine. During startup, each messaging engine establishes an exclusive lock on its data store. While the messaging engine is running, it maintains that lock to ensure the integrity of the data within the data store.
Note: In WAS v8.5 a new feature called Restrict long running locks was introduced which when enabled causes the messaging engine to hold lock on the SIBOWNER table for a short duration.
2. Why is my JVM terminating abnormally with the following messages in the SystemOut.log?
| CWSIS1546I: The messaging engine, ME_UUID=<ME_UUID>, INC_UUID=<INC_UUID>, has lost an existing lock or failed to gain an initial lock on the data store.
|
From WebSphere v7.0 onward, the High Availability Manager stops the messaging engine and its hosting application server when a messaging engines looses its connection to the datastore. This behavior is due to the property sib.msgstore.jdbcFailoverOnDBConnectionLoss which is set to true by default.
Note: In WAS 8.5.5 you may not see this behavior and the JVM will not terminate when the messaging engine data store connection is lost. This is because the two properties sib.msgstore.jdbcFailoverOnDBConnectionLoss and sib.meEnableInstanceOnFailure are true by default and when both the properties are enabled ,sib.meEnableInstanceOnFailure will take precedence over sib.msgstore.jdbcFailoverOnDBConnectionLoss preventing the termination of JVM. This problem was fixed in apar PM92447 .
3. Why does my messaging engine take a long time to startup?
The startup of a messaging engine depends on two factors:
a.The number of destinations configured
b.The number of messages on the destinations
If there are a large number of destinations configured or if there are too many messages on the destinations before shutting down the messaging engine, then during the subsequent startup, the messaging engine has to load all the destinations and the messages from the messagestore into the JVM memory. Hence there will be a delay in the startup of the messaging engine. If there are millions of messages on the destination, then the messaging engine may take several hours to get started. The best practice is to have all the messages on the destination consumed by the consuming application(JMS/MDB) and then restart the JVM, if a JVM restart is required.
4. What is remote get and how can I avoid this in my environment?
When a JMS application or an MDB is connected to a messaging engine that does not host the queue point, the messaging engine will attempt to create an internal stream to the remote messaging engine that hosts the queue point. This is known as Remote Get and causes network latency because the local messaging engine has to communicate with the remote messaging engine hosting the destination for consumption of every message. This causes a lot of messaging engine to messaging engine communication and can lead to high CPU issues in the environment.
You can avoid Remote Gets by setting the Target properties on the JMS Connection Factory/Activation Specification depending on what is configured in your environment for message consumption.
The 3 Target properties are:
Target
Target Significance
Target Type
To avoid Remote Gets, set the properties as below:
Target:name of the remote messaging engine that hosts the destination
Target Type:Messaging engine name
Target Significance:Required
Collect javacores from the jvm that is not hosting the queue point. If there are no occurrences of "AnycastOutputHandler" or "AnycastInputHandler" in the javacores then that confirms there are no Remote Gets happening in the environment.
5. What are SIB001 and SIB002 tables and why do I sometimes see a lot of data written to these tables?
The default settings for the messaging engine datastore are:
Number of tables for permanent objects :1
Number of tables for temporary objects :1
(click image to enlarge)
SIB uses Permanent tables to store persistent objects and Temporary tables to store non-persistent objects. For Example, If the number of these tables is set to 2 there would be 4 different SIB* tables.
Permanent table (storing persistent objects)
SIB001
SIB002
Temporary table (storing non persistent objects)
SIB003
SIB004
Messages are stored in the appropriate tables depending on the Quality of Service. When using Reliable Persistent messages, the following flow is expected:
- Message sent
- Message committed
- A separate thread storing the data to datastore
If a request to consume the message is made before the message is written to the persistent store, the message would be given to the consumer and not written to the permanent table. If the messages are consumed as soon as they are received we would expect the message to be not written to datastore. In case of persistent messages if there is a consumer already waiting for the message, the message would be delivered without writing into the datastore else the message would be written to the datastore. In case of non-persistable messages, the messages are written into the database based on the amount of memory that is occupied by these messages on the heap.
To reduce the data flowing into the datastore, the consumers of these messages have to be tuned to consume messages as soon as they arrive.
Please refer to the following links for details on the SIB tables.
Increasing the number of data store tables to relieve concurrency bottleneck
How are the JMS messages distributed across the SIBnnn tables in WebSphere Application server
6. Can I configure my messaging engine filestore on a local drive?
If the Bus member is a server, then the filestore can be created locally on the same node where the messaging engine is running but if the Bus member is a cluster then the filestore MUST be configured on a shared file system(like NFs v4) so all the cluster members can access the filestore.
Note: You should never ever configure multiple copies of filestore for one single messaging engine. For Example, if there are 2 nodes in the cluster and you have configured high availability policy for the messaging engine, you cannot configure one filestore on node1 and one filestore on node2. At any point of time, there must be only one filestore for each messaging engine in the environment and that must be on a shared file system that supports failover. To test if your file system supports failover , you can run the IBM File System Locking Protocol Test for WebSphere Application Server
7. Does Service Integration Bus rely on HAManager?
High availability manager, commonly called HAManager, enhances the availability of singleton services in WebSphere as well as provide group service and group messaging capabilities to WebSphere internal components.
These singleton services include:
- Transaction service (Transaction log recovery)
- Messaging service (Messaging engine restarting)
The HAManager runs as a service within each application server process(Deployment Manager, Node Agents, or application servers) and monitors the health of WebSphere clusters. In the event of a server failure, the HAManager will failover any singleton services that were running on the server that just failed. Examples of such services would include the recovery of any in-flight transactions and/or restarting any messaging engines that were running there. For a successful failover of the messaging engine in a cluster, the High Availability Manager must be enabled on all the cluster members.
8. Can I run database maintenance tools like "RUNSTAT" or "REORG" while my messaging engine is running?
It is not recommended to run any Database maintenance tools or take backups while the messaging engine is running. The recommended approach is to either shut down the messaging engine and then perform the maintenance/backups OR exclude the SIB* tables from RUNSTAT/backup.
Note: Before shutting down the messaging engine, make sure all the messages on the destinations are consumed by the application and the transactions are committed successfully.
9. Are there any SIB Tools available?
Yes, SIB tools are available. Please refer to the following links for details:
Service Integration Bus Destination Handler Tool
Service Integration Bus Performance Tool
10. Are there any jython/wsadmin scripts available for WebSphere SIBUS?
Yes, please refer to my previous BLOG entry Commonly used wsadmin scripts for WebSphere Application Server SIBus
UID
ibm11080495