Enhancements to Content Engine asynchronous event queue processing in 5.2.0 FP2 (Updated for 5.5.7)

Problem
When asynchronous events are fired in Content Platform Engine, the Content Platform Engine server writes a record to the QueueItem table as a part of the same transaction that fired the event. An asynchronous background task later reads this queued event from the QueueItem table, and invokes the event handler to perform the event action associated with the subscription that fired the event.

Events in the QueueItem table can be processed by any Content Platform Engine server in the domain. On each Content Platform Engine server, there is a background Event Dispatcher thread that reads events from the QueueItem table and assigns them to worker threads to perform the work. The number of asynchronous event worker threads is configurable, and defaults to five per Content Platform Engine server. So the total number of worker threads is the number of server instances times the number of worker threads per server instance. In environments where many asynchronous events are firing, the number of worker threads might need to be increased in order to keep the asynchronous event queue from backing up.

On each Content Platform Engine server, when a dispatcher needs to retrieve a set of events to process, it must issue a query that retrieves any events from the QueueItem table that are ready to be processed, and that are not already locked by another dispatcher. After a dispatcher obtains a set of events to process, it must mark those events as locked to prevent other dispatchers from attempting to process the same events. In order to keep dispatchers on different Content Engine Platform servers from conflicting with each other, each dispatcher must lock the QueueItem table prior to querying for eligible rows. This process has some scalability limitations in environments where there are many Content Platform Engine servers and there are many events being fired, due to the locking process.

Solution
To address this problem, a different locking algorithm for QueueItem processing is implemented in the 5.2.0 FP2 release. This algorithm allows for higher rates of asynchronous event processing when there are many (that is, four or more) Content Platform Engine servers.

Implementation Details
By default, when a DB query incurs a locked row, the processing of the query blocks and waits for the lock to be released. Some DB servers can change this default behavior when instructed, and skip the locked rows without blocking, as if those locked rows do not meet the query condition. This behavior can be utilized to improve the scalability in the QueueItem dispatcher process.

In this new scenario, the QueueItem dispatcher does not lock the whole QueueItem table. It instead issues a query statement to retrieve eligible events and instructs the DB server to put a lock on each of the retrieved rows and skip rows that are locked, typically by other QueueItem dispatchers. This behavior enables multiple QueueItem dispatchers to work concurrently on the QueueItem table without the scalability limitation of the prior approach. This new mechanism is implemented in the 5.2.0 FP2 release for SQLServer and Oracle.

For SQLServer and Oracle, the QueueItem dispatching can fall back to the old mechanism with a JVM switch, "-Dcom.filenet.engine.queueitem.FetchDBConcurrent=false". If it is DB2 server, the old mechanism is always used.

Update

DB2 11.5.5.4 introduces support for the SkipLock feature. More information on the feature is available here: https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.admin.perf.doc/doc/c_improving_c_incomplete_results.html. The feature can be used with P8 5.5.7 and later. However, performance improvement in QueueItem processing might not be seen. To enable this feature with DB2, set the following JVM argument: "-Dcom.filenet.engine.queueitem.FetchDBConcurrent=true".