How to Troubleshoot TM1 Server Lock Contention

Resolving The Problem

Step 1 - Confirm that you really are experiencing lock contention
-To confirm you are experiencing lock contention, you need to review the active threads on the TM1 Server.  
-This can be done using Planning Analytics Administration, TM1Top, Operations Console, or the TM1 REST API.  
-When working with TM1 Support, the ideal method is TM1Top or Planning Analytics Administration.

Here is an example of TM1Top/Thread output demonstrating a lock:

*In the above snippet the Wait:IXC confirms that you are experiencing lock contention (a thread that is waiting on another thread)
**The example above tells us that thread 34212 is waiting on thread 45260 to complete, before it can proceed
***If the last column (time elapsed) continues to count up, this confirms that the TM1 Server is not hung and that the threads are still active
 

Step 2 - Get a 'generic' understanding of the lock contention
-Review the TM1Top/Thread output captured to determine what kind of activities were being performed on the server.  
-Reach out to the users and ask them to explain what they were doing at the time the problem had started.  
-Using the example above, you would reach out to UserB (owner of thread 34212), as well as UserA (owner of thread 45260).  
-The actions that these two users have performed is key to understanding why the contention occurred.  
-From the thread output, we know that UserB's REST Request (from PM Hub/CAFE) is waiting on UserA's TI Process.  
     -What was the TI Process doing?  
     -What was UserB doing in PM Hub/CAFE?

When possible (if enough detail is available), attempt to recreate the behaviour to understand if the lock contention is consistent.  IBM Support may require a reproducible testcase in order to troubleshoot and/or provide a fix.
 

Step 3 - Additional Debugging to better understand lock contention
It may not always be very clear as to why locking has occurred.  Perhaps the users involved are not certain of the exact actions that were performed leading up to the contention, or perhaps there are a lot more locks involved than the simplified version in this example.

In order to troubleshoot a lock contention further, additional DEBUG logging is required.  
To enable the additional debugging, the following must be included in your tm1s-log.properties file:  log4j.logger.TM1.Lock.Exception=DEBUG
*This particular logger is a low cost logger, that is OK to leave enabled at all times (inlcuding in production)

With the above logger enabled, the following output can be extracted from the tm1server.log file:

34212   [9]   DEBUG   2017-08-08 19:18:05.310   TM1.Lock.Exception   Contention encountered attempting to acquire lock (0x000000000764E4C8) on object [Dimension "DimensionX", addr=0x000000000764E410, index=R-2452] in IX mode at ..\tm1_r7s\TM1DimensionImpl.cpp:8941 during function 'ExecuteRESTRequest'.
Entering wait state 'IXC'.
Blocked by the following 1 thread:
Thread 45260 holds the lock in READONLY mode

-This DEBUG output is more useful to Support/Development, however does help us confirm that the lock contention is on a Dimension Object (DimensionX).  
-The output tells us that thread 34212 (UserB's PM Hub/CAFE request) is attempting to acquire a Write Lock (IX) on DimensionX.  
-The lock can not be obtained because thread 45260 (UserA's TI Process) has a READONLY lock on DimensionX.  

At a high level, this tells us that thread 34214 needs Write access to the dimension - but is only allowed ReadOnly access, hence why the thread must wait for the lock to be released.  This would be expected behaviour.  Whether the behaviour in the scenario you are experiencing is expected or not, may warrant further investigation in to the design of the model.