Technical Blog Post
Abstract
IBM APM 8.1 - Synthetic agent SN - Useful info - tips - Part 2 of 3
Body
Environment:
IBM Cloud Application Performance Management Server, V8.1.4 IF7
IBM Cloud Application Performance Management advanced, Synthetic agent, V8.1.4 IF3
Selenium scripts (.html and .zip)
Problem:
After you have recorded your Selenium test scripts (using Selenium IDE plug-in), uploaded them to the APM Server, and then deployed them to a Synthetic SN agent, using APM UI Synthetic Script Manager, you notice that some transactions show :
'Latest status' field = grey "?", meaning unknown.
If the observed phenomenon is that that some transactions can be played back, and some can not, and after SN agent is restarted, some others/different ones may have been played back but some others may not have, it looks like the SN agent does not have enough capacity or resources to run more or even all the currently scheduled playbacks.
The best approach then is to edit the latest ksn_trace*.log log files (e.g /opt/ibm/apm/agent/logs/ksn_trace0.log) and search for the key word :
threadPoolExecute queue size
What is the queue size value mostly seen in this log file?
If the queue size value is always high, that means that the scheduled playback tasks can not be completed in time, and those playback tasks will be accumulated.
This would explain that some Transactions appear in APM UI under 'Synthetic Transactions' widget with 'Latest status' = unknown
Example:
[2018-12-10T12:27:23.120-0400] - FINEST - xxxxxxx- Thread-41 - com.ibm.csi.apm.synthetics.playback.Playback.startScriptRunnable() - threadPoolExecute queue size :51448 task count : 81022
here above, 51448 playback tasks heap up, so any new added script can not get its turn to playback
Solution:
If you determine that the workload is too heavy for this agent, we can consider several approaches :
1) deploy another SN agent and distribute some scripts into that new synthetic agent, to share the load between several agents
2) increase the playback interval (for example from 2 minutes to 5 minutes)
3) increase the capacity (RAM, CPU) of the machine hosting the SN agent
A first approach can consist simply in increasing the current playback interval of all transactions, then restart the SN agent, and verify if everything is now OK. Check again in ksn_trace*.log log file if the queue size keeps increasing. If its value is larger than 1000, then you need to consider to increase again the playback interval, or consider next step like adding more capacity to the SN agent host machine.
UID
ibm11083615