IBM APM 8.1 - Synthetic agent - Useful information - tips - Part2

Body

Environment:

IBM Cloud Application Performance Management Server, V8.1.4 IF7

IBM Cloud Application Performance Management advanced, Synthetic agent, V8.1.4 IF3

Selenium scripts (.html and .zip)

Problem:

You recorded your Selenium test scripts with Selenium IDE plug-in, uploaded them to the APM Server; and then, deployed them to a Synthetic (SN) agent; then, using APM UI Synthetic Script Manager, you notice that some transactions show:

'Latest status' field is gray with "?" value, meaning unknown.

If the observed phenomenon is that some transactions can be played back, and some cannot; and after SN synthetic agent is restarted, some others or different ones have been played back but some others do not have; then, it looks like the SN agent does not have enough capacity, or resources, to run more or even all the currently scheduled playbacks.

The best approach then, is to edit the latest ksn_trace*.log files (/opt/ibm/apm/agent/logs/ksn_trace0.log) and search for the keyword:

threadPoolExecute queue size

What is the queue size value mostly seen in this log file? If the queue size value is always high, that means that the scheduled playback tasks cannot be completed in time, and those playback tasks are accumulated.

It would explain that some Transactions appear in APM UI under 'Synthetic Transactions' widget with 'Latest status' = unknown.

Example:

[2018-12-10T12:27:23.120-0400] - FINEST - xxxxxxx- Thread-41 - com.ibm.csi.apm.synthetics.playback.Playback.startScriptRunnable() - threadPoolExecute queue size :51448 task count : 81022

Here, 51448 playback tasks heap up, so any new added script cannot get its turn to play back

Solution:

If you determine that the workload is too heavy for this agent, we can consider several approaches:
1) deploying another SN synthetic agent, and distributing some scripts into that new synthetic agent, to share the load between several agents.
2) increasing the playback interval (for example, 2 minutes to 5 minutes).

3) increasing the capacity (RAM, CPU) of the machine hosting the SN synthetic agent.

A first approach can consist simply in increasing the current playback interval of all transactions; then, restart the SN synthetic agent, and verify that everything is now OK. Check again if, in ksn_trace*.log log file, the queue size keeps increasing. If its value is larger than 1000 then, you need to consider increasing again the playback interval; or consider next step like adding more capacity to the SN synthetic agent host machine.