Monitoring Synthetic PoP

After you install the Instana host agent in a Kubernetes cluster, the Synthetic PoP sensor is automatically installed to monitor Instana Synthetic PoP. You can view the Instana Synthetic PoP metrics in the Instana UI after you configure Synthetic PoP sensor as outlined in this topic.

Supported information
- Supported platforms
- Supported Synthetic PoP versions
Installing
- Enabling TLS on the Synthetic PoP sensor
Viewing metrics
Built-in events
Custom events
Monitoring issues

Supported information

Supported platforms

Instana supports the stable version of Kubernetes. According to the Kubernetes version compatibility, two versions earlier than the stable version are considered as soft-deprecated. If the Kubernetes stable version is 1.26, the Kubernetes sensor is pinned to version 1.24. The minimum-supported version for the Kubernetes distribution is 1.10.

Supported Synthetic PoP versions

Synthetic PoP sensor supports metrics and configuration data for Synthetic PoP 1.0.9 and later.

Installing

By default, the Instana Synthetic PoP sensor is automatically installed when you install the Instana host agent. To install the Instana host agent, see Installing the host agent on Kubernetes.

By default, poll_rate is set to 60 seconds. To change poll_rate, update the Configuration file as shown in the following example:

# Synthetic PoP
com.instana.plugin.syntheticpop:
  enabled: true
  poll_rate: 60 # seconds, minimum 60 seconds by default

Enabling TLS on the Synthetic PoP sensor

You can enable TLS encryption on the Synthetic PoP sensor. If TLS encryption is enabled, the Synthetic PoP sensor uses HTTPS to call the health check service of the Synthetic PoP.

Configure the key or certificate files for Instana Synthetic PoP sensor. Use the same TLS key files and certificate files that are used for deploying Synthetic PoP with TLS encryption.

To install the Instana host agent, complete the following steps:

Find the TLS key files when you deploy Synthetic PoP with TLS encryption.

Create a Kubernetes secret for all TLS key files and certificate files in the Instana agent namespace:

kubectl create secret generic pop-cert-secret -n instana-agent-namespace \
--type='kubernetes.io/tls' \
--from-file=ca.crt=/path/to/ca.crt \
--from-file=tls.crt=/path/to/tls.crt \
--from-file=tls.key=/path/to/tls.key

You can use helm chart or the YAML file to install Instana Synthetic PoP sensor with TLS encryption.

Installing the Synthetic PoP sensor with Helm chart

To install and configure the Synthetic PoP sensor by using the Helm chart, complete the following steps:

Download and decompress the Instana agent Helm chart:

helm repo add instana-agent-repo https://agents.instana.io/helm
helm repo update
helm pull instana-agent-repo/instana-agent
tar xzvf instana-agent-*.tgz

Add a property for all key and certificate files in both the volumeMounts and volumes sections in the instana-agent/templates/agent-daemonset.yaml file as shown in the following example:

volumeMounts:
  - name: pop-cert
    mountPath: /opt/instana/agent/etc/pop
volumes:
  - name: pop-cert
    secret:
      secretName: pop-cert-secret

Set the caCertificate, clientCertificate, and clientKey fields as TLS encryption configuration. Create the agent_with_pop_sensor.yaml configuration file as shown in the following example:

com.instana.plugin.syntheticpop:
  enabled: true
  poll_rate: 60 # seconds, minimum 60 seconds by default
  caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
  clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
  clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller

Install Instana host agent with the Synthetic PoP sensor:

helm install instana-agent \
./path/to/instana-agent \
--namespace instana-agent \
--set agent.key='<your_agent_key>' \
--set agent.endpointHost='<your_host_agent_endpoint>' \
--set agent.endpointPort=443 \
--set cluster.name='<your_cluster_name>' \
--set zone.name='<your_zone_name>' \
--set-file agent.configuration_yaml=agent_with_pop_sensor.yaml

To verify the TLS configuration, enter the Instana host agent container, and check whether the ca.crt, tls.key, and tls.crt files exist in the /opt/instana/agent/etc/pop/ directory.

If the Instana host agent is installed but Synthetic PoP is deployed with TLS encryption, you need to upgrade your Instana host agent by using TLS configuration to monitor Synthetic PoP.

Upgrading the Instana host agent

To upgrade your Instana host agent, complete the following steps:

Follow step 1 to 3 in Installing the Synthetic PoP sensor with Helm chart.
Run helm upgrade:

helm upgrade instana-agent \
./path/to/instana-agent \
--namespace instana-agent \
--set agent.key='<your_agent_key>' \
--set agent.endpointHost='<your_host_agent_endpoint>' \
--set agent.endpointPort=443 \
--set cluster.name='<your_cluster_name>' \
--set zone.name='<your_zone_name>' \
--set-file agent.configuration_yaml=agent_with_pop_sensor.yaml

Installing the Synthetic PoP sensor with the YAML file

To install and configure the Synthetic PoP sensor by using a yaml file within your Kubernetes cluster, see Installing the Host Agent on Kubernetes. Customize the deployment.yaml to add TLS certificates.

In the Instana UI, go to Installing Instana Agents, select your platform, usually is Kubernetes - Yaml or OpenShift - Yaml.
Enter a cluster name and download the deployment.yaml file.

Add a property for all key and certificate files in both the volumeMounts and volumes sections in the deployment.yaml file as shown in the following example:

volumeMounts:
  - name: pop-cert
    mountPath: /opt/instana/agent/etc/pop
volumes:
  - name: pop-cert
    secret:
      secretName: pop-cert-secret

Update the key or certificate file path in the com.instana.plugin.syntheticpop section of the deployment.yaml file as in the following content:

com.instana.plugin.syntheticpop:
  enabled: true
  poll_rate: 60 # seconds, minimum 60 seconds by default
  caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
  clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
  clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller

Install the Instana agent by applying the updated deployment file:
```
kubectl apply -f deployment.yaml
```

Installing the Synthetic PoP sensor with Operator

To install and configure the Synthetic PoP sensor by using the Operator, see Installing the Host Agent on Kubernetes. You must customize the instana-agent.customresource.yaml and edit daemonset to add TLS certificates.

Log on to the Instana UI.
In the navigation menu, click More > Agents.
Click Instana Agents, and select your platform. For example, Kubernetes operator or OpenShift operator.
In the Cluster name field, enter a cluster name, and download the instana-agent.customresource.yaml file.

Update the key or certificate file path in the configuration_yaml section of the instana-agent.customresource.yaml file as shown in the following example:

configuration_yaml: |
  com.instana.plugin.syntheticpop:
    enabled: true
    poll_rate: 60 # seconds, minimum 60 seconds by default
    caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
    clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
    clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller

Install the Instana agent by applying the updated customer resource file as shown in the following example:
```
kubectl apply -f instana-agent.customresource.yaml
```
Add key and certificate files in both the volumeMounts and volumes sections in daemonset.apps as shown in the following example:
```
kubectl edit daemonset.apps/instana-agent -n instana-agent
```
Find volumeMounts and volumes sections to add pop-cert
```
volumeMounts:
  - name: pop-cert
    mountPath: /opt/instana/agent/etc/pop
volumes:
  - name: pop-cert
    secret:
      secretName: pop-cert-secret
```
Updates of daemonset.apps are lost when the Instana agent operator is redeployed. You must edit daemonset.apps whenever you install the Instana agent by an operator.

To verify the TLS configuration, enter the Instana host agent container, and check whether the ca.crt, tls.key, and tls.crt files exist in the /opt/instana/agent/etc/pop/ directory.

Viewing metrics

To view the Synthetic PoP metrics, complete the following steps:

In the navigation menu, select Infrastructure.
Click the Comparison Table tab, and select Synthetic PoP in the Table content dropdown list.
Click the name of Synthetic PoP to open the dashboard.

The Synthetic PoP dashboard is displayed with all the collected performance metrics and configurations.

Configuration data

The configuration data that is collected from the Instana Synthetic PoP is listed in the following tables:

Instana Synthetic PoP

Instana Synthetic PoP	Description
Location Name	Identified name of the Synthetic PoP.
Location Display Name	Displayed name of the Synthetic PoP.
Location Description	Location details of the Synthetic PoP.
Version	Version of the Synthetic PoP.
Started At	Start time of the Synthetic PoP.
Playback Capabilities	Supported Synthetic type of the Synthetic PoP.
Instana Synthetic Endpoint	Backend endpoint of the Synthetic PoP.
Country	Deployed country of the Synthetic PoP.
City	Deployed city of the Synthetic PoP.
Redis TLS	Status of Redis TLS (enabled or disabled).

Instana Synthetic PoP tags

Name	Description
Label	The label of customized tags or properties for this PoP.
Value	The value of customized tags or properties for this PoP.

Instana Synthetic PoP Kubernetes

Synthetic PoP is deployed in a Kubernetes cluster with a namespace. You can go to the Kubernetes namespace and cluster from the Synthetic PoP snapshot or the metric dashboard.

Kubernetes	Description
Namespace	The namespace of the Kubernetes cluster where the PoP is deployed.
Cluster	The cluster name of the Kubernetes cluster where the PoP is deployed.

Supported Synthetic types

Synthetic PoP supports the following Synthetic types:

API Simple: API Simple test (type: HTTPAction)
API Script: API Script test (type: HTTPScript)
Browser Test: Browser Script test (type: BrowserScript), Webpage Script test (type: WebpageScript), and Webpage Simple test (type: WebpageAction)
ISM (Internet Service Monitoring) Test: SSL Certificate Check (type: SSL Certificate)

Performance metrics

The performance metrics that are collected from Instana Synthetic PoP are listed in the following tables:

Synthetic PoP Metric	Description
Status	The status of the Synthetic PoP.
Scheduled/Completed	The number of total scheduled and completed tests for enabled Synthetic types.
Active Test Count	The number of running Synthetic tests for all Synthetic types.
API Simple Workload	The workload for API Simple tests that is running in the HTTP playback engine.
API Script Workload	The workload for API Script tests that is running in the Javascript playback engine.
Browser Test Workload	The workload for Browser script, Webpage script, and Webpage simple tests that are running on the Browserscript playback engine.
ISM Workload	The workload for SSL Certificate tests that are running on the ISM playback engine.
Queued Tasks	The number of tasks in the task queue that are waiting to be executed for each type.
Result Queue Depth	The number of all Synthetic test results in the result queue that are waiting to be sent to backend.

Built-in events

Built-in events trigger issues or incidents based on failing health signatures on entities.

For more information about built-in events for the Synthetic PoP sensor, see Built-in events reference.

Custom events

Custom events trigger issues or incidents based on the thresholds of an individual metric of any specific entity.

The "Synthetic pop status" built-in event can trigger an offline event if the status becomes Offline in some conditions, for example, communication error with Instana backend. However, after a PoP is uninstalled, the PoP sensor is deactivated, and no default events are triggered. This behavior can be considered as normal.

To trigger an event for the PoP uninstallation case, you can create a customized event. Use the Offline event detection system rule, and choose the entity.type:syntheticpop scope. Select a period in the Grace Period list to keep the event open.

Monitoring issues

TLS not configured

Monitoring issue type: tls_not_configured

This monitoring issue indicates that TLS key and certificate files are not configured. To access the PoP health API through HTTPS, configure the TLS key and certificate files in the sensor configuration.

Redis TLS not enabled

Monitoring issue type: redis_tls_not_enabled

This monitoring issue indicates that Redis TLS is not enabled. To ensure secure communication, enable Redis TLS in the Synthetic PoP production environment. For more information, see Redis TLS.

Call PoP health failed

Monitoring issue type: call_pop_health_api_failed

This monitoring issue indicates that the Instana host agent cannot get the health data of Synthetic PoP. This issue is usually caused by the HTTPS communication error when TLS is enabled in Synthetic PoP. For more information, check Instana agent logs.