Monitoring Synthetic PoP

After you install the Instana host agent in a Kubernetes cluster, the Synthetic PoP sensor is automatically installed to monitor Instana Synthetic PoP. You can view the Instana Synthetic PoP metrics in the Instana UI after you configure Synthetic PoP sensor as outlined in this topic.

Supported information

Supported platforms

Instana supports the stable version of Kubernetes. According to the Kubernetes version compatibility, two versions earlier than the stable version are considered as soft-deprecated. If the Kubernetes stable version is 1.26, the Kubernetes sensor is pinned to version 1.24. The minimum-supported version for the Kubernetes distribution is 1.10.

Supported Synthetic PoP versions

Synthetic PoP sensor supports metrics and configuration data for Synthetic PoP 1.0.9 and later.

Installing

The Instana Synthetic PoP sensor can work only with the Instana host agent installed on a Kubernetes cluster. By default, the Instana Synthetic PoP sensor is automatically installed if the Instana host agent is installed on the Kubernetes cluster.

By default, poll_rate is set to 60 seconds. To change poll_rate, update the Configuration file as shown in the following example:

# Synthetic PoP
com.instana.plugin.syntheticpop:
  enabled: true
  poll_rate: 60 # seconds, minimum 60 seconds by default

Enabling TLS on the Synthetic PoP sensor

If Synthetic PoP is deployed with TLS encryption, you must enable TLS encryption on the Synthetic PoP sensor to call the Synthetic health check service by HTTPS.

Configure the key or certificate files for the Synthetic PoP sensor. Use the same TLS key and certificate files that are used for deploying Synthetic PoP with TLS encryption.

To install the Instana host agent, complete the following steps:

  1. Find the TLS key and certificate files when you deploy Synthetic PoP with TLS encryption.
  2. Create a Kubernetes secret for all TLS key files and certificate files in the Instana agent namespace:
    kubectl create secret generic pop-cert-secret -n instana-agent-namespace \
    --type='kubernetes.io/tls' \
    --from-file=ca.crt=/path/to/ca.crt \
    --from-file=tls.crt=/path/to/tls.crt \
    --from-file=tls.key=/path/to/tls.key
    

You can use the Helm chart, the YAML file, or operator to install Synthetic PoP sensor with TLS encryption.

Installing the Synthetic PoP sensor with Helm chart

To install and configure the Synthetic PoP sensor by using the Helm chart, complete the following steps:

  1. Download and decompress the Instana agent Helm chart:
    helm repo add instana-agent-repo https://agents.instana.io/helm
    helm repo update
    helm pull instana-agent-repo/instana-agent
    tar xzvf instana-agent-*.tgz
    
  2. Add a property for all key and certificate files in both the volumeMounts and volumes sections in the instana-agent/templates/agent-daemonset.yaml file as shown in the following example:
    volumeMounts:
      - name: pop-cert
        mountPath: /opt/instana/agent/etc/pop
    
    volumes:
      - name: pop-cert
        secret:
          secretName: pop-cert-secret
    
  3. Set the caCertificate, clientCertificate, and clientKey fields as TLS encryption configuration. Create the agent_with_pop_sensor.yaml configuration file as shown in the following example:
    com.instana.plugin.syntheticpop:
      enabled: true
      poll_rate: 60 # seconds, minimum 60 seconds by default
      caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
      clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
      clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller
    
  4. Install Instana host agent with the Synthetic PoP sensor:
    helm install instana-agent \
    ./path/to/instana-agent \
    --namespace instana-agent \
    --set agent.key='<your_agent_key>' \
    --set agent.endpointHost='<your_host_agent_endpoint>' \
    --set agent.endpointPort=443 \
    --set cluster.name='<your_cluster_name>' \
    --set zone.name='<your_zone_name>' \
    --set-file agent.configuration_yaml=agent_with_pop_sensor.yaml
    

To verify the TLS configuration, enter the Instana host agent container, and check whether the ca.crt, tls.key, and tls.crt files exist in the /opt/instana/agent/etc/pop/ directory.

If the Instana host agent is installed but Synthetic PoP is deployed with TLS encryption, you need to upgrade your Instana host agent by using TLS configuration to monitor Synthetic PoP.

Upgrading the Instana host agent

To upgrade your Instana host agent, complete the following steps:

  1. Follow step 1 to 3 in Installing the Synthetic PoP sensor with Helm chart.
  2. Run helm upgrade:
helm pull --repo https://agents.instana.io/helm --untar instana-agent && kubectl apply -f instana-agent/crds && helm upgrade instana-agent \
./path/to/instana-agent \
--namespace instana-agent \
--set agent.key='<your_agent_key>' \
--set agent.endpointHost='<your_host_agent_endpoint>' \
--set agent.endpointPort=443 \
--set cluster.name='<your_cluster_name>' \
--set zone.name='<your_zone_name>' \
--set-file agent.configuration_yaml=agent_with_pop_sensor.yaml

The Instana Helm chart deploys the instana-agent-operator internally, which then deploys the agent resources. This deployment is based on the agent custom resource (CR), which is created based on the Helm values. The operator pattern requires a custom resource definition (CRD) in the cluster before defining any custom resources. Therefore, the Helm chart includes the required CRD, which is deployed during the initial installations before submitting other artifacts. Currently, Helm can handle only the initial installations of CRDs and not their upgrade, deletion, and uninstallation. Because of this limitation, CRD updates are automatically submitted to the cluster only during the initial installation, and not during the upgrades. For more information on this limitation, see Custom Resource Definitions.

Installing the Synthetic PoP sensor with the YAML file

To install and configure the Synthetic PoP sensor by using a yaml file within your Kubernetes cluster, see Installing the Host Agent on Kubernetes. Customize the deployment.yaml to add TLS certificates.

  1. In the Instana UI, go to Installing Instana Agents, select your platform, usually is Kubernetes - Yaml or OpenShift - Yaml.

  2. Enter a cluster name and download the deployment.yaml file.

  3. Add a property for all key and certificate files in both the volumeMounts and volumes sections in the deployment.yaml file as shown in the following example:

    volumeMounts:
      - name: pop-cert
        mountPath: /opt/instana/agent/etc/pop
    
    volumes:
      - name: pop-cert
        secret:
          secretName: pop-cert-secret
    
  4. Update the key or certificate file path in the com.instana.plugin.syntheticpop section of the deployment.yaml file as in the following content:

    com.instana.plugin.syntheticpop:
      enabled: true
      poll_rate: 60 # seconds, minimum 60 seconds by default
      caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
      clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
      clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller
    
  5. Install the Instana agent by applying the updated deployment file:

    kubectl apply -f deployment.yaml
    

Installing the Synthetic PoP sensor with Operator

To install and configure the Synthetic PoP sensor by using the Operator, see Installing the Host Agent on Kubernetes. You must customize the instana-agent.customresource.yaml and edit daemonset to add TLS certificates.

  1. Log on to the Instana UI.

  2. In the navigation menu, click More > Agents.

  3. Click Instana Agents, and select your platform. For example, Kubernetes operator or OpenShift operator.

  4. In the Cluster name field, enter a cluster name, and download the instana-agent.customresource.yaml file.

  5. Update the key or certificate file path in the configuration_yaml section of the instana-agent.customresource.yaml file as shown in the following example:

    configuration_yaml: |
      com.instana.plugin.syntheticpop:
        enabled: true
        poll_rate: 60 # seconds, minimum 60 seconds by default
        caCertificate: '/opt/instana/agent/etc/pop/ca.crt' # path to X.509 CA certificate file if HTTPS is enabled for PoP Controller
        clientCertificate: '/opt/instana/agent/etc/pop/tls.crt' # path to X.509 client certificate file if HTTPS is enabled for PoP Controller
        clientKey: '/opt/instana/agent/etc/pop/tls.key' # path to client private key file if HTTPS is enabled for PoP Controller
    
  6. Install the Instana agent by applying the updated customer resource file as shown in the following example:

    kubectl apply -f instana-agent.customresource.yaml
    
  7. Add pop-cert-secret to Instana agent DaemonSet by running the following command:

    kubectl patch daemonset.apps/instana-agent -n instana-agent --type='json' \
    -p='[{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/0","value":{"mountPath": "/opt/instana/agent/etc/pop","name": "pop-cert"}},{"op":"add","path":"/spec/template/spec/volumes/0","value":{"name": "pop-cert","secret":{"defaultMode": 420,"secretName": "pop-cert-secret"}}}]'
    

    When the Instana agent is reinstalled by using the Operator, the updates of daemonset.apps are lost. You must run this command again whenever you install the Instana host agent by using the Operator.

To verify the TLS configuration, enter the Instana host agent container, and check whether the ca.crt, tls.key, and tls.crt files exist in the /opt/instana/agent/etc/pop/ directory.

Viewing metrics

To view the Synthetic PoP metrics, complete the following steps:

  1. In the navigation menu, select Infrastructure.
  2. Click the Comparison Table tab, and select Synthetic PoP in the Table content dropdown list.
  3. Click the name of Synthetic PoP to open the dashboard.

The Synthetic PoP dashboard is displayed with all the collected performance metrics and configurations.

Configuration data

The configuration data that is collected from the Instana Synthetic PoP is listed in the following tables:

Instana Synthetic PoP

Instana Synthetic PoP Description
Location Name Identified name of the Synthetic PoP.
Location Display Name Displayed name of the Synthetic PoP.
Location Description Location details of the Synthetic PoP.
Version Version of the Synthetic PoP.
Started At Start time of the Synthetic PoP.
Playback Capabilities Supported Synthetic type of the Synthetic PoP.
Instana Synthetic Endpoint Backend endpoint of the Synthetic PoP.
Country Deployed country of the Synthetic PoP.
City Deployed city of the Synthetic PoP.
Redis TLS Status of Redis TLS (Enabled or Disabled).
Tenant Type Type of tenant (Single or Multi).

Instana Synthetic PoP tags

Name Description
Label The label of customized tags or properties for this PoP.
Value The value of customized tags or properties for this PoP.

Instana Synthetic PoP Kubernetes

Synthetic PoP is deployed in a Kubernetes cluster with a namespace. You can go to the Kubernetes namespace and cluster from the Synthetic PoP snapshot or the metric dashboard.

Kubernetes Description
Namespace The namespace of the Kubernetes cluster where the PoP is deployed.
Redis and Playback Engine Namespace The namespace of the common Redis and playback engines for multi-tenant PoP.
Cluster The cluster name of the Kubernetes cluster where the PoP is deployed.

Supported Synthetic types

Synthetic PoP supports the following Synthetic types:

  • API Simple: API Simple test (type: HTTPAction)
  • API Script: API Script test (type: HTTPScript)
  • Browser Test: Browser Script test (type: BrowserScript), Webpage Script test (type: WebpageScript), and Webpage Simple test (type: WebpageAction)
  • ISM (Internet Service Monitoring) Test: SSL Certificate Check (type: SSL Certificate)

Performance metrics

The performance metrics that are collected from Instana Synthetic PoP are listed in the following tables:

Synthetic PoP Metric Description
Status The status of the Synthetic PoP.
Scheduled/Completed The number of total scheduled and completed tests for enabled Synthetic types.
Active Test Count The number of running Synthetic tests for all Synthetic types.
API Simple Workload The workload for API Simple tests that is running in the HTTP playback engine.
API Script Workload The workload for API Script tests that is running in the Javascript playback engine.
Browser Test Workload The workload for Browser script, Webpage script, and Webpage simple tests that are running on the Browserscript playback engine.
ISM Workload The workload for SSL Certificate tests that are running on the ISM playback engine.
Queued Tasks The number of tasks in the task queue that are waiting to be executed for each type.
Result Queue Depth The number of all Synthetic test results in the result queue that are waiting to be sent to backend.

Built-in events

Built-in events trigger issues or incidents based on failing health signatures on entities.

For more information about built-in events for the Synthetic PoP sensor, see Built-in events reference.

Custom events

Custom events trigger issues or incidents based on the thresholds of an individual metric of any specific entity.

The "Synthetic pop status" built-in event can trigger an offline event if the status becomes Offline in some conditions, for example, communication error with Instana backend. However, after a PoP is uninstalled, the PoP sensor is deactivated, and no default events are triggered. This behavior can be considered as normal.

To trigger an event for the PoP uninstallation case, you can create a customized event. Use the Offline event detection system rule, and choose the entity.type:syntheticpop scope. Select a period in the Grace Period list to keep the event open.

Monitoring issues

TLS not configured

Monitoring issue type: tls_not_configured

This monitoring issue indicates that TLS encryption is enabled in Synthetic PoP but TLS key and certificate files are not configured in PoP sensor. To configure TLS key and certificate files, see Enabling TLS on the Synthetic PoP sensor.

Redis TLS not enabled

Monitoring issue type: redis_tls_not_enabled

This monitoring issue indicates that TLS encryption is not enabled in Synthetic PoP. To ensure secure communication, it is recommended to enable TLS in the Synthetic PoP production environment. For more information, see Synthetic PoP TLS encryption.

Call PoP health failed

Monitoring issue type: call_pop_health_api_failed

This monitoring issue indicates that the Synthetic PoP sensor cannot get the health data of Synthetic PoP. Check Instana agent logs to see more error messages. To resolve this issue, try the following steps:

  • Check whether the Instana agent is installed on a Kubernetes cluster by using Helm chart, YAML, or Operator
  • If Synthetic PoP is installed with TLS enabled, check whether TLS is configured on the Synthetic PoP sensor correctly, see Enabling TLS on the Synthetic PoP sensor.