Deploying a self-hosted PoP

Installing a self-hosted Synthetic PoP

Use the helm install command to install a new Synthetic PoP, and pass the following values. You are suggested to install the PoP in a separate namespace.

• downloadKey: The download agent key for pulling image from containers.instana.io Docker registry.
• controller.location: The format is label; displayLabel; country; city; latitude; longitude; description. This label acts as the PoP identifier and can only contain letter, number, hyphen, and underscore.
• controller.instanaKey: The Instana agent key, which is used for connecting to Instana backend.
• controller.instanaSyntheticEndpoint: The ingress endpoint of Synthetic acceptor in Instana backend.
• controller.clusterName: Optionally. It needs to be set with the name of the Kubernetes cluster that is to be displayed in the Instana UI when Instana agent is installed to monitor this PoP.
• redis.password: The authentication password to redis server. Specify a password with length at least 10.
• redis.tls.enabled: Define if enabling Redis TLS or not (default false). If the value is true, the communication with Redis will be encrypted with TLS.
• redis.tls.secretName: If the value of the redis.tls.enabled field is true, you need to specify a secret name for Redis TLS key/cert files.

Sign in to Instana UI, click Synthetic Monitoring, and then click New Location and select Private. Copy the values of downloadKey, controller.instanaKey, and controller.instanaSyntheticEndpoint.

To configure the installation, specify the values in the command line by using the --set flag, or provide a yaml file with your values by using the -f flag.

In the following example, a Synthetic PoP with the location name MyPoP is deployed, and Redis uses password for authentication. You need to update the following parameters based on your requirements:

helm install synthetic-pop \
    --repo https://agents.instana.io/helm  \
    --namespace syn \
    --create-namespace \
    --set downloadKey="yourdownloadkey" \
    --set controller.location="MyPoP;My PoP;MyCounty;MyCity;0;0;This is a testing Synthetic Point of Presence" \
    --set controller.instanaKey="instanaAgentkey" \
    --set controller.instanaSyntheticEndpoint="https://synthetics-green-saas.instana.io" \
    --set redis.tls.enabled=false \
    --set redis.password="a1fc5d01bcbb" \
    synthetic-pop

You can also provide your values.yaml to deploy a PoP, below examples shows how to get the values.yaml template and then pass the modified yaml file to deploy a PoP.

helm repo add synthetic-pop-repo https://agents.instana.io/helm
helm repo update
helm pull synthetic-pop-repo/synthetic-pop
tar xzvf synthetic-pop-*.tgz
# Modify the synthetic-pop/values.yaml file
# Run helm install to pass the modified values.yaml
 helm install synthetic-pop \
    --repo https://agents.instana.io/helm  \
    --namespace syn \
    --create-namespace \
    -f /path/to/values.yaml \
    synthetic-pop

After PoP is installed, make sure that each pod is in Running and Ready status.

kubectl get pod -n syn

Sign in to Instana UI, click Synthetic _Monitoring > Locations, and then you can see your location is showed up in the location list. If not, go to the Troubleshooting section to check the log files.

Security enhancement

The following configurations are recommended for security enhancement in production environment.

openssl genrsa -out tls.key 4096
openssl req -x509 -new -nodes -sha256 -key tls.key -days 3650 -subj '/O=Instana/CN=Certificate Authority' -out ca.crt
openssl req -new -sha256 -key tls.key -subj '/O=Instana/CN=Server' | openssl x509 -req -sha256 -CA ca.crt -CAkey tls.key -CAserial ca.txt -CAcreateserial -days 3650 -out tls.crt

# Create a new TLS secret named pop-tls-secret with the given key/cert files
kubectl create secret generic pop-tls-secret -n syn \
--type='kubernetes.io/tls' \
--from-file=ca.crt=path/to/ca.crt \
--from-file=tls.crt=path/to/tls.crt \
--from-file=tls.key=path/to/tls.key

helm install synthetic-pop \
    --repo https://agents.instana.io/helm \
    --namespace syn \
    --create-namespace \
    --set downloadKey="yourdownloadkey" \
    --set controller.location="MyPoP;My PoP;MyCounty;MyCity;0;0;This is a testing Synthetic Point of Presence" \
    --set controller.instanaKey="instanaAgentkey" \
    --set controller.instanaSyntheticEndpoint="https://synthetics-green-saas.instana.io" \
    --set redis.tls.enabled=true \
    --set redis.tls.secretName="pop-tls-secret" \
    synthetic-pop

helm install synthetic-pop \
    --repo https://agents.instana.io/helm \
    --namespace syn \
   ...
    --set seccompDefault=true \
    synthetic-pop

# Use the following command to print apiserver endpoint
# For example, if the following command prints "Endpoints: 10.128.0.127:16443", you need to block the IP address 10.128.0.127 
kubectl describe service kubernetes

playbackEngineNetworkPolicy:
  # Change the value to true to enable the network policy
  enabled: true
  egress:
    - to:
        - ipBlock:
            # Allow all IP address v4
            cidr: 0.0.0.0/0
            except:
              # Block Kubernetes API server IP address
              - 10.128.0.127/32
              # Block cloud provider metadata service IP address
              - 169.254.169.254/32

# --set-json only be supported since Helm V3.11
helm install synthetic-pop \
    --repo https://agents.instana.io/helm \
    --namespace syn \
   ...
    --set playbackEngineNetworkPolicy.enabled=true \
    --set-json 'playbackEngineNetworkPolicy.egress=[{"to":[{"ipBlock":{"cidr":"0.0.0.0/0","except":["10.128.0.127/32","169.254.169.254/32"]}}]}]' \
    synthetic-pop

Migration scenario

To support the backend migration scenario, you can specify multiple Synthetic acceptor endpoints and Instana agent keys with semicolon. Instana agent keys and Synthetic acceptor endpoints are one-to-one correspondence. During the migration period, Synthetic PoP can send data to both old and new backend to avoid downtime.

See the following example:

helm install synthetic-pop \
    --repo https://agents.instana.io/helm \
    --namespace syn \
    --set controller.instanaKey="instanaAgentkey;anotherInstanaAgentKey" \
    --set controller.instanaSyntheticEndpoint="https://synthetics-green-saas.instana.io;https://synthetics-green-saas.instana.host" \
    ...

Playback engine enablement

You can disable some playback engines depending on your requirements. For example, you can disable the BrowserScript playback engine if you no longer require browser tests. Disable the playback engine by passing the following parameter to reduce the CPU and memory resources for your Synthetic PoP.

--set browserscript.enabled=false

Capabilities

The controller.capabilities property defines the playback capabilities for the Synthetic PoP, including the supported Synthetic test and web browser types.

CustomProperties

The controller.customProperties property defines the customized tags or properties for this PoP. The property is a list of key:value pairs, separated by a semicolon, such as key1=value1;key2=value2.

Use an API such as $env.key1 to access the custom property or tag in an API or browser script.

Proxy

You can configure a proxy server for Synthetic PoP to connect the Instana backend server. The following options are supported.

• proxy.popProxyProtocol: the protocol used by the proxy server. Set the value to http or socks.
• proxy.popProxyServer: the proxy server address in the ipaddress:port format.
• proxy.popProxyUserPass: the username and password to authenticate the proxy server. Enter the username and password in the username:password format.

To configure the proxy server for Synthetic PoP to connect to the services that are being monitored, you can use the $network API to set a proxy server in the API script or browser script.

Using external secrets

If you don't want to use Helm chart to create new secrets for the keys controller.instanaKey, redis.password, proxy.popProxyUserPass, and downloadKey, you can use the existing secrets that you have created for the Synthetic PoP deployment by specifying the secret names as follows:

helm install synthetic-pop \
    --repo https://agents.instana.io/helm \
    --namespace syn \
    --set downloadSecret="my-pull-secret" \
    --set controller.instanaKeySecret="my-pop-secret" \
    --set redis.passwordSecret="my-pop-secret" \ 
    --set proxy.popProxyUserPassSecret="my-pop-secret" \
    ...

For the keys controller.instanaKey, redis.password, and proxy.popProxyUserPass, you can use the same secret or different existing secrets. However, when you create the secret, you must use the specified key fields instana-key, redis-password, and proxy-user-pass as follows:

 kubectl create secret generic my-pop-secret  --namespace <namespace> \
 --from-literal=instana-key='<instana agent key>'  \
 --from-literal=redis-password='<redis password>'  \
 --from-literal=proxy-user-pass='<proxy username and password>'  

You need to replace <namespace> with the namespace that you use to install the Synthetic PoP.

The secret for the downloadKey key must use the kubernetes.io/dockerconfigjson type. You can create the secret with the kubernetes.io/dockerconfigjson type as follows:

kubectl create secret docker-registry my-pull-secret --namespace <namespace> --docker-server="https://containers.instana.io" --docker-username="_" --docker-password="<downloadKey>" 

Notes:

• Replace <namespace> with the namespace that you use to install the Synthetic PoP.
• Don't change the value of the --docker-username argument.
• Replace <downloadKey> with the Instana download key.

Synthetic PoP Sensor

To monitor your Synthetic PoP, you are suggested to install the Instana host agent on the Kubernetes cluster where your PoP is installed.

Synthetic PoP sensor is automatically installed to monitor your PoP after the Instana host agent is installed. You can view the metrics in the Instana UI and get alerts if any health issue happens for your Synthetic PoP.

Assigning a PoP controller pod to a fixed node

When you upgrade a Synthetic PoP in a Kubernetes cluster with multiple nodes, the PoP controller pod might be assigned to a different node than you intended. If the PoP controller is assigned to a different node, the PoP entityId is changed. The PoP health data in the PoP monitoring dashboard is lost because the data is associated with the POP entityId. The result and result detail of the Synthetic tests running in this PoP are not impacted.

To prevent losing PoP health data, assign the PoP controller pod to a fixed node before you upgrade or install a Synthetic PoP.

To assign the PoP controller to a fixed node, complete the following steps:

1. Add a label to the node by running the following command:

   kubectl label nodes <yournode> scheduler=synthetic-pop-controller
   

2. Specify the following parameter in the helm install or helm upgrade command: --set controller.nodeSelector.enabled=true

To assign a PoP controller pod to a fixed node and exclude other pods from this node, complete the following steps:

1. Add a taint to the node by running the following command:

   kubectl taint nodes <yournode> toleration=synthetic-pop-controller:PreferNoSchedule
   

2. Specify the following parameter in the helm install or helm upgrade command: --set controller.taintSelector.enabled=true

403 Forbidden

If you see the following error message in the PoP controller log file: Request failed with status code: 403

Possible reasons for this error:

• Instana key is incorrect
• Your license is not validated.
• The Synthetic endpoint URL is valid, but it belongs to a different self-hosted installation or a different production region.

To fix this issue, complete the following steps:

1. In the Instana UI, go to Synthetic Monitoring > Locations.
2. Click New Location. The New Location dialog is displayed.
3. Select Private.
4. Click Next.
5. In the Simple tab, check whether the controller.instanaKey and controller.instanaSyntheticEndpoint parameters are set correctly.
6. Check whether your license is validated and not expired.

404 Not Found

If you see the following error message in the PoP controller log file: Request failed with status code: 404

Possible reasons for this error:

• The synthetic endpoint URL is not set correctly.
• If you are using self-hosted Instana backend, something is wrong with the Synthetic acceptor or load balancer.

To fix this issue, complete the following steps:

1. In the Instana UI, go to Synthetic Monitoring > Locations.
2. Click New Location. The New Location dialog is displayed.
3. Select Private.
4. Click Next.
5. In the Simple tab, check whether the controller.instanaKey and controller.instanaSyntheticEndpoint parameters are set correctly.

If you are using self-hosted Instana backend and the controller.instanaSyntheticEndpoint parameter is correctly set, check whether load balancer and Synthetic acceptor are working correctly. For more information, see Setting up load balancers and DNS

503 Service Unavailable

If you see the following error message in the PoP controller log file: Request failed with status code: 503

Possible reasons for this error:

• Something is wrong with the Ingresses, Routes, or LoadBalancer service that was created when you installed the self-hosted Instana backend.

To fix this issue, check the available Ingresses, Routes, or LoadBalancer services created when you installed the self-hosted Instana backend. Create or update the missing services or services that are not valid. For more information, see Setting up load balancers and DNS.

Name or service not known

If you see the following error message in the PoP controller log file: java.net.UnknownHostException: <hostname>: Name or service not known

Possible reasons for this error:

• The hostname in the controller.instanaSyntheticEndpoint parameter is incorrect.
• A DNS issue in your Kubernetes cluster.

To fix this issue, try the following steps:

• Check whether the controller.instanaSyntheticEndpoint parameter is set correctly.
• Check the DNS configuration in your Kubernetes cluster, see Debugging DNS Resolution.
• Change the hostname to an IP address in controller.instanaSyntheticEndpoint parameter.