Monitoring Kubernetes

If you still need to reinstall the Legacy Kubernetes sensor on a system if it is not migrated to the Next Generation K8sensor, you can use the final version of the deprecated 1.X agent Helm chart 1.2.74 with the --set k8s_sensor.deployment.enabled=false setting during installation.

To check the status and running version of the Legacy Kubernetes sensor, complete the following steps:

1. From the navigation menu, select Infrastructure.
2. On the Maps tab, click your Kubernetes node.
3. In the node details panel, expand Instana Agent (1) and then click on your agent.
4. On the agent panel, click Open Dashboard.
5. On the agent page, click Sensors Info in the Info area.
6. In the Sensors Info window, search Instana - Kubernetes - Sensor in the search field.

If the sensor is properly enabled, you see a match in the table. The State column displays the value Active, and the Version column shows the version number, for example, 1.2.143.

If the search for Instana - Kubernetes - Sensor and filtering the agent logs for com.instana.sensor-kubernetes yield no results, then check the configuration map of the Instana agent by running the following command:

kubectl -n instana-agent get cm instana-agent -o yaml
 

Make sure that the following setting is not present:

    com.instana.plugin.kubernetes:
      enabled: false
 

If the State column for the Instana - Kubernetes - Sensor displays Waiting for several minutes, it means that the sensor failed to activate.

Search the agent logs, filter for com.instana.sensor-kubernetes, and look for Activating Kubernetes Sensor. Specifically, check for the following message:

ERROR : bundle com.instana.sensor-kubernetes:1.2.143 (246)[com.instana.agent.kubernetes.sensor.Kubernetes(479)] : The activate method has thrown an exception
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/apis/apps/v1/deployments?labelSelector=app%3Dk8sensor. Message: Forbidden!Configured serv
ice account doesn't have access. Service account may have been revoked. deployments.apps is forbidden: User "system:serviceaccount:instana-agent:instana-agent" cannot list resource "deployme
nts" in API group "apps" at the cluster scope.
 

This error message indicates that a service account is not usable for accessing the Kubernetes API server.

When you install the agent, the Next Generation K8sensor is installed automatically by default via the icr.io/instana/k8sensor:latest tag . If you specify the k8s_sensor.deployment.enabled value, make sure that it is set to true (the default value).

You can configure the K8sensor polling rate by setting k8s_sensor.pollrate in one of the following locations:

• Helm chart (for example, --set k8s_sensor.pollrate=15s)
• InstanaAgent custom resource (for example, k8s_sensor.pollrate: "15s")

The default value is 10 seconds. Valid values must be specified in seconds, ranging from 1s to 30s. Values outside this range are automatically clamped to the nearest threshold.

To determine if the K8sensor is running on a cluster, run the following command to list deployments with the label app: k8sensor.

kubectl get deployments --all-namespaces -l app=k8sensor
 

If K8sensor is in the list, then K8sensor is enabled. When K8sensor is enabled, the Legacy Kubernetes sensor is automatically disabled. In the configuration.yaml file, you can see the enabled value is changed to false as follows:

    com.instana.plugin.kubernetes:
      enabled: false
 

The K8sensor version is best identified by the sha256 hash of its image. To check the image hash of the running K8sensor container, run the following command:

kubectl get po -n instana-agent -l app=k8sensor  -o jsonpath='{ .items[0].status.containerStatuses[0].imageID }'
 

Instana does not currently provide a built-in autoscaling (HPA template) for the Next Generation K8sensor. However, you can enable autoscaling by applying a standard Kubernetes HorizontalPodAutoscaler (HPA) to the k8sensor deployment. This workaround requires a cluster with metrics-server installed.

The Instana agent operator manages the initial replica count of the k8sensor deployment from the InstanaAgent custom resource (CR). To avoid conflicts with HPA, set the CR replica count once and keep it equal to the HPA's minReplicas.

If listing the deployments with the label app: k8sensor returns no results, the cluster is not running K8sensor. This issue means that K8sensor was disabled when the agent was deployed. To resolve this issue, redeploy the agent with K8sensor enabled.

On the Kubernetes page, click a cluster or a namespace. You can see Kubernetes dashboards that present all the information for a certain Kubernetes entity. The context is always accessible by the context path. In the following screenshot, you can see a namespace that is named "robot-shop" in a cluster called "k8s-demo-cluster".

Kubernetes dashboards are structured as follows:

• Summary shows the most relevant information for a certain entity. This dashboard starts with a status line that shows the status and related information, such as age. In the next section, you can see the CPU, memory, and pod information, which provides the consumed resources, including the pods. Sections like Top Deployments and Top Pods in the following screenshot show potential hotspots, which you might want to have a look at. The Logs section shows you the distribution chart of relevant logs for that entity, which complements the entity metrics. The chart is interactive and allows selection and highlight over all the measured values. You can focus on the selected time period or jump to the Analyze section to continue a troubleshooting journey.

• Details shows detailed information like "labels", "annotation", and the "spec".

• Events shows all relevant Kubernetes events and links them to the respective dashboards.

• Related Entities like "Deployments", "K8s Services", and "Pods" are shown as tabs of Kubernetes dashboards. What is shown depends on the entity that you selected.

To monitor a PV, complete the following steps in the Instana UI:

1. From the navigation menu, select Infrastructure.

2. On the Infrastructure dashboard, click Analyze Infrastructure.

3. On the Analyze Infrastructure dashboard, search for Kubernetes Persistent Volume and view the total count as shown in the following image:

   

4. To view the list of PVs, select Kubernetes Persistent Volume from the list of infrastructure types as shown in the following image:

   

5. To view PV metrics, select a PV from the list. You can view a list of PV metrics with type and current value.

To set up a Smart Alert for PV metrics, complete the following steps in the Instana UI:

1. From the navigation menu, select Infrastructure > Smart Alerts > Create Smart Alert.

2. In the Create Smart Alert window, select any of the available PV metrics, such as Capacity Used %, and filter according to the required entity.

   

For more information about configuring smart alerts, see Smart Alerts.

Instana supports the most popular storage classes provided by most cloud providers, for both static and dynamic provisioning. The following support matrix indicates different storage classes with verified support.

To monitor a PVC, complete the following steps in the Instana UI:

1. From the navigation menu, select Infrastructure.

2. On the Infrastructure dashboard, click Analyze Infrastructure.

3. On the Analyze Infrastructure dashboard, search for Kubernetes Persistent Volume and view the total count as shown in the following image:

   

4. To view the list of PVCs, select Kubernetes Persistent Volume Claim from the list of infrastructure types as shown in the following image:

   

5. To view PVC metrics, select a PVC from the list of PVCs. You can view a list of PVC metrics with type and value.

To set up a Smart Alert for PVC metrics, complete the following steps in the Instana UI:

1. From the navigation menu, select Infrastructure > Smart Alerts > Create Smart Alert

2. In the Create Smart Alert window, select any of the available PV metrics, such as Capacity Used in % and filter according to the required entity.

   

For more information about configuring Smart Alerts, see Smart Alerts.

To learn more about PV or PVC monitoring and its need, see the Demystifying PVCs and PVs IBM blog post .

To debug the service mesh by-pass, follow the steps:

1. Verify that the service mesh by-pass is enabled.
2. Verify that the iptable rules are applied to the container.

To verify whether the service mesh by-pass is enabled, check in the Instana agent logs by running the following command:

kubectl logs -l app.kubernetes.io/instance=instana-agent -n instana-agent -c instana-agent
 

If the service mesh by-pass is enabled, you can find the following log lines, which indicate that an inbound or outbound by-pass entry is written for the denoted process:

Inbound by-pass:

2021-04-26T08:13:57.065+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying inbound service mesh bypass for process '764670'
 

Outbound by-pass:

2021-04-26T08:13:57.140+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying outbound service mesh bypass for process '764670'
 

The easiest way to verify that the iptable rules is to shell into the Instana agent and list the target container iptables rules as follows. Replace ${PID} with the pid of the JVM process:

kubectl -n instana-agent exec -it ${INSTANA_AGENT_POD} -c instana-agent  -- /bin/bash
nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_OUTPUT
 

If the chains are applied, you can see an output as follows:

Chain INSTANA_OUTPUT (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            10.128.15.237
ACCEPT     tcp  --  0.0.0.0/0            10.64.0.1
ACCEPT     tcp  --  0.0.0.0/0            169.254.123.1
 

Check whether bidirectional communication between the Instana agent and your JVM processes is supported by running the following command:

nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_INBOUND
 

The result is similar to the following output:

Chain INSTANA_INBOUND (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  10.128.15.237        10.64.0.14
ACCEPT     tcp  --  10.64.0.1            10.64.0.14
ACCEPT     tcp  --  169.254.123.1        10.64.0.14
 

Depending on when the iptable rules were applied, it can take a few minutes for the process to be instrumented and the data to be visible in Instana's dashboards.

1. Clone the serviceability repository.

2. Navigate to the agent/k8s directory.

3. Read the instructions in the README.md file in that directory, making sure that the prerequisites are fulfilled and that you are logged in into the cluster.

4. Run instana-k8s-mustgather.sh.

5. After the script finishes, a .tgz file is generated, which contains all the diagnostic information that is required for the customer support team to help service the ticket.