Monitoring Host

Note:

System ID can be used for correlation with asset management systems. Instana agent collects the System ID by default for Linux OS. For other supported operating systems (Windows, macOS, Solaris, AIX), you need to explicitly enable the feature by using the agent configuration.yaml file:

com.instana.plugin.host:
  collectSystemId: true

CPU usage

Overall CPU usage as a percentage.

• In an AIX LPAR environment, set useMpstat to true to collect more accurate CPU usage.

com.instana.plugin.host:
  useMpstat: true

Datapoint: Filesystem

Granularity: 1 second

Memory usage

• On Linux, the used value is computed in percentage by using the formula (total - actualFree) / total. The sensor uses the actualFree value, the real constrained memory which includes free and cached memory, instead of just free which is usually a pretty low value (used for caching/buffering).

Datapoint: Filesystem

Granularity: 1 second

• On AIX, the used value is computed in percentage by using the formula (computational + non-computational) / real total. The non-computational is considered as part of used memory, potentially resulting in a relatively high used value. However, a high used value doesn't necessarily indicate a need for more memory.{: note} The determination of memory over-commitment is based on the comparison between computational memory and the real memory in the system. Therefore, the percentage of computational is more informative for estimating memory usage on AIX.

Datapoint: AIX perfstat_memory_total interface

Granularity: 1 second

CPU load

The average number of processes being or waiting to be executed over past selected period of time.

Datapoint: Filesystem

Granularity: 5 seconds

CPU usage

CPU usage values as a percentage; user, system, wait, nice, and steal. The values are displayed on a graph over a selected time period.

Datapoint: Filesystem

Granularity: 1 second

Context switches

The total number of context switches. This is supported only on Linux hosts. The value is displayed on a graph over a selected time period.

Datapoint: Filesystem

Granularity: 1 second

CPU load

CPU load. The value is displayed on a graph over a selected time period.

Datapoint: Filesystem

Granularity: 1 second

Individual CPU usage

Individual CPU usage values as a percentage; user, system, wait, nice, and steal. The values are displayed on a graph over a selected time period.

Datapoint: Filesystem

Granularity: 1 second

Individual GPU usage

Individual GPU usage values.

Supported Nvidia Graphic Cards:

Supported OS: Linux

Prerequisites: Installed latest official Nvidia drivers.

Starting Instana Agent Docker container with GPU support is documented here: Enable GPU monitoring through Instana Agent container.

GPU Memory/Process

The following list of processes utilizes GPU.

Supported Nvidia Graphic Cards:

Supported OS: Linux

Prerequisites: Installed latest official Nvidia drivers.

Starting Instana Agent Docker container with GPU support is documented here: Enable GPU monitoring through Instana Agent container.

Memory

• Linux:
  • memory used and swap used values are displayed in percentage.
  • swap total, swap free, buffers, cached, and available are valued as a byte. The values are displayed on a graph over a selected time period.

Datapoint: Filesystem

Granularity: 1 second

• AIX:
  • memory used, swap used and virtual used values are displayed in percentage.
  • swap total, swap free, virtual total, virtual free are values as a byte.
  • computational and non-computational values are displayed both as a percentage and in byte format in two separate charts.
  • page-in per second and page-out per second are displayed as the number of page-in and page-out events per second. You can visualize all these values on a graph over a user-selected time period.

Datapoint: AIX perfstat_memory_total interface

Granularity: 1 second

Open files

Open files usage when available on the operating system; current vs max. The values are displayed on a graph over a selected time period.

Filesystems

Filesystems per device.

* The total, read, and write utilization datapoint metrics display the disk I/O utilization as a percentage. This functionality is compatible only with Linux.

* Leaked (refers to deleted files that are in use and equates to capacity - used - free. On Linux you can find these files with lsof | grep deleted).

** The Total Utilization, Read Utilization, and Write Utilization datapoints are not supported for Network File Systems.

Instana will by default only monitor local filesystems. It is possible to explicitly list the filesystems that shall be monitored or excluded in the configuration.yaml file. The name for the config setting is the device name, which can be obtained from the first column of mtab file or df command output.

Temporary filesystems need to be specified in the following format: tmpfs:/mount/point. For example, list of filesystems to be monitored:

com.instana.plugin.host:
  filesystems:
    - '/dev/sda1'
    - 'tmpfs:/sys/fs/cgroup'
    - 'server:/usr/local/pub'

or to be included / excluded:

com.instana.plugin.host:
  filesystems:
    include:
      - '/dev/xvdd'
      - 'tmpfs:/tmp'
      - 'server:/usr/local/pub'
    exclude:
      - '/dev/xvda2'

com.instana.plugin.host:
  nfs_all: true

Network interfaces

Network traffic and errors per an interface.

TCP activity

TCP activity values are displayed on a graph over a selected time period.

Instana doesn't support the TCP activity metric for Sun Solaris hosts.

Process top list

The top process list is updated every 30 seconds, and it contains only the processes with significant system usage. For example, the processes with more than 10% CPU usage over the last 30 seconds or processes with more than 512 MB memory usage (RSS) are displayed in the process top list.

To create a combined list of processes from the top 10 CPU and memory usage lists, set combineTopProcesses to true. The processes are included in the combined list even if their CPU usage is less than 10% or memory usage is less than 512 MB. If the same process is listed in the top 10 CPU and top 10 memory usage lists, it is listed only once in the combined list, which can include up to 20 entries.

com.instana.plugin.host:
  combineTopProcesses: true

Linux top semantics are used. 100% CPU refers to full use of a single CPU core, and you can search a history of snapshots from the previous month. The normalized CPU is calculated by dividing the CPU by the number of logical processors.

Installed Packages List

When the collectInstalledSoftware is set to true in the configuration.yaml file, installed packages on an operating system can be extracted once a day.

The following Linux distributions are currently supported:

• Debian-based (dpkg)
• Red Hat-based (rpm and yum)

com.instana.plugin.host:
  collectInstalledSoftware: true # [true, false]

SELinux policy blocking eBPF

If you have SELinux installed on your host, you usually need to create a policy to allow the agent to leverage eBPF. SELinux may prevent unconfined services like the host agent from issuing the bpf_* syscall that the eBPF sensor uses to instrument the Linux kernel. To verify that this is happening, one must look in the log entries of the Audit system, which is stored by default in the /var/log/audit/audit.log.

The following is an example is from a Red Hat Linux machine:

$ cat /var/log/audit/audit.log | grep ebpf
type=AVC msg=audit(1598891569.452:193): avc:  denied  { map_create } for  pid=1612 comm="ebpf-preflight-" 
scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 
tclass=bpf permissive=0
type=SYSCALL msg=audit(1598891569.452:193): arch=c000003e syscall=321 success=no exit=-13 
a0=0 a1=7ffc0e1f5020 a2=78 a3=fefefefefefefeff items=0 ppid=1502 pid=1612 auid=4294967295 
uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ebpf-preflight-" 
exe="/opt/instana/agent/data/repo/com/instana/ebpf-preflight/0.1.6/ebpf-preflight-0.1.6.bin" 
subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=PROCTITLE msg=audit(1598891569.452:193):
proctitle="/opt/instana/agent/data/repo/com/instana/ebpf-preflight/0.1.6/ebpf-preflight-0.1.6.bin"

Note that audit log files are usually rotated, so we have to run this command not long after starting the host agent.

In the log file, we see that the map_create syscall was denied. To allow the eBPF sensor to make this syscall we need to create an SELinux policy. For this we need the program audit2allow. On Red Hat systems this can be installed as follows:

yum install policycoreutils-python

With audit2allow, we can then create raw policy files based on the log entries:

grep ebpf /var/log/audit/audit.log | audit2allow -M instana_ebpf

The command above will create the following files:

ls -Al | grep instana_ebpf

-rw-r--r--. 1 root                    root                      886 31. Aug 18:31 instana_ebpf.pp
-rw-r--r--. 1 root                    root                      239 31. Aug 18:31 instana_ebpf.te

The raw policy file, called instana_ebpf.te, now contains an instruction to allow the denied syscall:

$ cat instana_ebpf.temodule instana_ebpf 1.0;require {
	type unconfined_service_t;
	class bpf map_create;
}#============= unconfined_service_t ==============#!!!! This avc is allowed in the current policy
allow unconfined_service_t self:bpf map_create;

This policy will allow any app of type unconfined (very generic) to make the map_create syscall.

Additionally, the eBPF sensor needs a few more syscalls. We have to edit the instana_ebpf.te file so it looks like this:

$ cat instana_ebpf.te module instana_ebpf 1.0;require {
	type unconfined_service_t;
	class bpf { map_create map_read map_write prog_load prog_run };
}#============= unconfined_service_t ==============#!!!! This avc is allowed in the current policy
allow unconfined_service_t self:bpf { map_create map_read map_write prog_load prog_run };

This file then must be re-written to a binary format as the instana_ebpf.mod file:

$ checkmodule -M -m -o instana_ebpf.mod instana_ebpf.te
checkmodule:  loading policy configuration from instana_ebpf.te
checkmodule:  policy configuration loaded
checkmodule:  writing binary representation (version 19) to instana_ebpf.mod

The instana_ebpf.mod file must be repackaged as a loadable module:

semodule_package -o instana_ebpf.pp -m instana_ebpf.mod

And finally we can apply the policy package:

semodule -i instana_ebpf.pp

Any unconfined process, such as the host agent, can now make those syscalls.