IBM Cloud Private cluster monitoring

You can use the IBM® Cloud Private cluster monitoring dashboard to monitor the status of your cluster and applications.

The monitoring dashboard uses Grafana and Prometheus to present detailed data about your cluster nodes and containers. For more information about Grafana, see the Grafana documentation Opens in a new tab. For more information about Prometheus, see the Prometheus documentation Opens in a new tab.

Accessing the monitoring dashboard

  1. Log in to the IBM Cloud Private management console.

    Note: When you log in to the management console, you have administrative access to Grafana. Do not create more users within the Grafana dashboard or modify the existing users or org.

  2. To access the Grafana dashboard, click Menu > Platform > Monitoring. Alternatively, you can open https://<IP_address>:<port>/grafana, where <IP_address> is the DNS or IP address that is used to access the IBM Cloud Private console. <port> is the port that is used to access the IBM Cloud Private console.
  3. To access the Alertmanager dashboard, click Menu > Platform > Alerting. Alternatively, you can open https://<IP_address>:<port>/alertmanager.
  4. To access the Prometheus dashboard, open https://<IP_address>:<port>/prometheus.

  5. From the Grafana dashboard, open one of the following default dashboards:

    • ElasticSearch
      Provides information about ElasticSearch cluster statistics, shard, and other system information.

    • Etcd by Prometheus
      Etcd Dashboard for Prometheus metrics scraper.

    • Helm Release Metrics
      Provides information about system metrics such as CPU and Memory for each Helm release that is filtered by pods.

    • ICP Namespaces Performance IBM Provided 2.5
      Provides information about namespace performance and status metrics.

    • Cluster Network Health (Calico)
      Calico hosts workload and system metric performance information.

    • ICP Performance IBM Provided 2.5
      Provides TCP system performance information about Nodes, Memory, and Containers.

    • Kubernetes Cluster Monitoring
      Monitors Kubernetes clusters that use Prometheus. Provides information about cluster CPU, Memory, and Filesystem usage. The dashboard also provides statistics for individual pods, containers, and systemd services.

    • Kubernetes POD Overview
      Monitors pod metrics such as CPU, Memory, Network pod status, and restarts.

    • NGINX Ingress controller
      Provides information about NGINX Ingress controller metrics that can be sorted by namespace, controller class, controller, and ingress.

    • Node Performance Summary
      Provides information about system performance metrics such as CPU, Memory, Disk, and Network for all nodes in the cluster.

    • Prometheus Stats
      Dashboard for monitoring Prometheus v2.x.x.

    • Storage GlusterFS Health
      Provides GlustersFS Health metrics such as Status, Storage, and Node.

    • Rook-Ceph
      Dashboard that provides statistics about Ceph instances.

    • Storage Minio Health
      Provides storage and network details about Minio server instances.

    Note: If you configure pods to use host level resource such as host network, the dashboards display the metrics of the host but not the pod itself.

If you want to view other data, you can create new dashboards or import dashboards from JSON definition files for Grafana.

Metrics collected out of the box

IBM Cloud Private provides the following exporters to provide metrics. The exporters expose metrics endpoints as Kubernetes services.

Some IBM Cloud Private Kubernetes pods provide metrics endpoints for Prometheus:

In addition, Prometheus has preconfigured scrape targets that communicate with several targets to scrape metrics:

Prometheus displays scrape targets in its user interface as links. These addresses are typically not accessible from a user's browser as they are on the Kubernetes cluster internal network. Only the Prometheus server needs to be able to access the addresses.

Role-based access

Role-base access for monitoring API

A user with role ClusterAdministrator，Administrator or Operator can access monitoring service. A user with role ClusterAdministrator or Administrator can perform write operations in monitoring service, including deleting Prometheus metrics data, and updating Grafana configurations.

Role-base access for monitoring data

Starting with version 1.2.0, the ibm-icpmonitoring Helm chart introduces an important feature. It offers a new module that provides role-based access controls (RBAC) for access to the Prometheus metrics data.

The RBAC module is effectively a proxy that sits in front of the Prometheus client pod. It examines the requests for authorization headers, and at that point, enforces role-based controls. In general, the rules concerning RBAC are as follows:

A user with the role ClusterAdministrator can access any resource. A user with any other role can only access data in the namespaces for which that user is authorized.

Installing monitoring service in IBM Cloud Private

Monitoring service is installed by default during IBM Cloud Private installation. You can also select to install monitoring service from the Catalog or CLI.

Installing monitoring service from the Catalog

You can deploy more monitoring stacks with customized configurations from the Catalog in the IBM Cloud Private management console.

  1. From the Catalog page, click the ibm-icpmonitoring Helm chart to configure and install it.

  2. Provide required values for the following parameters.

    • Helm release name: "monitoring"
    • Target namespace: "kube-system"
    • Mode of deployment: "Managed"
    • Cluster access address: Specify the Domain Name Service (DNS) or IP address that is used to access the IBM Cloud Private console.
    • Cluster access port: Specify the port that is used to access the IBM Cloud Private console. The default port is 8443.
    • etcd address: Specify the Domain Name Service (DNS) or IP address for etcd node(s).

Installing monitoring service from the CLI

  1. Install the Kubernetes command line (kubectl). For information about the kubectl CLI, see Accessing your cluster from the Kubernetes CLI (kubectl).
  2. Install the Helm command line interface (CLI). For information about the Helm CLI, see Installing the Helm CLI (helm).
  3. Install the ibm-icpmonitoring Helm chart. Run the following command:
    helm install -n monitoring --namespace kube-system --set mode=managed --set clusterAddress=<IP_address> --set clusterPort=<port> ibm-icpmonitoring-1.4.0.tgz

<IP_address> is the DNS or IP address that is used to access the IBM Cloud Private console.

<port> is the port that is used to access the IBM Cloud Private console.

For more information about parameters that you can configure during installation, see Parameters.

Data persistence configuration

By default, user data in the monitoring service components such as Prometheus, Grafana, or AlertManager, is not stored in persistent volumes. The user data is lost if the monitoring service component crashes. To store user data in persistent volumes, you must configure related parameters when you install the monitoring service. Use one of the following options to enable persistent volumes:

During configuration, select the checkbox for Persistent volume, and provide values for the following parameters:

In the following example, the value of Field to select the volume is component. The value of Value of the field to select the volume is prometheus: 

    apiVersion: v1
    kind: PersistentVolume
    metadata:
        name: monitoring-prometheus-pv
        labels:
            component: prometheus
    .......

For information about creating storage classes, PersistentVolume, and PersistentVolumeClaim, see Storage.

Configuring Prometheus server

You can configure the following Prometheus server parameters during preinstallation or post installation:

Preinstallation configuration

For monitoring service installation and IBM Cloud Private, you can configure the parameters in the config.yaml file before installation. For example, your config.yaml file might resemble the following content:

monitoring:
  prometheus:
    scrape_Interval: 1m
    evaluation_Interval: 1m
    retention: 24h
    resources:
      limits:
        memory: 2048Mi

If you choose to install monitoring service from the Catalog, you can configure the parameters in related console fields.

Post installation configuration

You can also update parameters after you install the monitoring service.

Notes:

Alerts

Default alerts

Capability to install default alerts is available in version 1.3.0 of the ibm-icpmonitoring chart. Some alerts provide customizable parameters to control the alert frequency. You can configure the following alerts during installation.

Field Default Value
prometheus.alerts.nodeMemoryUsage.nodeMemoryUsage.enabled true
prometheus.alerts.nodeMemoryUsage.nodeMemoryUsageThreshold 85
Field Default Value
prometheus.alerts.highCPUUsage.enabled true
prometheus.alerts.highCPUUsage.highCPUUsageThreshold 85
Field Default Value
prometheus.alerts.failedJobs true
Field Default Value
prometheus.alerts.elasticsearchClusterHealth false
Field Default Value
prometheus.alerts.podsTerminated true
Field Default Value
prometheus.alerts.podsRestarting true

Managing alert rules

You can use the Kubernetes custom resource, AlertRule, to manage alert rules in IBM Cloud Private.

The following sample-rule.yaml file is an example of an AlertRule resource definition.

apiVersion: monitoringcontroller.cloud.ibm.com/v1
kind: AlertRule
metadata:
  name: sample-rule
spec:
  enabled: true
  data: |-
    groups:
      - name: a.rules
        rules:
          - alert: NodeMemoryUsage
            expr: ((node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes))/ node_memory_MemTotal_bytes) * 100 > 5
            annotations:
              DESCRIPTION: '{{ $labels.instance }}: Memory usage is above the 15% threshold.  The current value is: {{ $value }}.'
              SUMMARY: '{{ $labels.instance }}: High memory usage detected'

You must provide the following parameter values:

Use kubectl to manage alert rules.

Configuring AlertManager

You can configure Prometheus AlertManager to integrate external alert service receivers, such as Slack or PagerDuty, for IBM Cloud Private.

Important: ConfigMap changes are lost when you upgrade, roll back, or update the monitoring release. In addition, the ConfigMap format can change between releases.

  1. Edit configuration map monitoring-prometheus-alertmanager to update AlertManager configurations.

    kubectl edit configmap monitoring-prometheus-alertmanager -n kube-system

    For more information about configuring AlertManager, see Configuration Opens in a new tab and Notification template examples Opens in a new tab

  2. Allow several minutes for the updates to take effect. Open the AlertManager dashboard at https://<Cluster Master Host>:<Cluster Master API Port>/alertmanager. Where, <Cluster Master Host>:<Cluster Master API Port> is defined in Master endpoint.

    • If you configured alerts, and they are triggered, you can see the alerts in the AlertManager dashboard.
    • If you configured an external alert receiver such as Slack or PagerDuty, you can view the alerts in the dashboard for that particular service.
    • You can return to the dashboards to view alerts at any time.

Managing Grafana dashboards

You can manage Grafana dashboards by operating on a Kubernetes custom resource MonitoringDashboard in IBM Cloud Private. The following sample-dashboard.yaml file is an example of a MonitoringDashboard resource definition.

apiVersion: monitoringcontroller.cloud.ibm.com/v1
kind: MonitoringDashboard
metadata:
  name: sample-dashboard
spec:
  enabled: true
  data: |-
    {
        "id": null,
        "uid": null,
        "title": "Marco Test Dashboard",
        "tags": [ "test" ],
        "timezone": "browser",
        "schemaVersion": 16,
        "version": 1
      }

You must provide the following parameter values:

You can use kubectl to manage the dashboard.

Configure applications to use monitoring service

Modify the application to expose the metrics.

Logs and metrics management for Prometheus

You can modify the time period for metric retention by updating the storage.tsdb.retention parameter in the config.yaml file. By default this value is set at 24h, which means that the metrics are kept for 24 hours and then purged. See Configuring the monitoring service.

However, if you need to manually remove this data from the system, you can use the rest API that is provided by the Prometheus component.

The target URL must have the format:

https://<IP_address>:<Port>/prometheus

Accessing monitoring service APIs

You can access monitoring service APIs such as Prometheus and Grafana APIs. Before you can access the APIs, you must obtain authentication tokens to specify in your request headers. For information about obtaining authentication tokens, see Preparing to run component or management API commands.

After you obtain the authentication tokens, complete the following steps to access the Prometheus and Grafana APIs.

  1. Access the Prometheus API at url, https://<Cluster Master Host>:<Cluster Master API Port>/prometheus/* and get boot times of all nodes.

    • $ACCESS_TOKEN is the variable that stores the authentication token for your cluster.
    • <Cluster Master Host> and <Cluster Master API Port> are defined in Master endpoints.
    curl -k -s -X GET -H "Authorization:Bearer $ACCESS_TOKEN" https://<Cluster Master Host>:<Cluster Master API Port>/prometheus/api/v1/query?query=node_boot_time_seconds

    For detailed information about Prometheus APIs, see Prometheus HTTP API Opens in a new tab.

  2. Access the Grafana API at url, https://<Cluster Master Host>:<Cluster Master API Port>/grafana/* and obtain the sample dashboard.

    • $ACCESS_TOKEN is the variable that stores the authentication token for your cluster.
    • <Cluster Master Host> and <Cluster Master API Port> are defined in Master endpoints.
    curl -k -s -X GET -H "Authorization: Bearer $ACCESS_TOKEN” "https://<Cluster Master Host>:<Cluster Master API Port>/grafana/api/dashboards/db/sample"

    For detailed information about Grafana APIs, see Grafana HTTP API Reference Opens in a new tab.

Support for custom cluster access URL in monitoring service

You can customize the cluster access URL. For more information, see Customizing the cluster access URL. After you complete the customization, you must manually edit the deployments for Prometheus and Alertmanager and verify that all external links are correct.

monitoring-prometheus deployment

Edit the monitoring-prometheus deployment from the IBM Cloud Private console or by using kubectl. For example:

 kubectl edit deployment monitoring-prometheus -n kube-system

In the monitoring-prometheus deployment, change --web.external-url=* to the following:

 --web.external-url=https://<custom_host>:<custom_port>/prometheus

<custom_host> and <custom_port> are the customized host name and port that you defined in the custom cluster access URL.

monitoring-prometheus-alertmanager deployment

Edit the monitoring-prometheus-alertmanager deployment from the IBM Cloud Private console or by using kubectl. For example:

 kubectl edit deployment monitoring-prometheus -n kube-system

In the monitoring-prometheus-alertmanager deployment, change --web.external-url=* to the following:

 https:<custom_host>:<custom_port>/alertmanager

<custom_host> and <custom_port> are the customized host name and port that you defined in the custom cluster access URL.