Monitoring applications on Red Hat OpenShift Container Platform with Prometheus and Grafana

You can set up application monitoring on Red Hat® OpenShift® Container Platform 4.15 by enabling monitoring for user-defined projects without the need to install an extra monitoring solution. You use Prometheus so that it monitors output metrics from your application while you use the Grafana dashboard to view the application metrics.

This solution deploys a second Prometheus operator instance inside the openshift-user-workload-monitoring namespace. This namespace is configured to monitor all namespaces except the openshift- prefixed namespaces, which are already monitored by the default platform monitoring for the cluster.

Before you begin

Ensure that the MicroProfile Metrics feature is built as part of your Liberty image.

Ensure that the Liberty image that is running your application is deployed to the Red Hat OpenShift Container Platform cluster inside your project namespace.

Before you install application monitoring on Red Hat OpenShift Container Platform, ensure that an application that has a service endpoint for outputting metrics in Prometheus format is running:

The running application must be deployed to the Red Hat OpenShift Container Platform cluster inside your project namespace.
The Prometheus metrics endpoint must be exposed on the /metrics path.

About this task

You configure a monitoring stack. Then, you enable user-defined projects and deploy a service monitor to define the service endpoint that is monitored by the Prometheus instance. You create Grafana custom dashboards to view your application metrics and create custom queries. These Grafana custom dashboards must be deployed on their own Grafana instance by using a community-powered Grafana Operator.

Procedure

Configure the monitoring stack.

You create the cluster-monitoring and user-workload-monitoring-config configuration maps so that you can enable monitoring for user-defined projects in the next step.

When you configure the monitoring stack, a Grafana dashboard is created. However, you delete this Grafana dashboard because you cannot create custom queries since it is read-only.
Enable monitoring for user-defined projects.
Deploy a service monitor inside your project namespace to define the service endpoint that is monitored by the Prometheus instance.
Add the following .yaml file to your WebsphereLibertyApplication custom resource (CR) to deploy the service monitor.
```
kind: WebSphereLibertyApplication
spec:
  …
  monitoring:
    endpoints:
      - interval: 30s
```
Enable Prometheus to scrape data.
The WebsphereLibertyApplication custom resource creates a service monitor for you. You must configure the service monitor so that it uses basic authentication credentials. These credentials are specified in the server.xml file for accessing the /metrics endpoint.
Add the following basicAuth section to the monitoring definition in your WebsphereLibertyApplication custom resource, and replace the basic-auth name with your secret.
```
spec:
…
  monitoring:
    endpoints:
      - basicAuth:
          username:
            key: username
            name: basic-auth
          password:
            key: password
            name: basic-auth
        interval: 30s
```
Confirm that your service is being monitored.

Click Observe > Metrics and insert your own queries specific to the metrics for your application.
Deploy a community-powered Grafana instance so that custom dashboards are created.
Follow the Red Hat blog about Custom Grafana dashboards for Red Hat OpenShift Container Platform 4.
Note: The Grafana Operator's apiVersion and Custom Resources were renamed since the creation of the Red Hat blog about Custom Grafana dashboards for Red Hat OpenShift Container Platform 4. When you are deploying the Grafana data source in the blog, use the following YAML instead.
```
kind: GrafanaDatasource
apiVersion: grafana.integreatly.org/v1beta1
metadata:
  name: prometheus-grafanadatasource
  namespace: my-grafana
spec:
  datasource:
    access: proxy
    editable: true
    isDefault: true
    jsonData:
      httpHeaderName1: 'Authorization'
      timeInterval: 5s
      tlsSkipVerify: true
    name: Prometheus
    secureJsonData:
      httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
    type: prometheus
    url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
instanceSelector:
    matchLabels:
      dashboards: grafana-a
  plugins:
    - name: grafana-clock-panel
      version: 1.3.0
```
You can use sample Grafana dashboards for the custom Grafana dashboards. Look in the featureManager section of the server.xml file for either the mpMetrics feature or the umbrella microProfile feature to determine which dashboard to use.

This Grafana instance is connected to Red Hat OpenShift monitoring in the openshift-monitoring namespace.

CAUTION:
Metrics queries that are run from Grafana dashboards might increase the load on the Prometheus service. Pay attention to the Prometheus CPU and memory usage when you deploy dashboards.

Deploy the Grafana dashboard template to see all the services that are currently being monitored.

Copy the following template into a file named grafana_dashboard.yaml. Ensure that the namespace in the .yaml file is configured to the namespace that you deployed in a previous step.

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  labels:
    app: grafana
  name: template-dashboard
  namespace: prometheus-operator
spec:
  json: |
    {
      "__inputs": [
         {
           "name": "Prometheus",
           "label": "Prometheus",
           "description": "",
           "type": "datasource",
           "pluginId": "prometheus",
           "pluginName": "Prometheus"
         }
       ],
      "__requires": [
         {
           "type": "grafana",
           "id": "grafana",
           "name": "Grafana",
           "version": "5.2.0"
         },
         {
           "type": "panel",
           "id": "graph",
           "name": "Graph",
           "version": "5.0.0"
         },
         {
           "type": "datasource",
           "id": "prometheus",
           "name": "Prometheus",
           "version": "5.0.0"
         },
         {
           "type": "panel",
           "id": "table",
           "name": "Table",
           "version": "5.0.0"
         }
       ],
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": "-- Grafana --",
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "editable": true,
      "gnetId": null,
      "graphTooltip": 0,
      "id": 1,
      "iteration": 1569353980677,
      "links": [],
      "panels": [
        {
          "columns": [],
          "datasource": "Prometheus",
          "fontSize": "100%",
          "gridPos": {
            "h": 6,
            "w": 24,
            "x": 0,
            "y": 2
          },
          "id": 26,
          "links": [],
          "options": {},
          "pageSize": null,
          "scroll": true,
          "showHeader": true,
          "sort": {
            "col": 0,
            "desc": true
          },
          "styles": [
            {
              "alias": "Time",
              "dateFormat": "YYYY-MM-DD HH:mm:ss",
              "pattern": "Time",
              "type": "date"
            },
            {
              "alias": "Namespace",
              "pattern": "namespace",
              "type": "custom"
            },
            {
              "alias": "Service",
              "pattern": "service",
              "type": "custom"
            },
            {
              "alias": "Endpoint",
              "pattern": "endpoint",
              "type": "custom"
            },
            {
              "alias": "",
              "mappingType": 1,
              "pattern": "Value",
              "type": "hidden",
              "unit": "short"
            }
          ],
          "targets": [
            {
              "expr": "count(up) by (namespace, service, endpoint)",
              "format": "table",
              "instant": true,
              "intervalFactor": 1,
              "refId": "A"
            }
          ],
          "title": "Endpoints",
          "transform": "table",
          "type": "table"
        }
      ],
      "refresh": "10s",
      "schemaVersion": 19,
      "style": "dark",
      "tags": [],
      "time": {
        "from": "now-15m",
        "to": "now"
      },
      "timepicker": {
        "refresh_intervals": [
          "1m",
          "5m",
          "15m",
          "1h",
          "1d"
        ]
      },
      "title": "Template-Dashboard",
      "uid": "Template-Dashboard"
    }

View the Grafana dashboard to use the application metrics that Prometheus gathered.

Click Networking > Routes and then, click the link to the Grafana dashboard.