Configuring Alert manager

The following instructions assume that you are configuring Alertmanager that are deployed as part of the Prometheus-operator deployment. Rules are configured as part of a configuration map and the rest of the configuration (routes and receivers) is configured as a Kubernetes secret object. The following steps walk you through the process of creating a sample alert configuration for a CICS® TX on Cloud abend count metric that be received on a slack channel. The instructions below assumes that your Prometheus deployment is under the monitoring namespace. If your deployment is on a different namespace, you need to change the namespace attribute accordingly.

Copy paste the following contents into the Prometheus-rules.yaml file:

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-example-rules
  namespace: monitoring
  labels:
    role: alert-rules
    prometheus: example
data:
   example.rules.yaml: |+
      groups:
      - name: transaction_abend_alert
        rules:
         - alert: cics_TranAbend
            expr: cics_NumTranAbends > 5
            labels:
               severity: critical
             annotations:
             description: Unexpected application abends on the cics region
             summary: Number of transaction abends has crosses the threshold value.

This defines the rule for our alert; The name of the alert is “cics_TranAbend” and it is triggered when the CICS TX on Cloud abend count exceeds 5 (cics_NumTranAbends > 5)

Follow the instructions in the following link to crate a slack application and enable incoming webhooks on it. Once done make a note of the webhook URI of your application.
https://api.slack.com/incoming-webhooks

Create a file that is called alertmanager.yaml and copy paste the following content into it:

global:
  slack_api_url:        "https://hooks.slack.com/services/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"
receivers:
  - name: default-receiver
    slack_configs:
    - channel: "#bluefriendsapp"
      send_resolved: true
route:
  receiver: default-receiver
  routes:
  - match:
      severity: critical
    receiver: default-receiver

Create a file called alertmanager-secret-k8s.yaml and copy paste the following contents into it:

apiVersion: v1
data:
  alertmanager.yaml: ALERTMANAGER_CONFIG
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
type: Opaque

Run the following command to create the Kubernetes secret object:

sed "s/ALERTMANAGER_CONFIG/$(cat alertmanager.yaml | base64 -w0)/g" alertmanager-secret-k8s.yaml | kubectl apply -f -

Run the following command to create the Kubernetes configuration map object:
```
Kubectl create -f  Prometheus-rules.yaml
```
Use the Prometheus web-UI to view the state of the configured alert. The web-UI can be accessed by using the node IP and port that is configured as part of the deployment:
http://<node IP>:<Prometheus service node port>/alerts