Adding Alertmanager webhooks

Edit online

You can add new webhooks to an existing Alertmanager configuration to receive real-time alerts about the health of the storage cluster. You have to enable incoming webhooks to allow asynchronous messages into third-party applications.

For example, if an OSD is down in an IBM Storage Ceph cluster, you can configure the Alertmanager to send notification on Google chat.

Prerequisites

Edit online

  • A running IBM Storage Ceph cluster with monitoring stack components enabled.

  • Incoming webhooks configured on the receiving third-party application.

Procedure

Edit online

  1. Log into the cephadm shell:

    Example 

    [root@host01 ~]# cephadm shell

  2. Configure the Alertmanager to use the webhook for notification:

    Syntax 

    service_type: alertmanager
spec:
  user_data:
    default_webhook_urls:
    - "_URLS_"

    The ‘default_webhook_urls is a list of additional URLs that are added to the default receivers’ webhook_configs` configuration.

    Example 

    service_type: alertmanager
spec:
  user_data:
    webhook_configs:
    - url: 'http:127.0.0.10:8080'

  3. Update Alertmanager configuration:

    Example 

    [ceph: root@host01 /]#  ceph orch reconfig alertmanager

Verification

Edit online

  • An example notification from Alertmanager to Gchat:

    Example 

    using: https://chat.googleapis.com/v1/spaces/(xx- space identifyer -xx)/messages
posting: {'status': 'resolved', 'labels': {'alertname': 'PrometheusTargetMissing', 'instance': 'postgres-exporter.host03.chest
response: 200
response: {
"name": "spaces/(xx- space identifyer -xx)/messages/3PYDBOsIofE.3PYDBOsIofE",
"sender": {
"name": "users/114022495153014004089",
"displayName": "monitoring",
"avatarUrl": "",
"email": "",
"domainId": "",
"type": "BOT",
"isAnonymous": false,
"caaEnabled": false
},
"text": "Prometheus target missing (instance postgres-exporter.cluster.local:9187)\n\nA Prometheus target has disappeared. An e
"cards": [],
"annotations": [],
"thread": {
"name": "spaces/(xx- space identifyer -xx)/threads/3PYDBOsIofE"
},
"space": {
"name": "spaces/(xx- space identifyer -xx)",
"type": "ROOM",
"singleUserBotDm": false,
"threaded": false,
"displayName": "_privmon",
"legacyGroupChat": false
},
"fallbackText": "",
"argumentText": "Prometheus target missing (instance postgres-exporter.cluster.local:9187)\n\nA Prometheus target has disappea
"attachment": [],
"createTime": "2022-06-06T06:17:33.805375Z",
"lastUpdateTime": "2022-06-06T06:17:33.805375Z"