How To
Summary
The requirement is to inspect the data at different stages of the pipeline when noticing that Alerts are not showing up in the AIOps UI.
So the user can check exactly where it is originating from.
Note: Please ensure to have the kafka tool creation script to run and test the below recommendations with. The user can utilize the consumer scripts for the work required in the investigation.
Objective
When an alert is observed to be missing, scan the north-bound kafka topics for the alert message:
This is the direction of north-bound data flow:
Netcool connector -> cp4waiops-cartridge.lifecycle.input.alerts -> cp4waiops-cartridge.irdatalayer.requests.alerts -> AIOps
Environment
Steps
(1) Check whether the alert message appears in cp4waiops-cartridge.lifecycle.input.alerts topic
(a) Look for the alert message (for example run command: consumer.sh | grep <pattern>). Inspect every single one of the find.
Here's a sample of alert message (an update event)
{"tenantid":"cfd95b7e-3bc7-4006-a4a8-a73a79c71255","requestid":"b6079c67-8b2c-4a84-ae1d-9e50df05bfa1","requestTime":"2024-04-16T03:23:37.443484418Z","type":"update","entityType":"alert","entity":{"insights":[{"details":{"lastProcessedEventOccurrenceTime":"2024-04-16T03:23:33Z"},"id":"event-occurrence","type":"aiops.ibm.com/insight-type/deduplication-details"}],"deduplicationKey":"{hostname=null, name=null}-UserLogoutSessionEvent-","eventCount":84341,"state":"open","id":"9e96637d-15cf-4618-bdf7-ef97e6016128","type":{"classification":"UserLogoutSessionEvent","eventType":"problem"},"lastOccurrenceTime":"2024-04-16T03:23:33.000Z"}}
Check the fields in fresh alert message contains these values:
"type":"create"
"eventCount":1
If the fresh alert message so happens to contain type=update, this will cause the system to reject that message due to out-of-order state, because no alert was created prior in order to accept this "update" event.
(2) If the fresh alert message appears in cp4waiops-cartridge.lifecycle.input.alerts and fulfill the positive criteria (type=create), check whether the message is delivered to cp4waiops-cartridge.irdatalayer.requests.alerts.
If the message is not found in the requests.alerts topic, the message is either blocked/rejected by the lifecycle process, or not yet consumed from the input.alerts topic.
Check the lifecycle process pod log for its activities.
(3) If the message passes through cp4waiops-cartridge.irdatalayer.requests.alerts, but not reflected in the AIOps Alert Viewer, check the data layer process.
Additional Information
To upgrade AIOps to 4.5.1, as the patches in 4.5.1 Netcool Connector resolve lots of commonly-seen integration related missing alert issues.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
07 May 2024
UID
ibm17150068