Troubleshooting
Problem
AIMgr installation is in ERROR state as ChatOps is reporting that the certificate expired
Symptom
The following error messages can be observed in the "chatops-orchestrator" log:
[2023-02-15 15:21:10,118] [ERROR] [aiokafka:444] Unable connect to node with id 0: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)
[2023-02-15 15:21:10,118] [ERROR] [aiokafka:324] Unable to update metadata from [0]
Cause
Kafka and elastic service CA root certificate expired. AI manager operator needs to regenerate truststores for clients and client pods need to be restarted. ChatOps is a Kafka client and "similar-incidents-service" is an elastic search client.
Resolving The Problem
Run the following commands to resolve the issue:
1.
2. Restart "aimanager-operator" pod
3. Verify that the job "aimanager-aio-create-truststore" is in completed state
4. Restart "aimanager" related pods
oc delete job aimanager-aio-create-truststore
2. Restart "aimanager-operator" pod
oc get pod | grep aimanager-operator | awk '{print $1}' | xargs oc delete pod
3. Verify that the job "aimanager-aio-create-truststore" is in completed state
oc get pod | grep aimanager-aio-create-truststore | grep Completed
4. Restart "aimanager" related pods
oc get pod | grep aimanager-aio-controller | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-ai-platform-api-server | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-change-risk | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-cr-api | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-chatops-orchestrator | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-chatops-slack-integrator | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-chatops-teams-integrator | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-similar-incidents-service | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-luigi-daemon-0 | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-oob-recommended-actions | awk '{print $1}' | xargs oc delete pod
oc get pod | grep aimanager-aio-log-anomaly-detector | awk '{print $1}' | xargs oc delete pod
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRS1M","label":"IBM Cloud Pak for Watson AIOps"},"ARM Category":[{"code":"a8m3p000000hAHuAAM","label":"Watson AIOps-\u003EAI Manager-\u003EChatOps"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
20 February 2023
UID
ibm16956950