IBM Support

Kafka pod errors - Failed authentication with ip_address

Troubleshooting


Problem

Unable to see any alerts

Symptom

The following log messages are observed in the iaf-system-kafka-0 pod log:
 
2023-01-23 16:05:24,151 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /<IP_ADDRESS> (channelId=<IP_ADDRESS> :9093-<IP_ADDRESS>:42830-1184423) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SASL_SSL-8]

Cause

Kafka is unable to access the following archiving components, pods:

"aiops-ir-core-esarchiving" and "aiops-ir-core-archiving"

This could happen if Kafka renews its certificate.

Diagnosing The Problem

Run the following oc command and find the pods by looking for the related IP addresses listed in the Kafka authentication errors:
oc get po -o wide

Resolving The Problem

To resolve this issue, first complete all the steps highlighted in the following technote:
 
Once completed, run the following command to restart the archiving components-related pods:
For example:
oc delete aiops-ir-core-archiving-78c995956f-77wk2
oc delete aiops-ir-core-esarchiving-67d7bfd9cd-b698s

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRS1M","label":"IBM Cloud Pak for Watson AIOps"},"ARM Category":[{"code":"a8m3p000000PCGIAA4","label":"Watson AIOps-\u003EAI Manager-\u003EData-\u003EDatastore-\u003EKafka"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
01 March 2023

UID

ibm16956920