Inference Service fail to re-establish a connection to Kafka
When Kafka is restarted, the inference service fails to re-establish a connection to the Kafka brokers.
Problem
Messages similar to the below are seen in the
custom_resource-ibm-hdm-analytics-dev-inferenceservice-<instance> pod's logs
when this failure
occurs:
WARN [2020-04-21 15:02:54,079] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Connection to node 0 (/10.254.23.10:9092) could not be established. Broker may not be available.
WARN [2020-04-21 15:02:57,151] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Connection to node 2 (/10.254.3.51:9092) could not be established. Broker may not be available.
WARN [2020-04-21 15:03:00,223] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Connection to node 1 (/10.254.27.165:9092) could not be established. Broker may not be available.
Where
custom_resource is the name of your
deployment, as specified by the value used for name (OLM UI install), or
metadata.name in noi.ibm.com_noihybrids_cr.yaml
/noi.ibm.com_nois_cr.yaml (CLI install).Resolution
To resolve this issue, restart the inference service pod, so that it reconnects to the new Kafka instance.oc delete pod custom_resource-ibm-hdm-analytics-dev-inferenceservice-<instance>
Where custom_resource is the name of your deployment, as specified by the value used for name (OLM UI install), or metadata.name in noi.ibm.com_noihybrids_cr.yaml /noi.ibm.com_nois_cr.yaml (CLI install).