Kafka
The Message Bus Probe can be configured to integrate with a Kafka server to consume events.
The Message Bus Probe connects to the Kafka server using the Kafka transport. This enables the probe support the Kafka Client version 2.3.1 and Zookeeper version 3.4.14.
Check the Apache Kafka compatibility matrix for the support of the target system with respect to the dependency.
Configuring the Message Bus Probe to consume new events from Kafka
- message_bus_kafka.props
- message_bus_kafka.rules
- kafkaTransport.properties
- kafkaConnectionProperties.json
- kafkaClient.properties
To configure the Message Bus Probe to consume new events from Kafka, use the following steps:
- Install/update the Message Bus Probe using IBM Installation Manager, see Installing probes.
- Edit the probe configuration in the following probe properties
file:
$OMNIHOME/probes/<arch>/message_bus_kafka.props
Where <arch> is the architecture directory, for example linux2x86
- Update the following property values with the appropriate path:
Manager : 'Kafka' MessageLog : '$OMNIHOME/log/message_bus_kafka.log' PropsFile : '$OMNIHOME/probes/linux2x86/message_bus_kafka.props' RulesFile : '$OMNIHOME/probes/linux2x86/message_bus_kafka.rules' TransportType : 'KAFKA' TransportFile : '$OMNIHOME/java/conf/kafkaTransport.properties' TransformerFile : '$OMNIHOME/probes/linux2x86/message_bus_parser_config.json' MessagePayload : 'JSON'
- Configure the Kafka transport properties.
- Edit the Kafka transport configuration in the following transport
properties file:
$OMNIHOME/probes/java/conf/kafkaTransport.properties
- Update the following property value with the appropriate path:
kafkaClientMode=CONSUMER connectionPropertiesFile=$OMNIHOME/java/conf/kafkaConnectionProperties.json
- Edit the Kafka transport configuration in the following transport
properties file:
- For descriptions of the Kafka transport properties, see the Configuring the Kafka transport.
- Configure the Kafka connection properties.
Kafka connection properties are defined in the kafkaConnectionProperties.json file. This file contains the following properties:
{ "zookeeper_client" : { "target" : "", "properties" : "", "java_sys_props" : "", "topic_watch": true, "broker_watch": true }, "brokers" : "", "topics": "", "kafka_client" : { "properties" : "", "java_sys_props" : "" } }
The ZooKeeper event access is required.
- Within the sample configuration file supplied with the probe,
update the path to the ZooKeeper client properties file.
"zookeeper_client" : { "target" : "localhost:2181", "properties" : "<Path to zookeeper client properties file>", "java_sys_props" : "", "topic_watch": true, "broker_watch": true },
- Update the path to the Kafka client properties file:
"brokers" : "PLAINTEXT://localhost:9092", "topics": "topicABC,topicXYZ", "kafka_client" : { "properties" : "<omnihome_path>/java/conf/kafkaClient.properties", "java_sys_props" : "" }
- Within the sample configuration file supplied with the probe,
update the path to the ZooKeeper client properties file.
- Specify additional configuration to use with SASL authentication,
if SASL authentication is required.Kafka brokers supports client authentication using SASL. Additional configuration settings are required to authenticate with SASL.
- Enable the following properties in the Kafka connection properties
file (kafkaConnectionProperties.json)
"java_sys_props" : "<Path to a text file containing java system properties configuration>"
Example:
"java_sys_props" : "java.security.auth.login.config=C:\\IBM\\Tivoli\\Netcool\\omnibus\\java\\conf\\java_sys_prop.conf"
Create and enable the following properties in the Java system properties file (java_sys_prop.conf):
Example:
KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="kafka" username="wfE7hGteeNl4i9JJ" password="xtPqsWiPXN4cSwi6h2BE4GbaJ3uheal2"; };
- Enable and set the following properties in the Kafka client properties
file (kafkaClient.properties)
security.protocol ssl.enabled.protocols ssl.keystore.location ssl.keystore.password ssl.keystore.type Included when trust stores are in use. ssl.truststore.location ssl.truststore.password ssl.truststore.type
Example:
security.protocol=SASL_SSL ssl.enabled.protocols=TLSv1.2 ssl.keystore.location=C:\\Programs\\ibm-java-sdk-80-win-x86_64\\sdk \\jre\\lib\\security\\cacerts ssl.keystore.password=changeit ssl.keystore.type=JKS ssl.truststore.location=C:\\Programs\\ibm-java-sdk-80-win-x86_64\\sdk \\jre\\lib\\security\\cacerts ssl.truststore.password=changeit ssl.truststore.type=JKS
- Enable the following properties in the Kafka connection properties
file (kafkaConnectionProperties.json)
- For descriptions of the Kafka connection properties, see the Configuring the Kafka transport.
- Start the probe with the propsfile option to
specify the Kafka properties file using the following command:
$OMNIHOME/probes/nco_p_message_bus -propsfile $OMNIHOME/probes/linux2x86/message_bus_kafka.props
- Verify that the probe is running.