Kafka client configuration

New in 20.0.2 You can configure the Kafka clients of IBM Cloud Pak® for Automation components on Kubernetes.

Use cases

The configuration of Kafka clients in the shared_configuration.kafka_configuration section provides the minimal set of parameters that allow IBM Business Automation Insights and other IBM Cloud Pak for Automation components to consume and produce Kafka events. For these components to connect to a Kafka server, you must pass the bootstrap servers that are set in the Kafka server configuration.

If the Kafka server is configured without user authentication and security, you can set all other parameters to their default values or skip them.
If, as a typical use case, you configure Kafka servers with authentication and security, all parameters are mandatory. Set them to the same values as the authentication and security parameters of the Kafka server.

You can further configure the Kafka clients of IBM Cloud Pak for Automation products. For example, you can increase the Kafka producer buffer size for the IBM Operational Decision Manager event emitter to cope with its particular needs. The parameters that you can configure for each component are documented in Certified Kubernetes configuration parameters.

Parameters

In evaluation deployments, Kafka parameters are automatically added to the shared_configuration section of the IBM Cloud Pak for Automation custom resource. In other types of deployments, if you use IBM Event Streams 10.x, a script automates this step.

For 20.0.2 You can use this script to automatically configure the IBM Cloud Pak for Automation custom resource for an existing IBM Event Streams 10.0 instance.
New in 20.0.3 You can use this script to automatically configure the IBM Cloud Pak for Automation custom resource for an existing IBM Event Streams 10.1 instance.

The following table presents the parameters of the shared_configuration.kafka_configuration section of your IBM Cloud Pak for Automation cluster custom resource (CR).

Table 1. Shared parameters for Kafka clients
Name	Description	Default value
bootstrap_servers	Apache Kafka bootstrap servers	None
New in 20.0.3 schema_registry_url	The URL of the schema registry of the Kafka server. If the Business Automation Insights processor for custom events is configured, set this parameter to the URL of the schema registry. Otherwise, you can leave it empty.	None
New in 20.0.3 schema_registry_type	The type of the schema registry of the Kafka server. Valid values: APICURIO, EVENT_STREAMS, and CONFLUENT.	APICURIO
security_protocol	The value of the Apache Kafka security.protocol property	SASL_SSL
sasl_mechanism	The value of the Apache Kafka sasl.mechanism property	PLAIN
connection_secret_name	The name of a secret that is already deployed to Kubernetes and that contains the keys as documented in Table 2. If the Kafka server does not require authentication and does not use SSL communications, you can omit this parameter.	None

Keys

The Kubernetes secret for a Kafka client configuration must contain the following keys, such as in this example.

apiVersion: 
v1 kind: Secret 
metadata: 
  name: kafka-connection-secret 
data: 
  kafka-server-certificate: <base64-encoded CA certificate in PEM format for secure communication with the Kafka server>
  kafka-username: <base64-encoded Kafka username>
  kafka-password: <base64-encoded Kafka password>

Table 2. Keys in the Kubernetes secret
Key	Description
kafka-server-certificate	The Apache Kafka server certificate for SSL communications. Leave the value of this key empty if the SSL protocol is not used.
kafka-username	The username to the Apache Kafka server. Leave the value of this key empty if no user authentication is configured in the Kafka server.
kafka-password	The password to the Apache Kafka server. Leave the value of this key empty if no user authentication is configured in the Kafka server.