KafkaRead node

Use the KafkaRead node in a message flow to read a single message from a Kafka topic. The message to be read is identified by the topic name, partition number, and message offset properties, which you can set in the node or as Local Environment overrides.

This topic contains the following sections:

Purpose

You can use a KafkaRead node in a message flow to read a specific message from a Kafka topic, which can be useful for error processing in a Kafka flow. For example, if the processing of a message fails in a Kafka consumer message flow, you can use the KafkaRead node to reprocess that individual message. You can also use a KafkaRead node in conjunction with a Timer node, to control the processing of a range of messages in a partition on a topic. For more information, see Reading an individual message from a Kafka topic.

For information about the supported versions of Kafka, see IBM App Connect Enterprise system requirements. For more information about Kafka version compatibility, see the Apache Kafka documentation.

The KafkaRead node handles messages in the following message domains:

  • DFDL
  • XMLNSC
  • JSON
  • BLOB
  • MIME
  • XMLNS
  • MRM

The KafkaRead node is contained in the Kafka drawer of the palette, and is represented in the IBM App Connect Enterprise Toolkit by the following icon:

KafkaRead node icon

Using the KafkaRead node in a message flow

Use the KafkaRead node in a message flow to read an individual message that was published to a topic on a Kafka server. For information about how to use the node, see Reading an individual message from a Kafka topic.

Terminals and properties

The KafkaRead node terminals are described in the following table.

Terminal Description
Failure The output terminal to which the message is routed if a failure is detected during processing in the node.
Out The output terminal to which the message is routed if it represents successful completion of the Kafka request, and if further processing is required within this message flow.
No match The output terminal to which the input message is routed if no message is available on the topic. The output message that is propagated to the No match terminal is constructed from the input message.

The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk on the panel if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the BAR file to deploy it).

The KafkaRead node Description properties are described in the following table.

Property M C Default Description
Node name No No The node type, KafkaRead The name of the node.
Short description No No   A brief description of the node.
Long description No No   Text that describes the purpose of the node in the message flow.

The KafkaRead node Basic properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Topic name Yes Yes   The name of the Kafka topic containing the message that you want to read. Only one topic name can be specified. The name can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). topicName
Bootstrap servers Yes Yes   A list of host/port pairs, separated by spaces, to use for establishing the initial connection to the Kafka cluster. bootstrapServers
Partition number Yes Yes 0 The number of the Kafka partition for the topic that you want to use (between 0-255). The default value is 0. partitionNumber
Message offset Yes Yes 0 The offset position of the message to be read. The default value is 0. offset
Action when message unavailable Yes Yes no match The action that should be taken if the message at the specified offset is unavailable. Possible values are:
  • no match - propagate the input message to the No match terminal. This is the default.
  • earliest - read the message with the earliest offset for the topic partition.
  • latest - read the message with the latest offset for the topic partition.
 notFoundAction
Client ID No Yes   The client name to be used when connecting to Kafka. This value can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). clientId
Add server name suffix to client ID No Yes Yes If selected, this property suffixes the provided client ID with either -integrationServerName (for an independent integration server) or -integrationNodeName-integrationServerName (for an integration server that is managed by an integration node). This option is selected by default. useClientIdSuffix
Timeout (s) Yes Yes 5 The timeout (in seconds) for the KafkaRead node to get the message. If the message has not been found within that time, the action taken is dependent on the value set by the notFoundAction property. The default is 5 seconds. connectionTimeout

The KafkaRead node Advanced properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Properties file No Yes null Full path name of a kafka.properties file to set Kafka client properties. propertiesFilePath

The KafkaRead node Security properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Security identifier No Yes null Identifier that is used to look up the user ID and password from the App Connect Enterprise vault, which was defined by using the mqsicredentials command.

If this property is left blank, the default kafka credential (defined by using the mqsicredentials command) is used. If a default credential has not been defined, the security ID KAFKA::KAFKA::{integrationServerName} is used.

 securityIdentity
Security protocol Yes Yes PLAINTEXT The protocol to be used when communicating with the Kafka server. Valid values are:
  • PLAINTEXT
  • SSL
  • SASL_PLAINTEXT
  • SASL_SSL

If you are using IBM Event Streams, this property must be set to SASL_SSL.

If either SASL_PLAINTEXT or SASL_SSL is selected, you must configure the user ID and password that will be used to authenticate with the Kafka server. You can configure these credentials by using either the mqsisetdbparms command or the mqsicredentials command. For more information, see Configuring security credentials for connecting to Kafka.

 securityProtocol
SSL protocol Yes Yes TLSv1_2 The SSL protocol that is used when the selected value for the Security protocol property is either SSL or SASL_SSL. You can select one of the following values from the editable list, or you can specify an alternative value:
  • TLSv1
  • TLSv1.1
  • TLSv1.2
 sslProtocol

The KafkaRead node Policy properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Policy No Yes null Name of a Kafka policy to be used to override node properties; for example, {PolicyProject}:kafkaPolicy policyUrl
The KafkaRead node Result properties are described in the following table.
Property M C Default Description
Output data location No No $OutputRoot

The location on the output message tree to which the KafkaRead node sends output. The input root is first copied to the output root, and then the result data is copied to the location on the output tree specified by Output data location. The default value, $OutputRoot, replaces the copied message tree with the result data, and propagates none of the input message.

For example, a value of $OutputRoot/XMLNS/MyData inserts the result message data into the output message tree, by merging or overwriting the message tree.

You can specify Output data location as an ESQL or an XPATH expression.

See Combining a result message with an input message when fetching data from external systems.
Result data location No No $ResultRoot The location in the message received from the Kafka queue that is copied to the Output data location field in the output message.

For example, the default, $ResultRoot, inserts the result message into the output tree at the location specified by the Output data location.

You can specify Result data location as an ESQL or an XPATH expression.

See Combining a result message with an input message when fetching data from external systems.
Copy local environment No No Selected
This property specifies whether the local environment is copied to the output message.
  • If Copy local environment is selected, a new copy of the local environment is created in the tree, and it is populated with the contents of the local environment from the preceding node. Therefore, if a node changes the local environment, those changes are not visible to the upstream nodes in the event of a message rollback, because they have their own copies. This value is the default.
  • If Copy local environment is cleared, the node does not generate its own copy of the local environment, but uses the local environment that is passed to it by the preceding node. Therefore, if a node changes the local environment, the changes are reflected by the upstream nodes in the event of a message rollback.

The KafkaRead node Input Message Parsing properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Message domain No No BLOB The parser domain that is used to parse the input message. If the field is blank, the default is BLOB.  
Message model No No Cleared The name or location of the message model schema file in which the message is defined.

When you click Browse, you see a list of available message model schema files for the selected Message domain.

 
Message No No Cleared The name or location of the message root within your message model schema file. This list is populated with all available messages that are defined in the Message model that you selected.  
Physical format No No Cleared The name of the physical format of the message. If you are using the MRM or IDOC parser, select the physical format of the incoming message from the list. This list includes all the physical formats that you have defined for the selected message model. If you set the Message domain property to DataObject, you can set this property to XML or SAP ALE IDoc. Set this property to SAP ALE IDoc when you have to parse a bit stream from an external source and generate a message tree.  

The KafkaRead node Parser Options properties are described in the following table.

Property M C Default Description
Parse timing No No On Demand This property controls when an input message is parsed. Valid values are On Demand, Immediate, and Complete.

For a full description of this property, see Parsing on demand.
Build tree using XML schema data types No No Cleared This property controls whether the syntax elements in the message tree have data types taken from the XML schema.
Use XMLNSC compact parser for XMLNS domain No No Cleared This property specifies whether the XMLNSC Compact Parser is used for messages in the XMLNS domain. If you set this property, the message data is displayed under XMLNSC in nodes that are connected to the output terminal when the input MQRFH2 header is XMLNS.
Retain mixed content No No Cleared This property controls whether the XMLNSC parser creates elements in the message tree for mixed text in an input message. If you select the check box, elements are created for mixed text. If you clear the check box, mixed text is ignored and no elements are created.
Retain comments No No Cleared This property specifies whether the XMLNSC parser creates elements in the message tree for comments in an input message. If you select the check box, elements are created for comments. If you clear the check box, comments are ignored and no elements are created.
Retain processing instructions No No Cleared This property controls whether the XMLNSC parser creates elements in the message tree for processing instructions in an input message. If you select the check box, elements are created for processing instructions. If you clear the check box, processing instructions are ignored and no elements are created.
Opaque elements No No Blank This property is used to specify a list of elements in the input message that are to be opaquely parsed by the XMLNSC parser. Opaque parsing is performed only if validation is not enabled (that is, if Validate is None); entries that are specified in Opaque Elements are ignored if validation is enabled.

The KafkaRead node Validation properties are described in the following table.

For a full description of these properties see Validation properties.

Property M C Default Description mqsiapplybaroverride command property
Validate No Yes None This property controls whether validation takes place. Valid values are None, Content and Value, and Content. validateMaster
Validation failure action No Yes Exception This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List. validateFailureAction
The Monitoring properties of the node are described in the following table.
Property M C Default Description
Events No No None Events that you have defined for the node are displayed on this tab. By default, no monitoring events are defined on any node in a message flow. Use Add, Edit, and Delete to create, change or delete monitoring events for the node.

You can enable and disable events that are shown here by selecting or clearing the Enabled check box.