IT33662: ENABLE.AUTO.COMMIT AND ASYNC OFFSET COMMIT OPTIONS FOR KAFKA CONSUMER NODE.

APAR status

• Closed as requirement.

Error description

• This APAR introduces following features for KafkaConsumer node.
  
  1) If the "enable.auto.commit" property is set to "true" in the
  Kafka properties file, this setting is now honoured and takes
  precedence over the "Commit message offset in kafka" setting on
  the KafkaConsumer node.
  
  2) The "Commit message offset in kafka" property on the
  KafkaConsumer node updates the position of the kafka consumer
  saved in the kafka Server as each message is processed by the
  message flow. The KafkaConsumer node waits for the save of
  consumer position to be completed before the message is
  processed by the message flow ensuring an at-most-once level of
  message reliability. The action of waiting for the response to
  the saving of the consumer offset introduces a significant delay
  before the message is processed by the message flow and in turn
  the maximum throughput of the message flow. This APAR optionally
  allows the saving of the consumer offset to be performed
  asynchronously, which means that the request to save the
  consumer offset is still made but we do not wait for the
  response before the message is processed by the message flow
  (this style of operation is called asynchronously committing the
  offset). Setting the environment variable
  "MQSI_KAFKA_USE_SYNC_OFFSET_COMMIT=FALSE" in the environment of
  the integration server enables this asynchronous mode of
  operation.
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  All users of IBM AppConnect Enterprise V11.0 and IBM Integration
  Bus V10.0 using the KafkaConsumer node.
  
  
  Platforms affected:
  z/OS, MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  This APAR introduces following features for KafkaConsumer node.
  
  1) If the "enable.auto.commit" property is set to "true" in the
  Kafka properties file, this setting is now honoured and takes
  precedence over the "Commit message offset in kafka" setting on
  the KafkaConsumer node.
  
  2) The "Commit message offset in kafka" property on the
  KafkaConsumer node updates the position of the kafka consumer
  saved in the kafka Server as each message is processed by the
  message flow. The KafkaConsumer node waits for the save of
  consumer position to be completed before the message is
  processed by the message flow ensuring an at-most-once level of
  message reliability. The action of waiting for the response to
  the saving of the consumer offset introduces a significant delay
  before the message is processed by the message flow and in turn
  the maximum throughput of the message flow. This APAR optionally
  allows the saving of the consumer offset to be performed
  asynchronously, which means that the request to save the
  consumer offset is still made but we do not wait for the
  response before the message is processed by the message flow
  (this style of operation is called asynchronously committing the
  offset). IBM AppConnect Enterprise V11.0 introduced a new node
  property 'Wait for message offset commit to complete' in the
  KafkaConsumer node to control asynchronous commit of message
  offsets. Setting the environment variable
  "MQSI_KAFKA_USE_SYNC_OFFSET_COMMIT=FALSE" in the environment of
  the integration server enables this asynchronous mode of
  operation in IBM Integration Bus V10.0.
  

Problem conclusion

• The product now honors "enable.auto.commit" property set in the
  Kafka properties file and also provides option to enable
  asynchronous commit of message offsets.
  

Temporary fix

Comments

• New properties introduced for KafaConsumer node
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels