Kafka transactionally consistent consumer

You can recreate the order of operations in source transactions across multiple Kafka topics and partitions and consume Kafka records that are free of duplicates by including the Kafka transactionally consistent consumer library in your Java applications.

When you include this library, the application makes calls to request records for a topic or set of topics that are part of a CDC subscription. The Kafka transactionally consistent consumer provides the records as well as information on the topic, partition, offset, operation ID, and operation bookmark for each record. The operation bookmark can be used to position the application that calls the Kafka transactionally consistent consumer. The application can then process these records in order and duplicate free.

To begin using the Kafka transactionally consistent consumer, first create a new instance of the CDC Replication Engine for Kafka. On that instance, change the system parameter kafka_bookmark_storage_type from the default value POINTBASE to the value COMMITSTREAMTOPIC. Create new subscriptions on that instance, or copy or promote subscriptions from the original instance. You can use this new instance with your Kafka transactionally consistent consumer applications.

Note: To use the Kafka transactionally consistent consumer with a subscription, you must not compact topics or redistribute topics that are part of that subscription.

The CDC Replication Engine for Kafka provides sample applications that you can extend or modify to suit your environment.