Connecting to a Kafka schema registry

If you're already using a schema registry, you can use App Connect to connect to a registry to retrieve schemas so that you can concentrate on the message transfer rather than the message's data infrastructure.

Availability: This feature is available in an instance of App Connect Designer 11.0.0.11-r2 or later and App Connect Enterprise as a Service

A schema defines the structure, including the metadata, of the messages that pass between Kafka producer and consumer applications. These messages often require complex decoding as they pass between applications and the metadata can occupy lots of space on the Kafka brokers. A schema makes encoding and decoding data more efficient because all the messages adhere to a predefined structure.

A schema registry provides a centralized repository for schemas and metadata. It can contain multiple schemas with numerous versions because schemas can evolve over time as you update them to add new structures. If you're already using a schema registry, you can use App Connect to connect to a registry to retrieve schemas so that you can concentrate on the message transfer rather than the message's data infrastructure. You can interact with the schema registry as well as composing AVRO or JSON messages. The generated payload wire format is: 
<MAGIC_BYTE><SCHEMA_ID><SERIALIZED AVRO/JSON DATA>
where:
  • <MAGIC_BYTE> is 0 by default
  • <SCHEMA_ID> represents the Schema ID (confluent) or Global ID (Apicurio) and is mapped to bytes 1 - 4.
  • <SERIALIZED AVRO/JSON DATA> represents the actual serialized AVRO or JSON data and is mapped to the 5th byte onwards.

App Connect provides you with a simple user interface to connect to a schema registry so that you can avoid the complexity of dealing with data serialization and decoding between Kafka producer and consumer applications.

Note: App Connect supports Apicurio and Confluent schema registries in JSON or AVRO wire formats.

Connecting to a Kafka schema registry

In addition to the credentials required to connect to Kafka as described in Connecting to Kafka, you need to supply the following details to connect to a schema registry.
Schema registry type
Select the schema registry that you want to connect to. App Connect supports Apicurio and Confluent schema registries. This field defaults to Confluent if you provide a schema registry API URL and do not specify a registry type.
Restriction: Apicurio registry version 2.x and the Event Streams schema registry are not supported.
Schema registry REST API URL
The REST API URL of the schema registry in the format http[s]://<hostname|ip address>:<port>. For example: https://192.168.0.1:9001.
Schema registry username
The username of the schema registry.
Schema registry password
The password of the schema registry.
Schema registry CA certificate
The certificate of the schema registry certificate authority (CA) in PEM format.

Consuming AVRO/JSON serialized messages

To consume messages from a specific schema registry, you need to use the Kafka new message event node. This enables you to pull in the specified registry so that you can map to it from an action node.

  1. From the App Connect Designer dashboard, click Create > Event-driven flow (or New > Event-driven flow in earlier versions) to open the flow editor.
  2. Enter a name that identifies the purpose of your flow.
  3. In the Applications tab, select Kafka.
  4. Select the Kafka account using the schema registry that you want to work with and then click New message. The event node is added to the flow and you are prompted to select the Kafka topic you are listening for messages on.
  5. Select the Kafka topic that you want to work with.
  6. Click Select output schema. The Schema settings window is displayed and you are prompted to select the schema type.
  7. Select the schema type. The options available are AVRO or JSON.
  8. Select the subject name and the schema version.
  9. Click Apply.
The output schema for the subject you selected is displayed and is available for you to map to the target application.
Output schema display

Generating and sending AVRO/JSON serialized messages

Custom headers in Kafka actions are a set of name:value pairs that are associated with a Kafka message as it is published and then retrieved when the message is retrieved. Kafka custom header properties enable you to interact with a schema registry and add metadata to Kafka messages, which can then be used during message processing. For example, the header properties can carry information about the format of the data, such as the schema subject name.

Kafka send message custom header

To send a message based on a specific schema format, you need to use the Kafka send message action. The following method works for both event driven flows and flows for an API.

  1. In the App Connect Designer dashboard, create an event driven flow or a flow for an API.
  2. For the action node, select Kafka > Send message.
  3. Select the Kafka topic that you want to work with.
  4. Click the drop-down arrow in the *Payload field.
    Drop down arrown in the payload field
  5. You are prompted to select a schema format based on the schema registry you logged in with. The following options are available:
    • AVRO
    • JSON
  6. You are then prompted to select the subject name of the schema and the version.
    AVRO schema settings window
  7. Click Apply.

The schema is pulled and presented in the format you selected in Step 5. You can now compose a message in your chosen schema format.

The schema registry displayed in JSON format