Setting up the Kafka server
Before you use the Kafka connector, Kafka server must be configured either as standalone or in a cluster environment.
Kafka connector works with Apache distribution of Kafka at he version 0.10.0.0. Kafka connector works with both offline as well as streaming messages. It is expected for Kafka connector to be configured with the Apache zookeeper as a synchronization service.
- Single node single broker cluster
- Single node multi broker cluster
- Multiple node multiple broker cluster
Modes | Description |
---|---|
Single node single broker cluster | In this configuration one or more instance of zookeeper and only one instance of Kafka is
running on a host which can be accessible from the machine where Information Server engine tier is
installed. This setup essentially means a Kafka cluster of size 1. If the Kafka Connector has to
be used for this type of configuration, the hostname of the machine where Kafka Server is running is
required along with the port number on which the Kafka server listens. To connect to a Kafka server,
the configuration property bootstrap.servers is required which should contain
the In this configuration the value for bootstrap.servers
property would be a single |
Single node multi broker cluster | In this configuration, on the same machine (node), one or more instances of zookeeper and
more than one Kafka broker should be running. Kafka brokers are uniquely identified by the
broker.id property. For Kafka Connector to establish a connection to the
Kafka server, the hostname along with the list of port numbers should be provided. For example, for
the bootstrap.servers property, the value would be
hostname:9092 , hostname:9093 (that is all the ports on the same
server where Kafka service would be running).This configuration guarantees that the messages that would be read from topics are not lost even if the port has gone down the messages, and ensures fault tolerance to a certain extent. |
Multiple node multiple broker cluster | In this configuration, Kafka server should be installed on multiple hosts (nodes) and again
multiple brokers can run on each node. The connection to Kafka server requires the hostname: port
values for all the nodes where Kafka is running. For example the bootstrap.servers property value should be a list of host/port pairs which would be used for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified for bootstrapping. Since these servers are only used for the initial connection, to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, if a server is experiencing a down time). |
When the Kafka connector is configured as a source stage, the messages are read from Kafka which is equivalent to implementing Kafka connector as consumer object. When Kafka connector is configured in the target stage, the messages are written into Kafka, which is equivalent to implementing Kafka connector as a Producer object. Kafka connector is available in the location //Stagetypes//Parallel//Real Time//Kafka connector