Technical Blog Post
Abstract
Log Analysis - Basic Maintenance Tasks of a Kafka Cluster
Body
If you have plans to expand the Kafka cluster of your Log Analysis setup or had already done so, it is always good to know how to manage your Kafka cluster properly.
There are certainly a few important tasks when it comes to managing a Kafka cluster:
(1) How many brokers are connected?
(2) What are the topics available?
(3) How do you create new topic?
(4) How do you check the content of an existing topic?
While management tasks are not limited to those listed above, but we will focus on those as a starter.
(1) How many brokers are connected?
To know how many Kafka brokers are currently connected, it is quite simple.
[danielyeap@kafka1 bin]$ ./zookeeper-server-start.sh -daemon ../config/zookeeper.properties ... ls / [0, 1] ** This means 2 Kafka brokers are connected with ID 0 and 1 respectively |
(2) What are the topics available?
To examine what are the topics available within the Kafka cluster, you can perform the following.
[danielyeap@kafka1 bin]$ ./zookeeper-shell.sh kafka1:2181 ls /brokers/topics |
(3) How do you create new topic?
To create a new topic, you can use the "kafka-topics.sh" tool provided with Kafka.
[danielyeap@kafka1 bin]$ pwd /opt/kafka_2.11-0.10.0.0/bin [danielyeap@kafka1 bin]$ ls -l *topic* -rwxr-xr-x 1 danielyeap danielyeap 863 May 18 2016 kafka-topics.sh [danielyeap@kafka1 bin]$ ./kafka-topics.sh --create --topic blogtest --partitions 2 --zookeeper kafka1:2181 --replication-factor 1 ... Created topic "blogtest". [danielyeap@kafka1 bin]$ ./kafka-topics.sh --list --zookeeper kafka1:2181 ... __consumer_offsets aixerr blogtest iis oracle oracle2 scala_access_log test web_access_log winevt [danielyeap@kafka1 bin]$ |
(4) How do you check the content of an existing topic?
To view the content of a particular topic, you can utilize the "kafka-console-consumer.sh" tool provided.
[danielyeap@kafka1 bin]$ ./kafka-console-consumer.sh --topic test --from-beginning --zookeeper kafka1:2181 OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Hello World! I am fine! How are you? Bye! |
Have fun Kafka-ing!
UID
ibm11081665