Apache Kafka is a popular open source data streaming platform providing high-throughput data pipelines. As such, Instana has included comprehensive Kafka Monitoring in our Infrastructure and Application Monitoring solution. Instana discovers all Kafka instances present in the environment, understands the dependencies and interactions between the cluster and its servers, and automatically begins monitoring the health and performance of each instance.
To help DevOps optimize their Kafka clusters, Instana applies automation and artificial intelligence to Kafka performance and health monitoring, including predictive problem detection and AI-assisted root cause analysis. Instana reduces your need for explicit Kafka expertise within operations.
14 days, no credit card, full version
Instana monitors Kafka’s Key Performance Indicators (KPIs) at the individual component and application level for each instance. The Kafka KPIs are used for real-time health monitoring of each instance and cluster. The Service KPIs are used to understand Kafka’s performance within the overall environment.
Instana AI uses advanced statistical analysis, a set of specific Knowledge Base rules, and applied machine learning to determine the real-time health status of each instance and cluster.
Instana’s Apache Kafka sensor gathers all critical metrics to determine the health of each Kafka instance and the performance of all data pipelines on the cluster. Tracking Kafka configuration data allows Instana to monitor changes to the Kafka instances and correlate those changes to any overall performance issues.
The Kafka performance and configuration is summarized in a single dashboard that combines all information about setup, infrastructure, service levels and transactions. The dashboard allows IT Ops and DevOps to see all relevant Kafka data in a single place for easy-problem solving and performance optimization.
Kafka performance monitoring centers around metrics relevant to its interactions with the data pipelines and dependent applications that live in and around the Kafka cluster. Instana automatically identifies and collects all relevant metrics. Below are some sample screenshots of an Instana Kafka dashboard:
Instana’s Kafka Monitoring includes three types of data: Tracked Configuration, Performance Metrics, and Health Signatures:
Tracked Configuration
- Version
Performance Metrics
- Broker Traffic
- Request Latency vs. Throughput
- Broker Failures
- Broker State Metrics
- Average Idle Time
- Partitions
Log Flushing
Health Signatures
- Active Controller Count
- isrShrinks, isrExpansions
- Leader Elections
- Unclean Leader Elections
- Network Processor Low Idle
- Offline Partitions
- Request Handler Low Idle
Under Replicated Partitions
Thanks to the open source nature of Apache Kafka, there are numerous solutions to operate Apache Kafka in cloud and on-premise situations. Instana supports most common deployment strategies, including
- Installation on dedicated virtual or physical hardware (Windows, Linux)
- Installation in on-premise or managed Kubernetes environments
- Installation in on-premise or managed OpenShift environments
- Installation in on-premise or managed Pivotal Cloud Foundry environments
- Hosted Kafka services Amazon MSK (Amazon Managed Streaming for Apache Kafka)
Furthermore, Instana supports multiple distributions of Apache Kafka, including
- Vanilla Apache Kafka
- Confluent Kafka
- Cloudera Kafka
- Amazon MSK