Monitoring Kafka

Support information

To make sure that the Kafka sensor is compatible with your current setup, verify the following support information sections:

Configuration

The Instana agent automatically detects the running Kafka agent. Therefore, no configuration is required.

Instana collects the first 400 topics that are sorted by topic name.

If you need to filter topics, you can configure it in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml as shown in the following example:

com.instana.plugin.kafka:
  ...
  topicsRegex: '<OPTIONAL_REGEX_HERE>'
  brokerPropertiesFilePath: '/path/to/server.properties'
  collectLagData: '' # true or false. The default value is true

• topicsRegex: Optional regular expression to select up to 400 topics by name. If the value is empty or does not exist, Instana collects the first 400 topics that are sorted by name.
• brokerPropertiesFilePath: The path to the broker server.properties file that the agent uses to obtain information about the broker network and security protocol settings.
• collectLagData: Flag that enables or disables lag data collection (enabled by default).

If the path to the broker properties is not specified, the agent tries to find server.properties in the following places:

• Kafka broker process arguments
• KAFKA_SERVER_PROPERTIES environment variable
• Using the predefined paths: /path_to_kafka_home/config/server.properties or /path_to_kafka_home/etc/kafka/server.properties for Confluent Kafka.

The Agent uses /opt/kafka/config/server.properties as a default path when the path to server.properties is not found in any of the previously mentioned ways.

SSL TLS support

If your Kafka broker instance requires SSL client connections, you need to configure the Instana agent via <agent_install_dir>/etc/instana/configuration.yaml to enable collecting Consumer lag metrics as shown in the following example:

com.instana.plugin.kafka:
  ...
  sslTrustStore: '/path/to/truststore.jks'
  sslTrustStorePassword: 'kafkaTsPassword'
  sslKeyStore: '/path/to/sslKeyStoreFile.jks'
  sslKeyStorePassword: 'kafkaKsPassword'

Make sure that Keys are in the Java keystore format (JKS). Use the keytool to create keys.

This action enables the Instana agent to connect to the Kafka broker via SSL and collect Consumer group lag metrics.

JMX authentication support

If your JMX authentication is enabled for your Kafka, you need to configure the Instana agent by using <agent_install_dir>/etc/instana/configuration.yaml to authenticate your JMX as shown in the following example:

com.instana.plugin.kafka:
  jmxUsername: ''
  jmxPassword: ''
  jmxPort: '' # default jmx port is 1099

If your JMX is not secured, Instana begins monitoring by connecting to your default 1099 JMX port.

Kafka node - metrics collection

Kafka node metrics collection gathers and analyzes data about the performance and health of individual nodes within a Kafka cluster.

Kafka cluster - metrics collection

Kafka cluster metrics collection gathers and analyzes data about the performance and health of the entire Apache Kafka cluster, rather than individual nodes.

Troubleshooting