IBM Support

How to clean up log collection for kafka in shared open source service in Information Server?

How To


Summary

Recent version IBM InfoSphere Information Server installs Apache Kafka, Solr, and ZooKeeper in <ISHOME>/shared-open-source directory in Services tier.
Kafka in shared open source uses "compact" log cleanup policy for the __consumer_offsets topic by installation default, which can result in many log files for the topic and filling up the disk space.

Objective

Change Kafka log retention policy for __consumer_offsets topic to "delete".

Environment

Information Server 11.5 with Governance Rollup 1 installed.
Information Server 11.5 fix pack 2.
Information Server 11.7.

Steps

1. Disable Kafka Log Cleaner thread.

In Services tier, open <ISHOME>/shared-open-source/kafka/conf/server1.properties file in a text editor. For example, open /opt/IBM/InformationServer/shared-open-source/kafka/conf/server1.properties in Unix or Linux, C:\IBM\InformationServer\shared-open-source\kafka\conf\server1.properties in Windows.

If log.cleaner.enable property is set to "true", change it to "false" like following, save and exit text editor.

log.cleaner.enable=false

Note: Log Cleaner thread is required for "compact" cleanup policy. Set it to "true" if you need to use "compact" policy for some topics. See log.cleaner.enable

2. Change the log cleanup policy for __consumer_offsets topic to "delete".

In Services tier, make sure Kafka service is up and running, and run following command.

## In Unix / Linux
cd /opt/IBM/InformationServer/shared-open-source/kafka/install/bin
./kafka-topics.sh --zookeeper localhost:52181 --topic __consumer_offsets --alter --config cleanup.policy=delete

# OR
./kafka-configs.sh --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete

## In Windows CMD window
cd C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows
kafka-topics.bat --zookeeper localhost:52181 --topic __consumer_offsets --alter --config cleanup.policy=delete

# OR
kafka-configs.bat --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete

Note: if the script fails because of java error or missing java command, set JAVA_HOME to <ISHOME>/jdk/jre and try again.

## In Unix / Linux
JAVA_HOME=/opt/IBM/InformationServer/jdk/jre; export JAVA_HOME

## In Windows
set JAVA_HOME=C:\IBM\InformationServer\jdk\jre

Confirm the log retention policy for __consumer_offsets is changed to "delete". The following command output must show "cleanup.policy=delete".

## In Unix / Linux
./kafka-topics.sh  --zookeeper localhost:52181 --describe --topic __consumer_offsets

# OR
./kafka-configs.sh --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets


## In Windows
kafka-topics.bat --zookeeper localhost:52181 --describe --topic __consumer_offsets

# OR
kafka-configs.bat --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
Note:
  (1) Recent version Kafka commands might not support "--zookeeper <zk_host>:<zk_port>" option anymore, use "--bootstrap-server <kafka_host>:<kafka_port>" option in that case. For example, use
    kafka-configs.sh --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
  command instead of
    kafka-configs.sh --zookeeper localhost:52181 --describe --entity-type topics --entity-name __consumer_offsets
  command.
  (2) Once the cleanup policy changed to "delete", old files in shared-open-source/kafka/data/1/__consumer_offsets-0 directory will be deleted by Kafka.
* Example session in v11.7.1.4 on Windows
C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows>prompt $g

>cd
C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows

>kafka-topics.bat --zookeeper localhost:52181 --describe --topic __consumer_offsets
Topic: __consumer_offsets       TopicId: dWguAomoTh2txeL6NEMKSQ PartitionCount: 1       ReplicationFactor: 1    Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 1       Replicas: 1     Isr: 1
>
>kafka-configs.bat --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete
Completed updating config for topic __consumer_offsets.
>
>kafka-configs.bat --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
Dynamic configs for topic __consumer_offsets are:
  compression.type=producer sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=producer, DEFAULT_CONFIG:compression.type=producer}
  cleanup.policy=delete sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=delete, STATIC_BROKER_CONFIG:log.cleanup.policy=delete, DEFAULT_CONFIG:log.cleanup.policy=delete}
  segment.bytes=104857600 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:segment.bytes=104857600, STATIC_BROKER_CONFIG:log.segment.bytes=10485760, DEFAULT_CONFIG:log.segment.bytes=1073741824}
>

3. Restart the shared open source services if you changed log.clean.enable property in step 1.

## In Unix / Linux
cd /opt/IBM/InformationServer
shared-open-source/bin/stop-*-services.sh
shared-open-source/bin/start-*-services.sh

## In Windows
cd C:\IBM\InformationServer
shared-open-source\bin\stop-windows-services.bat
shared-open-source\bin\start-windows-services.bat

Additional Information

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"ARM Category":[{"code":"a8m0z000000bqp5AAA","label":"Zookeeper - Kafka - Solr - Shared Open Source Services"},{"code":"a8m0z000000bqp5AAA","label":"Zookeeper - Kafka - Solr - Shared Open Source Services"},{"code":"a8m0z000000bqp5AAA","label":"Zookeeper - Kafka - Solr - Shared Open Source Services"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.5.0;11.7.0"}]

Document Information

Modified date:
03 October 2024

UID

ibm13387555

Page Feedback