How To
Summary
Recent version IBM InfoSphere Information Server installs Apache Kafka, Solr, and ZooKeeper in <ISHOME>/shared-open-source directory in Services tier.
Kafka in shared open source uses "compact" log cleanup policy for the __consumer_offsets topic by installation default, which can result in many log files for the topic and filling up the disk space.
Objective
Environment
Steps
1. Disable Kafka Log Cleaner thread.
In Services tier, open <ISHOME>/shared-open-source/kafka/conf/server1.properties file in a text editor. For example, open /opt/IBM/InformationServer/shared-open-source/kafka/conf/server1.properties in Unix or Linux, C:\IBM\InformationServer\shared-open-source\kafka\conf\server1.properties in Windows.
If log.cleaner.enable property is set to "true", change it to "false" like following, save and exit text editor.
log.cleaner.enable=false
Note: Log Cleaner thread is required for "compact" cleanup policy. Set it to "true" if you need to use "compact" policy for some topics. See log.cleaner.enable
2. Change the log cleanup policy for __consumer_offsets topic to "delete".
In Services tier, make sure Kafka service is up and running, and run following command.
## In Unix / Linux
cd /opt/IBM/InformationServer/shared-open-source/kafka/install/bin
./kafka-topics.sh --zookeeper localhost:52181 --topic __consumer_offsets --alter --config cleanup.policy=delete
# OR
./kafka-configs.sh --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete
## In Windows CMD window
cd C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows
kafka-topics.bat --zookeeper localhost:52181 --topic __consumer_offsets --alter --config cleanup.policy=delete
# OR
kafka-configs.bat --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete
Note: if the script fails because of java error or missing java command, set JAVA_HOME to <ISHOME>/jdk/jre and try again.
## In Unix / Linux
JAVA_HOME=/opt/IBM/InformationServer/jdk/jre; export JAVA_HOME
## In Windows
set JAVA_HOME=C:\IBM\InformationServer\jdk\jre
Confirm the log retention policy for __consumer_offsets is changed to "delete". The following command output must show "cleanup.policy=delete".
## In Unix / Linux
./kafka-topics.sh --zookeeper localhost:52181 --describe --topic __consumer_offsets
# OR
./kafka-configs.sh --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
## In Windows
kafka-topics.bat --zookeeper localhost:52181 --describe --topic __consumer_offsets
# OR
kafka-configs.bat --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows>prompt $g
>cd
C:\IBM\InformationServer\shared-open-source\kafka\install\bin\windows
>kafka-topics.bat --zookeeper localhost:52181 --describe --topic __consumer_offsets
Topic: __consumer_offsets TopicId: dWguAomoTh2txeL6NEMKSQ PartitionCount: 1 ReplicationFactor: 1 Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
Topic: __consumer_offsets Partition: 0 Leader: 1 Replicas: 1 Isr: 1
>
>kafka-configs.bat --bootstrap-server localhost:59092 --alter --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete
Completed updating config for topic __consumer_offsets.
>
>kafka-configs.bat --bootstrap-server localhost:59092 --describe --entity-type topics --entity-name __consumer_offsets
Dynamic configs for topic __consumer_offsets are:
compression.type=producer sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=producer, DEFAULT_CONFIG:compression.type=producer}
cleanup.policy=delete sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=delete, STATIC_BROKER_CONFIG:log.cleanup.policy=delete, DEFAULT_CONFIG:log.cleanup.policy=delete}
segment.bytes=104857600 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:segment.bytes=104857600, STATIC_BROKER_CONFIG:log.segment.bytes=10485760, DEFAULT_CONFIG:log.segment.bytes=1073741824}
>
3. Restart the shared open source services if you changed log.clean.enable property in step 1.
## In Unix / Linux
cd /opt/IBM/InformationServer
shared-open-source/bin/stop-*-services.sh
shared-open-source/bin/start-*-services.sh
## In Windows
cd C:\IBM\InformationServer
shared-open-source\bin\stop-windows-services.bat
shared-open-source\bin\start-windows-services.bat
Additional Information
Related Information
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
03 October 2024
UID
ibm13387555