IBM Support

How to reduce the disk space of logs and data for Zookeeper

How To


Summary

Zookeeper log files and data files taking up considerable disk space in
C:\IBM\InformationServer\shared-open-source\zookeeper\data\1\version-2 and
C:\IBM\InformationServer\shared-open-source\zookeeper\logs

Steps

To control the log files located in:

AIX/Linux: <IS Install>/shared-open-source/zookeeper/logs
Windows: <IS Install>\shared-open-source\zookeeper\logs

Default log file name:
zookeeper.log.yyyy-mm-dd

Clear out current logs:
Stop shared services and delete the log files.

On AIX/Linux:
cd <IS Install>/shared-open-source/bin
./stop-aix-services.sh or ./stop-linux-services.sh

On windows:
cd <IS Install>\shared-open-source\bin
stop-windows-services.bat

Update log4j.properties, located in <IS Install>\shared-open-source\zookeeper\conf, set:

log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
instead of
log4j.appender.ROLLINGFILE=org.apache.log4j.DailyRollingFileAppender

Logs file names will now be:
zookeeper.log
zookeeper.log.1
zookeeper.log.n

You can also change the following from default to adjust the size and amount of logs retained:
log4j.appender.ROLLINGFILE.MaxFileSize=10MB
log4j.appender.ROLLINGFILE.MaxBackupIndex=5

Next start the shared services:

On AIX/Linux
cd <IS Install>/shared-open-source/bin
./start-aix-services.sh or ./start-linux-services.sh

On windows
cd <IS Install>\shared-open-source\bin
start-windows-services.bat

To control the transaction logs and snapshots located in

AIX/Linux:
/opt/IBM/InformationServer/shared-open-source/zookeeper/data/1/version-2
Windows:
C:\IBM\InformationServer\shared-open-source\zookeeper\data\1\version-2


To control you can add the following settings to the zoo1.cfg file:

autopurge.snapRetainCount
   When enabled, ZooKeeper auto purge feature retains the autopurge.snapRetainCount most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 3. Minimum value is 3.

autopurge.purgeInterval
    The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. Defaults to 0.

preAllocSize
    To avoid seeks ZooKeeper allocates space in the transaction log file in blocks of preAllocSize kilobytes. The default block size is 64M. One reason for changing the size of the blocks is to reduce the block size if snapshots are taken more often. (Also, see snapCount).

snapCount
    Clients can submit requests faster than ZooKeeper can process them, especially if there are a lot of clients. To prevent ZooKeeper from running out of memory due to queued requests, ZooKeeper will throttle clients so that there is no more than globalOutstandingLimit outstanding requests in the system. The default limit is 1,000. ZooKeeper logs transactions to a transaction log. After snapCount transactions are written to a log file a snapshot is started and a new transaction log file is started. The default snapCount is 10,000.
    
Example:
autopurge.snapRetainCount=3 (three snapshots are retained, minimum 3)
autopurge.purgeInterval=1 (once an hour the purging will be scheduled)
snapCount=10 (snapshots are taken after 10 transactions)
preAllocSize=1000 (the transaction log files will have a size of 1000kb)    
    
zoo1.cfg
tickTime=2000
initLimit=5
syncLimit=2
dataDir=C:/IBM/InformationServer/shared-open-source/zookeeper/data/1
clientPort=52181
maxClientCnxns=0
server.1=myserver:52888:53888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
snapCount=10
preAllocSize=1000

Next restart shared open source services:

On AIX/Linux
cd <IS Install>/shared-open-source/bin
./stop-aix-services.sh or ./stop-linux-services.sh
./status-aix-services.sh or ./status-linux-services.sh
./start-aix-services.sh or ./start-linux-services.sh

On Windows
cd <IS Install>\shared-open-source\bin
stop-windows-services.bat
status-windows-services.bat
start-windows-services.bat

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.7","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
10 December 2018

UID

ibm10742613