Technical Blog Post
Abstract
Message Broker Agent: Troubleshooting large history files filling up the file system
Body
The history files for the ITCAM Message Broker agent can become large very quickly. Warehousing is enabled for history collection, and Warehouse Proxy agent is working fine, but all history files continue to grow.
A fast growing history file can fill up file system. Agent needs sufficient disk space to perform trimming of the history files, and when this space is not available, trimming fails, and all history files start growing, making the problem worse. To control the history file size, the history collection settings need to be modified to control the problem history file size.
Following 'ls -l' output shows file sizes of some of the history files for a IIB agent (qi)
/opt/IBM/ITM/ls3263/qi/hist/AGT
-rw-r--r-- 1 mqm mqm 223387648 May 22 23:00 KQITRSJV
-rwxrwxrwx 1 mqm mqm 510 May 8 21:42 KQITRSJV.hdr
-rw-r--r-- 1 mqm mqm 3078807552 May 22 23:00 KQITRSPS <--- large file
-rwxrwxrwx 1 mqm mqm 608 May 8 21:42 KQITRSPS.hdr
-rw-r--r-- 1 mqm mqm 362799104 May 22 23:00 KQITRSSP
-rwxrwxrwx 1 mqm mqm 550 May 8 21:42 KQITRSSP.hdr
host_qi_hexnumber-01.log has following log entry, indicating that it is unable to trim history files.
(573B2559.0011-6:khdxhist.cpp,2367,"trimHistoryFile") Unable to trim history file. Table <KQITRSSP>. CopyHistoryFile status = 29
Troubleshooting
1) In the history files listed above, the KQITRSPS file is the largest in size, and corresponding history collection configuration needs to be investigated. The next step is to identify the attribute group which corresponds to this history table.
The attribute group and corresponding history file name are found in Historical data tables section of the agent User Guide:
https://www.ibm.com/support/knowledgecenter/SS3JRN_7.2.1.1/com.ibm.itcama.doc_7.2.1.1/kqi_userguide/r-dskspace_table.html
You can also find this information in the summarization and pruning agent log named host_sy_java*-01.log on the sy agent installation host - which lists all associations for the table and attribute groups, such as:
............Table : KQITRSPS, Parsers_Resource_Statistics
............Table : KQITRSSP, SOAPInput_Resource_Statistics
............Table : KQITSTBR, Broker_Statistics
Once the attribute group which is causing large history file is identified, Parsers_Resource_Statistics in this case, locate the corresponding collection in history configuration panel. Review the collection settings, then determine whether increasing the collection interval or disabling the history collection for the agent is preferred. Delete the large history file from the file system before restarting collection with increased interval, or after disabling the history collection.
2) To prevent the history files from growing indefinitely, set limit on short term history file size as described in the following document:
Limiting the growth of short-term history files:
https://www.ibm.com/support/knowledgecenter/SSTFXA_6.3.0.2/com.ibm.itm.doc_6.3fp2/adminuse/historyshortterm_limitgrowth.htm
In every agent configuration file where agent history files are to be limited in size, set KHD_TOTAL_HIST_MAXSIZE and KHD_HISTSIZE_EVAL_INTERVAL environment variables as described.
This will prevent the history file size from increasing indefinitely. Gaps in history will occur if the file size reaches size limit set, but the problem of file system out of space will be prevented.
UID
ibm11084251