Technical Blog Post
Abstract
How Log File Agent storing duplicate Events and Summary Events in Tivoli Data Warehouse
Body
I recently received a question on how LFA handles Duplicate events and Summary events when they are being stored in TDW.
Event filtering and summarization configuration options are described very good well in the Technote "Disabling Summary Events with Log File Agent. "http://www-01.ibm.com/support/docview.wss?uid=swg21680594"
To create this instance I used the autodiscovery feature. I placed my duplicate.conf and duplicate.fmt files under KLO_FILE_DISCOVERY_DIR=$\{CANDLE_HOME\}/config/lo. When you are using this feature, you must make sure that create a pairs of configuration and format files. In my exampe I am using duplicate.fmt and duplicate.conf files.
Example of my duplicate.fmt file:
// Matches a simple error message like:
// Error: C disk full
REGEX REBase
Error: ([A-Z])(.*)
severity $1 CustomSlot1
msg $2
END
Example of my duplicate.conf file:
LogSources=c:/test.log
DupDetectionKeyAttributes=msg,CustomSlot1
EventSummaryInterval=60
EventFloodThreshold=
# In examples below I will changing EventFloodThreshold tag for every recreate to send_none, send_first, send_all and n_integer
Test with EventFloodThreshold=send_none tag:
I append below messages in test.log log.
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
When summary interval expired EventSummaryInterval=60 I run below command against short term history file
C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_None.out -s KLOLOGPEVT
From KLOLOGPEVT_Send_None.out can be seen that only Summary events were sent to TDW. Rest of duplicate events were dropped.
TECCLASS | LOGNAME | EIFEVENT | MSG | CUSLOT1 | OCOUNT | EVTYPE | SAMPLES |
REBase | test.log | Disk Full | E | 1 | 1 | 10 | |
REBase | test.log | Disk Full | D | 4 | 1 | 11 | |
REBase | test.log | Disk Full | C | 4 | 1 | 12 |
Test with EventFloodThreshold=send_first.tag:
I append below messages in test.log log.
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
After that I run command:
C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_First.out -s KLOLOGPEVT
From KLOLOGPEVT_Send_First.out can be seen that only the first event is sent to TDW, rest of duplicate events are dropped. Summary event at the end of summary interval with count of all duplicated events for each message is also sent to TDW:
TECCLASS | LOGNAME | EIFEVENT | MSG | CUSLOT1 | OCOUNT | EVTYPE | SAMPLES |
REBase | test.log | Disk Full | C | 1 | 0 | 10 | |
REBase | test.log | Disk Full | D | 1 | 0 | 11 | |
REBase | test.log | Disk Full | E | 1 | 0 | 12 | |
REBase | test.log | Disk Full | E | 1 | 1 | 10 | |
REBase | test.log | Disk Full | D | 4 | 1 | 11 | |
REBase | test.log | Disk Full | C | 4 | 1 | 12 |
Test with EventFloodThreshold=send_all.tag:
I append below messages in test.log log.
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
After that I run command:
C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_All.out -s KLOLOGPEVT
From KLOLOGPEVT_Send_All.out can be seen that all event are sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:
TECCLASS | LOGNAME | EIFEVENT | MSG | CUSLOT1 | OCOUNT | EVTYPE | SAMPLES |
REBase | test.log | Disk Full | C | 1 | 0 | 10 | |
REBase | test.log | Disk Full | D | 1 | 0 | 11 | |
REBase | test.log | Disk Full | E | 1 | 0 | 12 | |
REBase | test.log | Disk Full | C | 1 | 0 | 13 | |
REBase | test.log | Disk Full | D | 1 | 0 | 14 | |
REBase | test.log | Disk Full | C | 1 | 0 | 15 | |
REBase | test.log | Disk Full | D | 1 | 0 | 16 | |
REBase | test.log | Disk Full | C | 1 | 0 | 17 | |
REBase | test.log | Disk Full | D | 1 | 0 | 18 | |
REBase | test.log | Disk Full | E | 1 | 1 | 10 | |
REBase | test.log | Disk Full | D | 4 | 1 | 11 | |
REBase | test.log | Disk Full | C | 4 | 1 | 12 |
Test with EventFloodThreshold=5 tag:
I append below messages in test.log log.
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
After that I run command:
C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_5.out -s KLOLOGPEVT
From KLOLOGPEVT_5.out can be seen that every 5th event is sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:
TECCLASS | LOGNAME | EIFEVENT | MSG | CUSLOT1 | OCOUNT | EVTYPE | SAMPLES |
REBase | test.log | Disk Full | C | 1 | 0 | 10 | |
REBase | test.log | Disk Full | D | 1 | 0 | 11 | |
REBase | test.log | Disk Full | C | 1 | 0 | 12 | |
REBase | test.log | Disk Full | D | 1 | 0 | 13 | |
REBase | test.log | Disk Full | E | 3 | 1 | 10 | |
REBase | test.log | Disk Full | D | 12 | 1 | 11 | |
REBase | test.log | Disk Full | C | 12 | 1 | 12 |
Subscribe and follow us for all the latest information directly on your social feeds:
|
|
|
Check out all our other posts and updates: | |
Academy Blogs: | h |
Academy Videos: | http://ow.ly/PIKFz |
Academy Google+: | http://ow.ly/Dj3nn |
Academy Twitter Handle: | http://ow.ly/Dj35c |
UID
ibm11277368