Technical Blog Post
Abstract
Log Analysis - Logstash Filter Plugin
Body
[HTML Source]
If you are just started off with Logstash, you would want to first read the previous post =>
There are basically 3 "sections" (plugins) in Logstash - input, filter and output.
This blog entry will talk about the "filter" plugin =>
https://www.elastic.co/guide/en/logstash/2.2/filter-plugins.html
Technically speaking, it is almost impossible to tell you what to do with the "filter" section, because it is where data manipulation happens. It is entirely up to you on how you want to massage the data. Of course, if you are perfectly happy with the data, you can always leave the "filter" section blank.
Since we are learning Logstash in the context of Log Analysis (IOALA), you need to know that almost all Insight Packs came with sample Logstash configuration file which you can find under (after installation of the pack) <IOALA-DIR>/unity_content/<pack-name>/logstash
Eg.
[danielyeap@lahost WindowsOSEventsInsightPack_v1.1.0.6]$ pwd filter{ |
If you need to do more than the provided example, then you can refer to Logstash documentation =>
https://www.elastic.co/guide/en/logstash/2.2/filter-plugins.html
There are several useful filters:
(1) drop (to use when you want to drop any event/data)
https://www.elastic.co/guide/en/logstash/2.2/plugins-filters-drop.html
(2) grok (advance parsing tool for unstructured data)
https://www.elastic.co/guide/en/logstash/2.2/plugins-filters-grok.html
[PATTERNS] https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
(3) mutate (amend data to add field, update field, etc)
https://www.elastic.co/guide/en/logstash/2.2/plugins-filters-mutate.html
Most of the filters are pretty straight forward. But, I would like to talk a little bit more about "grok".
When you use "grok", you will need to know regular expression.
There are some default ones provided =>
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
As for IOALA, one of the most famous GROK pattern would be the LFAMESSAGE which is used to remove special characters in LFA messages =>
Certain insight packs will also provide their own GROK patterns that you would need to copy into your Logstash pattern directory (refer to the pack documentation).
Eg.
[danielyeap@lahost logstash]$ pwd ORACLEDBTIMESTAMP1 ^%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} PSTART (?:%{PSTART_PROCESS:process} started with pid=%{NUMBER:pid}, OS id=%{NUMBER:osid})? ORAMSGID [A-Z]{3}\-(?:[0-9]{5}) STARTING_INSTANCE Starting ORACLE instance \(%{DATA:start_type}\)
|
In a nutshell, "grok" filter allows you to extract data from event based on certain regular expressions and use the extracted data for further processing.
That is all that I would like to share here, happy coding!
UID
ibm11081617