Logstash之filter:
json filter:
input{
stdin{
}
}
filter{
json{
source => "message"
}
}
output{
stdout{
codec => json
}
输入:
{"name": "CSL", "age": 20}
输出:
Grok filter:
pattern:
https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
创建一个测试log:
[sky@hadoop1 bin]$ cat spark-test-log.log
05/30/17 17:13:24 INFO StartingSparkmasteratspark
05/30/17 17:13:24 INFO RunningSparkversion1
05/30/17 17:13:25 INFO jetty
创建conf:
input{
file{
path => "/usr/local/logstash-5.6.1/bin/spark-test-log.log"
type => "sparkfile"
start_position => "beginning"
}
}
filter{
grok{
match => ["message", "%{DATE:date} %{TIME:time} %{LOGLEVEL:loglevel} %{WORD:word}"]
}
}
output{
stdout{
codec => rubydebug
}
}
运行结果:
自定义正则表达式:
[sky@hadoop1 patterns]$ cat selfpattern
SKYTIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
修改conf:
input{
file{
path => "/usr/local/logstash-5.6.1/bin/spark-test-log.log"
type => "sparkfile"
start_position => "beginning"
}
}
filter{
grok{
patterns_dir => '/usr/local/logstash-5.6.1/patterns/selfpattern'
match => ["message", "%{DATE:date} %{SKYTIME:time} %{LOGLEVEL:loglevel} %{WORD:word}"]
}
}
output{
stdout{
codec => rubydebug
}
}
输出结果:
定义多个match:使用,分隔。
测试正则表达式网址:
https://grokdebug.herokuapp.com/