Codec: 解码编码 数据格式
json,msgpack,edn
logstash处理流程:
input->decode->filter->encode->output
plain 是一个空的解析器,它可以让用户自己制定格式
[elk@db01 0204]$ cat plain01.conf
input {
stdin {
}
}
output {
stdout{
codec=>plain
}
}
[elk@db01 0204]$ logstash -f plain01.conf
Settings: Default pipeline workers: 4
Pipeline main started
333333
44444
2017-01-17T21:16:27.548Z db01 333333
2017-01-17T21:16:34.774Z db01 44444
json编码:
如果事件数据是json格式,可以加入codec=>json来进行解析
[elk@db01 0204]$ cat plain02.conf
input {
stdin {
}
}
output {
stdout{
codec=>json
}
}
[elk@db01 0204]$ logstash -f plain02.conf
Settings: Default pipeline workers: 4
Pipeline main started
aaaa
{"message":"aaaa","@version":"1","@timestamp":"2017-01-17T21:18:22.160Z","host":"db01"}
json_lines 编码:
input {
tcp{
port=>12388
host=>"127.0.0.1"
codec=>json_lines{
}
}
}
output{
stdout{}
}
如果你的json文件比较长,需要换行的话,那么就得用到json_lines了
rubydebug
采用Ruby库来解析日志
[elk@db01 0204]$ cat ruby.conf
input {
stdin {
codec=>json
}
}
output {
stdout{
codec=>rubydebug
}
}
[elk@db01 0204]$ logstash -f ruby.conf
Settings: Default pipeline workers: 4
Pipeline main started
{"bookname":"elk","price":12}
{
"bookname" => "elk",
"price" => 12,
"@version" => "1",
"@timestamp" => "2017-01-17T21:40:28.601Z",
"host" => "db01"
}
multiline 多行事件
有时候有的日志用多行去展现,这么多行其实都是一个事件
比如JAVA的异常日志
what=>"previous" 未匹配的内容向前合并
[elk@db01 0204]$ cat mulit.conf
input {
stdin {
codec=>multiline {
pattern=>"^\["
negate=>true
what=>"previous"
}
}
}
output {
stdout{}
}
[elk@db01 0204]$ logstash -f mulit.conf
Settings: Default pipeline workers: 4
Pipeline main started
[03-Jun-2014 13:34:13:] PHP err01:aaaaaaaaa
111111111111111
222222222222222
[09-Aug-2015 44:33:22] PHP 9999
2017-01-17T21:59:39.654Z db01 [03-Jun-2014 13:34:13:] PHP err01:aaaaaaaaa
111111111111111
222222222222222
为什么[09-Aug-2015 44:33:22] PHP 9999 这条没输出,因为需要匹配下一个 pattern=>"^\["