logstash编码插件

2019独角兽企业重金招聘Python工程师标准>>>

Codec: 解码编码 数据格式  
  
json,msgpack,edn  
  
  
logstash处理流程:  
  
input->decode->filter->encode->output  
  
  
plain 是一个空的解析器,它可以让用户自己制定格式  
  
[elk@db01 0204]$ cat plain01.conf   
input {  
 stdin {  
   }  
}  
  
output {  
 stdout{  
  codec=>plain  
}  
}  
  
[elk@db01 0204]$ logstash -f plain01.conf   
Settings: Default pipeline workers: 4  
Pipeline main started  
333333  
44444
2017-01-17T21:16:27.548Z db01 333333  
2017-01-17T21:16:34.774Z db01 44444  
  
 
json编码：  
  
如果事件数据是json格式,可以加入codec=>json来进行解析  
[elk@db01 0204]$ cat plain02.conf   
input {  
 stdin {  
   }  
}  
  
output {  
 stdout{  
codec=>json  
}  
}  
  
[elk@db01 0204]$ logstash -f plain02.conf   
Settings: Default pipeline workers: 4  
Pipeline main started  
aaaa  
{"message":"aaaa","@version":"1","@timestamp":"2017-01-17T21:18:22.160Z","host":"db01"}  

  
json_lines 编码：  
  
input {  
  tcp{  
      port=>12388  
      host=>"127.0.0.1"  
      codec=>json_lines{  
   }  
 }  
}  
  
output{  
  stdout{}  
}  
  
如果你的json文件比较长,需要换行的话,那么就得用到json_lines了  
  
rubydebug   
  
采用Ruby库来解析日志  
  
[elk@db01 0204]$ cat ruby.conf   
input {  
  stdin {  
  codec=>json  
}  
}  
  
output {  
 stdout{  
 codec=>rubydebug  
 }  
}  
  
[elk@db01 0204]$ logstash -f ruby.conf   
Settings: Default pipeline workers: 4  
Pipeline main started  
{"bookname":"elk","price":12}    
{  
      "bookname" => "elk",  
         "price" => 12,  
      "@version" => "1",  
    "@timestamp" => "2017-01-17T21:40:28.601Z",  
          "host" => "db01"  
}  
  

multiline 多行事件  
  
有时候有的日志用多行去展现,这么多行其实都是一个事件  
  
比如JAVA的异常日志  
  
what=>"previous" 未匹配的内容向前合并  
[elk@db01 0204]$ cat mulit.conf   
input {  
 stdin {  
 codec=>multiline {  
 pattern=>"^\["  
 negate=>true  
 what=>"previous"  
 }  
}  
}  
  
output {  
 stdout{}  
}  
  
  
[elk@db01 0204]$ logstash -f mulit.conf   
Settings: Default pipeline workers: 4  
Pipeline main started  
[03-Jun-2014 13:34:13:] PHP err01:aaaaaaaaa  
111111111111111  
222222222222222  
[09-Aug-2015 44:33:22] PHP 9999  
2017-01-17T21:59:39.654Z db01 [03-Jun-2014 13:34:13:] PHP err01:aaaaaaaaa  
111111111111111  
222222222222222  
为什么[09-Aug-2015 44:33:22] PHP 9999 这条没输出,因为需要匹配下一个 pattern=>"^\["  

 

转载于:https://my.oschina.net/u/2000675/blog/893434