logstash 格式处理

1.java抛出exception后日志不在一行

通过使用codec multiline

在logstash::input中添加

codec => multiline {
            pattern => "^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}"  正则匹配
            negate => true  false为取反
            what => "previous"  合并到上面一行内容
        }

2.处理nginx日志，可以直接设置为json格式也可以使用grok分词(分词在线验证网站：http://grokdebug.herokuapp.com/)，下面是我的配置:

input {
  file {
    type => "xx"
    path => "xx/access*"
    exclude => "*.gz"
    sincedb_path => "string"
    codec => "json"
  }
}


filter {
  if [http_x_forwarded_for] == "-" {
    mutate {
        replace => {"http_x_forwarded_for"=>"%{remote_addr}"} 
    }
  }
  geoip {
    source => "http_x_forwarded_for"
    target => "geoip"
    database => "/etc/logstash/GeoLiteCity.dat"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
  }
  mutate {
    convert => [ "body_bytes_sent", "integer"]
  }
}