logstash处理多行日志数据,网上大部分方法是在input插件中添加codec => multiline,如下beats插件
beats {
port => 5044
codec => multiline {
pattern => "^(%{TIMESTAMP_ISO8601}|%{MONTHNUM}\s%{WORD}\s%{YEAR}\s%{TIME})"
negate => true
what => "previous"
}
}
如果是使用beats插件,不仅会导致日志数据最后一行无法读取,还会丢失最后一行日志的所有filebeat字段。
使用logstash-filter-multiline插件可以解决此问题
logstash5.0 默认没有安装此插件,需要自行安装
./bin/logstash-plugin install logstash-filter-multiline
如果安装失败或者需要离线安装,可以到已安装的机器上复制gem文件
目录 ./logstash/vendor/bundle/jruby/1.9/cache/logstash-filter-multiline-3.0.2.gem (http://download.csdn.net/download/abnersunyh/9683875)
./bin/logstash-plugin install ./vendor/bundle/jruby/1.9/cache/logstash-filter-multiline-3.0.2.gem
logstash-filter-multiline插件使用示例
filter{
multiline {
pattern => "^(%{TIMESTAMP_ISO8601}|%{MONTHNUM}\s%{WORD}\s%{YEAR}\s%{TIME})" negate => true what => "previous"
}
}
后续:
logstash5.1貌似找不到logstash-filter-multiline插件,但仍然可以安装logstash-filter-multiline-3.0.2.gem,官方并没指明是否取消该插件。https://www.elastic.co/guide/en/logstash/5.1/breaking-changes.html#_logstash_with_all_plugins_download