Logstash-安装logstash-filter-multiline

最新推荐文章于 2024-07-09 08:49:10 发布

weixin_34186128

最新推荐文章于 2024-07-09 08:49:10 发布

阅读量289

点赞数

原文链接：https://my.oschina.net/xpx/blog/1939172

版权

为什么80%的码农都做不了架构师？>>>

ELK-logstash在搬运日志的时候会出现多行日志，普通的搬运会造成保存到ES中单条单条，很丑，而且不方便读取，logstash-filter-multiline可以解决该问题
github地址：https://github.com/logstash-plugins/logstash-filter-multiline
其他插件的地址：https://github.com/logstash-plugins
官网地址：https://www.elastic.co/cn/products/logstash

接下来演示下问题：

普通日志如下：

2018-08-31 15:04:41.375 [http-nio-18081-exec-1] ERROR c.h.h.control.**-自定义的msg
java.lang.ArithmeticException: / by zero
	at com.hikvision.hikserviceassign.control.ServiceMonitorManageController.reAssign(ServiceMonitorManageController.java:170)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
.....省略

记录到es会的记录则是：

2018-08-31 15:04:41.375 [http-nio-18081-exec-1] ERROR c.h.h.control.**-自定义的msg
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]java.lang.ArithmeticException: / by zero
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at com.hikvision.hikserviceassign.control.ServiceMonitorManageController.reAssign(ServiceMonitorManageController.java:170)
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at java.lang.reflect.Method.invoke(Method.java:497)
2018-08-31 15:04:41.375 [http-nio-18081-exec-1]at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
.....省略

我们希望的结果肯定是这样的

接下来就是安装：进入logstash/bin目录下使用命令
```
./logstash-plugin install logstash-filter-multiline
```
- 如果报错 certificate verify failed 则在install 后面加上 --no-verify
windows就是使用后缀加bat的那个脚本
安装成功后，增加config，demo如下：

input {
	file {
		path => "文件路径" 例如：/root/home/logs/err.log
		type => "自定义识别类型" 例如：web or admin or search
		start_position => "beginning" #从文件开始处读写
	}
}

filter {
       multiline {
            pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}" #正则
            negate => true  #The negate can be true or false (defaults to false). If true, a message #not matching the pattern will constitute a match of the multiline filter and the what will be #applied. (vice-versa is also true)
            what => "previous" 
#The what must be previous or next and indicates the relation to the multi-line event.
        }       
}

output{
	elasticsearch{
		hosts=>["地址多个用,隔开"]  
		index => "es的index默认logstash-%{+YYYY.MM.dd}"
	}
	stdout{codec => rubydebug}   #输出打印到控制台
}

转载于:https://my.oschina.net/xpx/blog/1939172