ELK之logstash7.5.1插件使用

最新推荐文章于 2021-04-23 16:49:20 发布

愚时般若生

最新推荐文章于 2021-04-23 16:49:20 发布

阅读量297

点赞数

分类专栏： ELK

本文链接：https://blog.csdn.net/linpxing1/article/details/105872748

版权

ELK 专栏收录该内容

7 篇文章 2 订阅

订阅专栏

ES logstash 的帮助页面：

input:
https://www.elastic.co/guide/en/logstash/current/input-plugins.html

output:
https://www.elastic.co/guide/en/logstash/current/output-plugins.html

plugins:
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

本文主要讲实战下的简单插件的使用
grok help:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

grok debug:

http://47.112.11.147:9999/ [网上找到的一位大神部署的grok debug服务！小生在此感谢了！]

grok parttern list: 官方提供的常用的正则表达式

https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

message:

2020-04-27 21:28:04.751 [tomcat-8.0.22-8001:20200427212802844541929492614291] [s0200427212802844541929492614291] [com.service.Device.manual:821] INFO 输入 hell world{“aa”:“bb”}

aa.config 文件

input {
    beats {
        port => "5044"
        client_inactivity_timeout => 3000
    }
    
}

filter {
        grok {
         	#match => ["message", "%{TIMESTAMP_ISO8601:logtime} (?<rid>R\d{1,32})\s*\[(?<method>[^\[\]]+)\]\s*%{LOGLEVEL:level}\s*%{GREEDYDATA:msg}"]
         	## 使用正则表达式 重新切分出自己要的组
	    	match => ["message","(?<logtime>[^\[\]]+) \[(?<tomcat>[^\[\]]+):(?<rid>[^\[\]]+)\] \[(?<tranceid>[^\[\]]+)\] \[(?<method>[^\[\]]+)\] (?<leve>[A-Z]+) %{GREEDYDATA:msg}"]
        }

## 不存在则丢弃这一行
	if ![rid] {
                drop {}
        }
## 重写timestamp 为日志写入的时间
        date {
                match => ["logtime", "yyyy-MM-dd HH:mm:ss.SSS"]
                target => "@timestamp"
        }

## mutate 插件，删除字段，增加自定义字段
        mutate {
#                replace => ["message", "%{msg}"]
                remove_field => ["msg"]
				
#                remove_field => ["YEAR"]
#                remove_field => ["MONTHNUM"]
#                remove_field => ["MONTHDAY"]
#                remove_field => ["HOUR"]
#                remove_field => ["MINUTE"]
#                remove_field => ["SECOND"]
#                remove_field => ["ISO8601_TIMEZONE"]
				
                replace => ["host", "192.168.9.82"]
                add_field => ["hostname", "filebeat"]
        }
}

output {
    elasticsearch {
            hosts => ["192.168.9.93:9200"]
            index => "bee-log-%{+YYYYMMdd}"
	    #template =>  "/app/soft/logstash-7.5.1/template/bee_logsatsh2.json"
	    #template_name => "bee-log-*"
	    #template_overwirte => true
    }
}