前言
在线调试工具:http://grokdebug.herokuapp.com/
一个例子
input {
file {
path => "/data/mosh/logstash-6.2.4/logs/test.log"
# 下面2个配置是为了从头开始读
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:tmp_log_date} \|- %{WORD} %{JAVACLASS} \[%{JAVAFILE}\] - taskId %{BASE10NUM:tmp_task_id} step %{INT:tmp_task_step}"]
add_field => {
"log_date" => "%{tmp_log_date}"
"task_id" => "%{tmp_task_id}"
"task_step" => "%{tmp_task_step}"
}
}
}
output {
# 不满足筛选条件的就不写入数据库了
if "_grokparsefailure" not in [tags] {
mongodb {
uri => "mongodb://username:userpassword@mad134:27019"
database => "cis-ws-monitor"
collection => "task"
}
}
}
输出ERROR级别的日志
input {
file {
path => "/data/mosh/logstash-6.2.4/logs/server.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "%{TIMESTAMP_ISO8601:logdate} \|-\s*%{LOGLEVEL}"
negate => true
what => "previous"
auto_flush_interval => 30 # 如果在规定时候内没有新的日志事件就不等待后面的日志事件
}
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logdate} \|-\s*ERROR"]
}
}
output {
if "_grokparsefailure" not in [tags] {
stdout {
codec => rubydebug
}
}
}
读取多个路径下的日志
input {
file {
path => ["/data/server.log","/data/server2.log"]
}
}
参考博客
每次从头读日志
[1]https://blog.csdn.net/jiao_fuyou/article/details/50777816
默认类型
[2]https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
查看插件和安装插件
[3]https://blog.csdn.net/laoyang360/article/details/65448962
清除解析失败的数据
[4]https://blog.csdn.net/qq1032355091/article/details/52953837?locationNum=3&fps=1
[5]http://www.mamicode.com/info-detail-1693015.html
自定义输出的json格式
[6]https://www.cnblogs.com/qq27271609/p/4762562.html
Logstash的简单使用
[7]https://blog.csdn.net/qq_33689414/article/details/80365029