问题:logstash@timestamp时间差8小时导致数据到前一天的索引中,比如凌晨00:57的数据跑到了前一天的索引中
很多文章的解决办法,是在原来的@timestamp的基础上进行+8小时来解决了,但我后来试过还是有问题。
给出我自己试验后的解决方案:
日志:
2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO com.yh.stock.controller.StockController - 库存减一19 -----
logstash配置:
input {
beats{
port => 5044
}
}
filter {
grok{
match => ['message','%{TIMESTAMP_ISO8601:logdate}']
}
grok{
match => { "logdate" => ["%{DATE_EU:ymd}"]}
}
date{
match => [ "logdate", "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}
output {
elasticsearch{
hosts => ["127.0.0.1:9200"]
index => "logstash-test-%{ymd}"
}
stdout{
codec=>rubydebug
}
}
input:数据来源于filebeat
filter:
grok:插件
自带正则:https://github.com/garethr/logstash-patterns/blob/master/patterns/logstash
logstash输出结果:
{
"agent" => {
"ephemeral_id" => "12ff0840-e828-4bd8-be26-658625fd9c81",
"hostname" => "DESKTOP-F55K9CL",
"id" => "776244d9-9922-4964-802c-a1423a61a3de",
"name" => "DESKTOP-F55K9CL",
"version" => "7.15.2",
"type" => "filebeat"
},
"log" => {
"offset" => 10326,
"file" => {
"path" => "D:\\yh\\ideaProjects\\springcouldalibaba\\stock\\src\\main\\resources\\logs\\20220405\\info.log"
}
},
"ymd" => "22-04-01",
"@timestamp" => 2022-03-31T16:57:49.979Z,
"host" => {
"name" => "DESKTOP-F55K9CL"
},
"ecs" => {
"version" => "1.11.0"
},
"@version" => "1",
"message" => "2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO com.yh.stock.controller.StockController - 库存减一19 -----",
"event" => {
"original" => "2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO com.yh.stock.controller.StockController - 库存减一19 -----"
},
"input" => {
"type" => "log"
},
"logdate" => "2022-04-01 00:57:49.979",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
日志里的时间是:2022-04-01 00:57:49.979
@timestamp的时间是:2022-03-31T16:57:49.979Z
ymd:22-04-01
因为我们output中配置的是index => "logstash-test-%{ymd}",所以虽然@timestamp变成了我们日志的里的时间减去8小时,但是他最后在es中生成的索引是
达到了我们要的目的。最后,可能有人有疑问,那就是kibana中的时间@timestamp会不会也是减去8小时的时间呢,我们来看一下:
没有问题,和我们日志中的时间是一致的
filebeat+elk刚接触,有问题,欢迎指出和讨论