elk-logstash 读取日志(2): messages和log日志生成时间替换时间戳

最新推荐文章于 2024-07-02 00:09:27 发布

追风09

最新推荐文章于 2024-07-02 00:09:27 发布

阅读量1.2k

点赞数 1

分类专栏： ELK 文章标签： elk java

本文链接：https://blog.csdn.net/qq_33834493/article/details/117093662

版权

ELK 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

目前日志：

2021-05-20 14:26:34.817 [http-nio-9031-exec-9] INFO com.abutment.producer.RocketMqHelper.sendMessage - >>>>生产者发送topic为: dispatch_topic msgId: AC1188CD08496B884D5764F234ACDB4E tag: GID_91610000786990367Y
2021-05-20 14:26:34.817 [http-nio-9031-exec-9] INFO com.abutment.producer.RocketMqHelper.sendMessage - 消息发送成功topic为: dispatch_topic msgId: AC1188CD08496B884D5764F234ACDB4E tag: GID_91610000786990367Y

目前日志信息中有两种时间格式，分别是以下：

Jan 22 10:01:01 # 对应SYSLOGTIMESTAMP
2021-05-20 08:53:00,321 或者是2021-05-20 14:26:34.817 # 对应TIMESTAMP_ISO8601
本篇博客中实现了将时间戳转换为日志信息中的时间而不是导入时间，代码可以直接看三和四板块。

主要要调整的是三个部分：

1.修改对应的时间格式（TIMESTAMP_ISO8601或者SYSLOGTIMESTAMP）
2.修改日期所在的字段名（本例中是message）
3.修改时间格式，2021-05-20 14:26:34.817对应的是YYYY-MM-dd HH:mm:ss.SSS；如果秒后面没有内容了，则对应的是YYYY-MM-dd HH:mm:ss。其余同理

一、参考网站
官方预定义的 grok 表达式：https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

grok·ELKstack中文指南：https://elkguide.elasticsearch.cn/logstash/plugins/filter/grok.html

Logstash基础正则地址：https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

二、grok
Grok Debug网站：

①http://grokdebug.herokuapp.com/（官网）

②https://www.5axxw.com/tools/v2/grok.html（可用）

用于grok语句调试，grok其实就是用正则表达式匹配内容，即我们的日志信息。

（第一个链接属于官方链接，但是我打开却无法运行出结果，建议使用第二个网页），用一下语句匹配我们的日志，发现时间信息可以成功获取

三、logstash——log时间戳替换
# input
2021-05-20 14:26:34.890 [ConsumeMessageThread_4] INFO com.abutment.consumer.ConsumerProcess.lambda$process$2 - 推送信息结束,推送给非更新接口

# pattern
%{TIMESTAMP_ISO8601:logdate}

input {
	file {
		path => "D:/SoftWare/elk/info8.log"
		type => "elasticsearch"
		start_position => "beginning" 
}}
 
filter{
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:logdate}" }
  }
  date {
    match => [ "logdate", "YYYY-MM-dd HH:mm:ss.SSS" ]
    target => "@timestamp"
    timezone =>"+00:00"
  }
  mutate{
    remove_field => "logdate"
  }
}
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "logstash-systemlog-%{+YYYY.MM.dd}"
  }
}

结果：时间已经转化为北京时间（目前想做到2021-05-20 14:26:34.839),@timesstamp值同message中时间一致

2.非规范日志读取时间问题：涉及字符串的拼接操作，时间匹配

日志内容：

2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3808 - 收方运输订单验证成功
2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3840 - 收方派单重量超过调运计划重量验证成功
2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3933 - 承运订单验证成功
2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3968 - 来源为0订单验证成功

匹配日志时间到@timestamp

conf:

input {
	file {
		path => "D:/SoftWare/elk/logs/info301.log"
		type => "elasticsearch"
		start_position => "beginning"
}}
 
filter{
  grok {
    match => { "message" => "%{GREEDYDATA:yearmonthday} at %{GREEDYDATA:time} CST" }
  }
  mutate{
    add_field => {"sss" =>",000"}
    add_field => {"logdate" => "%{yearmonthday} %{time}%{sss}"}
 
   }
    mutate{
  
    gsub => ["logdate", "\.", "-"]
   }
  date {
    match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
    target => "@timestamp"
    timezone =>"+00:00"
  }
  #mutate{
  #  remove_field => "logdate"
  #}
}
output {
  stdout {} 
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "logstash-systemlog-%{+YYYY.MM.dd}"
  }
}

验证结果：

{
    "yearmonthday" => "2021.05.23",
            "type" => "elasticsearch",
            "time" => "12:06:33",
      "@timestamp" => 2021-05-23T12:06:33.000Z,
            "path" => "D:/SoftWare/elk/logs/info301.log",
             "sss" => ",000",
         "message" => "2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3933 - \\xB3\\xD0\\xD4?\\xA9\\xB5\\xA5\\xD1\\xE9?\\xB3?\\xA6\\r",
            "host" => "DESKTOP-ELQVSAJ",
        "@version" => "1",
         "logdate" => "2021-05-23 12:06:33,000"
}
{
    "yearmonthday" => "2021.05.23",
            "type" => "elasticsearch",
            "time" => "12:06:33",
      "@timestamp" => 2021-05-23T12:06:33.000Z,
            "path" => "D:/SoftWare/elk/logs/info301.log",
             "sss" => ",000",
         "message" => "2021.05.23 at 12:06:33 CST INFO  com.zkkj.tms.web.DispatchController isDispatchByTms() @3801 - \\xCA?\\xBD\\xB5\\xF7\\xD4??\\xAE\\xD1\\xE9?\\xB3?\\xA6\\r",
            "host" => "DESKTOP-ELQVSAJ",
        "@version" => "1",
         "logdate" => "2021-05-23 12:06:33,000"
}