elk之logstash 将日志中字符串类型的时间转化成@timestamp

默认情况下@timestamp字段显示的是写入es集群数据的时间，但我们可能需要记录的是日志中的字符串类型的时间（也就是打印日志里记录的时间），所以我们需要把日志中字符串类型的时间覆盖掉@timestamp中的当前时间。这样就可以实现es数据里的日志和程序存储在本地文件的时间一致，排查问题和分析问题也更加方便，就和在服务器上查看日志一样。
先附上全部的配置

input {
    kafka {
        bootstrap_servers => ["xx:9092,xx:9092,xx:9092"]
        topics => ["logsxxof"]
        codec => "json"
        consumer_threads => 1
        auto_offset_reset => "latest"
        group_id => "logxxof"
        decorate_events => true
        #auto_commit_interval_ms => 5000
        max_partition_fetch_bytes => "52428700"
        max_poll_records => "200"
        session_timeout_ms => "50000"
        request_timeout_ms => "510000"
        heartbeat_interval_ms => "1000"
        }
    }

filter {
   grok  {
        patterns_dir => [ "/etc/logstash/patterns.d" ]
        match => [ "message", "%{TIMESTAMP_ISO8601:log_time}\s+\[%{THREADID:threadId}\]\s+\[%{THREADNAME:traceid}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}","message", "%{TIMESTAMP_ISO8601:log_time}\s+\[%{THREADID_1:threadId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}","message","%{TIMESTAMP_ISO8601:log_time}\s+%{TID:TID}\s+\[%{THREADID_1:threadId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}"]
        remove_field => [ "message","beat","timestamp","topic","hostname","name","index","host","tags"]
   }
   ruby {
        code => "event.timestamp.time.localtime"
      }
   date {match=>["log_time","yyyy-MM-dd HH:mm:ss.SSS"]}
   mutate {remove_field => ["log_time"]}
}
output {
 if [logtype] == "producxxprovider" {
  elasticsearch {
        hosts => ["xx:9200","xx:9200","xx:9200"]
        index => "logstashxxof-provider-%{+YYYY-MM-dd}"
        user => xx
        password => xx
        }
        stdout { codec => rubydebug }
 }
 if [logtype] == "produxxf-web" {
  elasticsearch {
        hosts => ["xx:9200","xx:9200","xx:9200"]
        index => "logstasxxpof-web-%{+YYYY-MM-dd}"
        user => xx
        password => xx
        }
        stdout { codec => rubydebug }
 }
}

其中

date {match=>["log_time","yyyy-MM-dd HH:mm:ss.SSS"]}
#mutate {remove_field => ["log_time"]}

是我们这次新增的配置，log_time 和 %{TIMESTAMP_ISO8601:log_time}字段相对应。大致意思是log_time 字段的内容 如果和yyyy-MM-dd HH:mm:ss.SSS 匹配上了，就会替换@timestamp，target默认指的就是@timestamp，所以log_time的时间更新@timestamp的时间。

重启logstash后验证：

红框内的时间一致说明以log_time的时间更新覆盖掉了@timestamp的写入数据的时间了。
验证完毕后去掉log_time字段，将mutate {remove_field => ["log_time"]}注释打开。
脚本

cd /etc/logstash/conf.d/ && for i in `ls *.conf`;do num=$(expr ` sed -n '/output/=' $i` - 2 ) && sed -i 's/timestamp}/log_time}/g' $i && sed  -i "$num a\ \ \ date\ \{match=>\[\"log_time\",\"yyyy-MM-dd\ HH\:mm\:ss.SSS\"\]\}" $i ;done