logstash 配置filter规则，解决Index切片慢8小时的问题

最新推荐文章于 2024-07-17 13:20:15 发布

stefan1240

最新推荐文章于 2024-07-17 13:20:15 发布

阅读量1.1k

点赞数

分类专栏： elk 文章标签： elk

本文链接：https://blog.csdn.net/stefan1240/article/details/87720290

版权

elk 专栏收录该内容

2 篇文章 1 订阅

订阅专栏

最近在搭建elk日志采集系统，发现index都是早上八点才开始切，这对于数据展示会影响，于是想把index改成凌晨过零点就切换。

我的logstash版本：5.4.1。先附上我的解决方案吧：

在filter里面加入：

 ruby {   
   code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"
 }  
 ruby {  
   code => "event.set('@timestamp',event.get('timestamp'))"  
 }  

 mutate {  
   remove_field => ["timestamp"]  
 }

我的整个logstash 配置文件我也发一下：

input {
    kafka {
        bootstrap_servers => ["10.7.1.2:9092"]
        client_id => "id1"
        group_id => "group1"
        auto_offset_reset => "latest"
        consumer_threads => 4
        decorate_events => true 
        codec => "json"
        topics => "collectcd"
        }
}
filter {
   mutate {
      gsub =>[
          "message", "@.", "@"
           ]
   }
  mutate {
     split => ["message"," "]     
  }
  mutate{
     add_field =>   {
        "function_name" => "%{[message][0]}"
        "value" => "%{[message][1]}"
        "max_value" => "%{[message][2]}"
        }
  }
  mutate {
     split => ["function_name","@"]
  }
  mutate {
     add_field => {
         "trainId" => "%{[function_name][0]}"
         "function" =>  "%{[function_name][1]}"
     }
  }
  ruby {   
    code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"
  }  
  ruby {  
    code => "event.set('@timestamp',event.get('timestamp'))"  
   }  
  # remove  the field containing the decorations, unless you want them to land into ES
   mutate {
      remove_field => ["kafka"]
      remove_field => ["message"]
      remove_field => ["beat"]
      remove_field => ["function_name"]
      remove_field => ["timestamp"]
   }
}
output {
     elasticsearch {
        action => "index"
        hosts => ["10.7.1.8:9200"]
        index => "%{function}-%{+YYYY.MM}"
       codec => json
       # template => "/home/elasticsearch-6.3.1/config/templates/logstash.json" 
       # manage_template => false 
       # template_name => "crawl" 
       # template_overwrite => true 
     }
}