Logstash学习总结（三） FILTER应用实例

一、解析apache日志，date日志处理
  
filter {  
  grok { 
    match => { "message" => "%{COMBINEDAPACHELOG}" }  
  }  
  
  date {  
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]  
    target => "@timestamp"
  }  
}


二、解析日志，丢弃不符合要求的日志，转换字段类型

filter {
  grok  {
    patterns_dir => ["/usr/local/test/logstash-5.0.0/etc/patterns"]
    match => {
       "message" => "\[%{T_AWS_DT:datetime}\] %{WORD:code} %{WORD:type} %{T_CONTENT}"
    }
  }

  if "_grokparsefailure" in [tags]{
    drop{}
  }
  
  mutate{
    convert => {"code" => "string"}
  }
}


三、解析日志，date日志转换，转换字段数据类型

filter {
  grok {
    patterns_dir => ["/usr/local/test/logstash-5.0.0/etc/patterns"]
    match => {
     "message" => "********"
    }
  }

  date {
    match => ["datetime","yyyy-MM-dd HH:mm:ss.SSS"]
    target => "@timestamp"
  }  

  mutate {
    convert => {"transCost" => "integer"}    #transCost是上面解析出来定义的字段
  }
}


四、解析日志，date日志转换，丢弃不符合要求日志，根据字段值判断处理
filter {
  grok  {
    patterns_dir => ["/usr/local/test/logstash-5.0.0/etc/patterns"]
    match => {
      "message" => "*******"  
    }
  }

  date {
    match => ["datetime","yyyyMMddHHmmssSS"]
    target => "@timestamp"
  }


  if "_grokparsefailure" in [tags]{
    drop{}
  }


  if "_groktimeout" in [tags]{
    drop{}
  }


  if !([transCode] in ["084015","040600"])
  {
    mutate {
     remove_field => ["accountNo"]
    }
  }
}


五、解析日志，丢弃不符合要求日志，date日志转换，转换字段数据类型，替换字段值


filter {
  grok {
    patterns_dir => ["/usr/local/test/logstash-5.0.0/etc/patterns"]
    match => {
     "message" => "*******"
    }
  }


  if "_grokparsefailure" in [tags]{
    drop{}
  }


  date {
    match => ["recvDatetime","yyyy/MM/dd HH:mm:ss.SSS"]
    target => "@timestamp"
  }


  mutate {
    convert => {"recvLength" => "integer"}
    convert => {"recvXmlLength" => "integer"}
    convert => {"sendXmlLength" => "integer"}
    convert => {"sendLength" => "integer"}
  }
  
  mutate {
    replace => { "datetime" => "%{date}%{time}" }
  }
}


六、关键词查找，根据条件结果添加字段
filter {
  if "is Down" in [message] or "is DOWN" in [message] {
    mutate{
       add_field => {"keyval" => "Down"}   
    }
  } else {
    mutate{
       add_field => {"keyval" => "Up"}   
    }
  }
}