logstash之grok解析

原始日志文件

 [2019-01-14 00:02:11] [INFO] - com.test.pushTest(PushMessageExecutor.java:103) - 消息推送结果:响应状态(200)、状态描述(成功。)、响应反馈()、请求响应耗时(232ms),deviceToken:7b64436eeea34a3ab4e0873b0682ad98e,userId:1659034,auId:null,

globalMessageId:2d09f8d389524c1f9c66b61,appId:p_ios,title:null,subTitle:null,alertBody:请及时查阅。.

logstash配置文件

input {
  file {
    path => "/data/liuzc/test_log/*"
    type => "aa"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  multiline {
    pattern => "%{DATESTAMP}"
    negate => true
    what => "previous"
  }

  

  if [type] == "aa" {
        grok {
            match => {
                        "message" => "\[%{DATA:time_local}\] \[%{LOGLEVEL:log_level}\] - %{NOTSPACE:pushExecute} - %{NOTSPACE:apns_push_result},deviceToken:%{NOTSPACE:deviceToken},userId:%{NOTSPACE:userId},auId:%{NOTSPACE:auId},globalMessageId:%{NOTSPACE:globalMessageId},appId:%{NOTSPACE:appId},title:%{NOTSPACE:title},subTitle:%{NOTSPACE:subTitle},alertBody:%{NOTSPACE:alertBody}"
      }
    }
  } else {
      grok {
        match => {
          "message" => "%{DATESTAMP:time_local} %{LOGLEVEL:log_level}"
        }
      }
    }
#ruby {
 #   code => '
  #    event["datestr"] = event["@timestamp"].time.getlocal("+08:00").strftime "%Y-%m-%d"
  #    event["hours"] = event["@timestamp"].time.getlocal("+08:00").strftime("%H").to_i
  #  '
#  }

  date {
    match => ["time_local", "yy/MM/dd-HH:mm:ss.SSS"]
  }

}

output {
    stdout{codec=>"rubydebug"}
}

解析结果：

{
             "message" => "[2019-01-14 00:02:11] [INFO] - com.test.pushTest(PushMessageExecutor.java:103) - 消息推送结果:响应状态(200)、状态描述(成功。)、响应反馈()、请求响应耗时(232ms),deviceToken:7b64436eeea34a3ab4e0873b0682ad98e,userId:1659034,auId:null,globalMessageId:2d09f8d389524c1f9c66b61,appId:p_ios,title:null,subTitle:null,alertBody:请及时查阅。.",
            "@version" => "1",
          "@timestamp" => "2019-01-17T01:16:06.468Z",
                "host" => "xy1",
                "path" => "/data/liuzc/test_log/test-2019-01-14.log",
                "type" => "aa",
          "time_local" => "2019-01-14 00:02:11",
           "log_level" => "INFO",
         "pushExecute" => "com.test.pushExecute(PushMessageExecutor.java:103)",
    "apns_push_result" => "消息推送结果:响应状态(200)、状态描述(成功。)、响应反馈()、请求响应耗时(232ms)",
         "deviceToken" => "7b64436eeea34a3ab4e0873b0682ad98e",
              "userId" => "1659034",
                "auId" => "null",
     "globalMessageId" => "2d09f8d389524c1f9c66b61",
               "appId" => "p_ios",
               "title" => "null",
            "subTitle" => "null",
           "alertBody" => "请及时查阅。."
}

Logstash 在线验证地址:

国内：http://grok.qiexun.net/

国外：http://grokdebug.herokuapp.com/