logstash 的grok 正则匹配URL

CHENYANG123

已于 2023-12-04 16:10:03 修改

阅读量310

点赞数

分类专栏：大数据文章标签： es

于 2023-10-19 17:01:24 首次发布

本文链接：https://blog.csdn.net/CHENYANG123/article/details/133929946

版权

大数据专栏收录该内容

2 篇文章 0 订阅

订阅专栏

官方文档

https://github.com/logstash-plugins/logstash-patterns-core/blob/main/patterns/ecs-v1/grok-patterns

需求数据：

[27/Sep/2023:14:07:18 +0800] 2409:8929:1240:992c:c036:fd71:fc30:a3d1 - 2471 "-" "POST https://3g.xxx.com/xxx/xxx/xxx/new/getAvPrice?APPTYPE=iPhone&APPVERSION=4.4.5&DEVICEALIAS=161a3797c8e0d5852b7&DEVICEID=fdb4973e88851a051be886faac3d9c42&DEVICETYPE=iPhone14,2&IDFA=00000000-0000-0000-0000-000000000000&ITVERSION=OLD&LANGTYPE=zh-Hans&PUSHTOKEN=37DF37ADE8BB1468810D069156C94C1A65F89975F32DB83038AE443CE441C310&REGISTRATIONID=161a3797c8e0d5852b7&SYSTEMVERSION=16.4.1&appversion=4.4.5&diviceid=fdb4973e88851a051be886faac3d9c42&ec=11KDcHE7gtysCnAT6ajT+FaxjcMSx+uApYBbkRtqeGBGm4GoC+0liMkVdtUMsWbUeoC217sORX8Xjyc6nVZD71ERX&lang=zh&os=IOS&osversion=16.4.1&secType=ijm&timeZone=GMT+8:00&token=N&type=MOBILE" 200 1428 687194 -,DYNAMIC|CHARGE|NOTLAST "CSMBP-AppStore-Package/4.4.5 (iPhone; iOS 16.4.1; Scale/3.00)" "application/json;charset=UTF-8"

Grok 模式

\[%{HTTPDATE:time_local}\] %{IPORHOST:ip} \- %{NUMBER:v1} %{QS:v2} \"(?:%{WORD:method} %{URL:url})%{GREEDYDATA:v3}

自定义模式

URL (http(s)?:\/\/)?%{URIHOST:domain_name}%{URIPATH:uripath}

结果

{
  "method": "POST",
  "ip": "2409:8929:1240:992c:c036:fd71:fc30:a3d1",
  "time_local": "27/Sep/2023:14:07:18 +0800",
  "uripath": "/xxx/xxx/xxx/new/getAvPrice",
  "url": "https://3g.xxx.com/xxx/xxx/xxx/new/getAvPrice",
  "domain_name": "3g.xxx.com",
  "v1": "2471",
  "v2": "\"-\"",
  "v3": "?APPTYPE=iPhone&APPVERSION=4.4.5&DEVICEALIAS=161a3797c8e0d5852b7&DEVICEID=fdb4973e88851a051be886faac3d9c42&DEVICETYPE=iPhone14,2&IDFA=00000000-0000-0000-0000-000000000000&ITVERSION=OLD&LANGTYPE=zh-Hans&PUSHTOKEN=37DF37ADE8BB1468810D069156C94C1A65F89975F32DB83038AE443CE441C310&REGISTRATIONID=161a3797c8e0d5852b7&SYSTEMVERSION=16.4.1&appversion=4.4.5&diviceid=fdb4973e88851a051be886faac3d9c42&ec=11KDcHE7gtysCnAT6ajT+FaxjcMSx+uApYBbkRtqeGBGm4GoC+0liMkVdtUMsWbUeoC217sORX8Xjyc6nVZD71ERX&lang=zh&os=IOS&osversion=16.4.1&secType=ijm&timeZone=GMT+8:00&token=N&type=MOBILE\" 200 1428 687194 -,DYNAMIC|CHARGE|NOTLAST \"CSMBP-AppStore-Package/4.4.5 (iPhone; iOS 16.4.1; Scale/3.00)\" \"application/json;charset=UTF-8\""
}

调试工具Kibana

自定义正则生效配置

如何使我们自定义的正则生效呢，在logstash 目录下搜索文件 grok-patterns，在文件末尾追加自定义的内容（名字不能与已有的名字重复），重启服务生效。

logstash-7.14.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.3.1/patterns/legacy

logstash-7.14.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.3.1/patterns/ecs-v1

logstash 配置文件

filter {

 grok {
    match => {
       "message" => "\[%{HTTPDATE:time_local}\] %{IPORHOST:ip} \- %{NUMBER:v1} %{QS:v2} \"(?:%{WORD:method} %{URL:url})%{GREEDYDATA:v3}"
   
       }
  }
  
 date {

    match => [ "time_local", "dd/MMM/yyyy:HH:mm:ss Z" ]

    timezone => "UTC"
    target => "@timestamp"


}


}