ELK日志系统开发,Logstash收集nginx日志(二)

一、Nginx日志例子

Nginx日志例子

nginx日志默认配置:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for" $upstream_response_time';

nginx日志记录:

183.3.226.234 - - [16/Jul/2018:17:23:33 +0800] "POST /app/user/exchangeRC?token=undefined HTTP/1.1" 200 124 "http://test.com/dist/myroomcard.html" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/044113 Mobile Safari/537.36 MicroMessenger/6.6.7.1320(0x26060739) NetType/4G Language/zh_CN" "117.136.40.160" 0.070

 

二、使用grok表达式进行匹配

针对以上的一条记录,我们先写一个正则表达式进行匹配:

PS:grokdebug网站:http://grokdebug.herokuapp.com/,可以线上测试gork表达式(国内google.api前端资源被墙了,因此需要本地代理资源或者翻墙才能使用)

grok表达式为:

%{COMBINEDAPACHELOG} %{QS:http_x_forwarded_for} %{NUMBER:upstream_time}

%{COMBINEDAPACHELOG}:这个是匹配标准的apache或者nginx日志。

%{QS:http_x_forwarded_for}:匹配来源IP,如果使用负载均衡的话,$remote_addr获取的是负载均衡的IP

%{NUMBER:upstream_time}:后端PHP的执行时间,单位为秒

匹配结果如下:

{
  "COMBINEDAPACHELOG": [
    [
      "183.3.226.234 - - [16/Jul/2018:17:23:33 +0800] "POST /app/user/exchangeRC?token=undefined HTTP/1.1" 200 124 "http://test.com/dist/myroomcard.html" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/044113 Mobile Safari/537.36 MicroMessenger/6.6.7.1320(0x26060739) NetType/4G Language/zh_CN""
    ]
  ],
  "COMMONAPACHELOG": [
    [
      "183.3.226.234 - - [16/Jul/2018:17:23:33 +0800] "POST /app/user/exchangeRC?token=undefined HTTP/1.1" 200 124"
    ]
  ],
  "clientip": [
    [
      "183.3.226.234"
    ]
  ],
  "HOSTNAME": [
    [
      "183.3.226.234"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "ident": [
    [
      "-"
    ]
  ],
  "USERNAME": [
    [
      "-",
      "-"
    ]
  ],
  "auth": [
    [
      "-"
    ]
  ],
  "timestamp": [
    [
      "16/Jul/2018:17:23:33 +0800"
    ]
  ],
  "MONTHDAY": [
    [
      "16"
    ]
  ],
  "MONTH": [
    [
      "Jul"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "TIME": [
    [
      "17:23:33"
    ]
  ],
  "HOUR": [
    [
      "17"
    ]
  ],
  "MINUTE": [
    [
      "23"
    ]
  ],
  "SECOND": [
    [
      "33"
    ]
  ],
  "INT": [
    [
      "+0800"
    ]
  ],
  "verb": [
    [
      "POST"
    ]
  ],
  "request": [
    [
      "/app/user/exchangeRC?token=undefined"
    ]
  ],
  "httpversion": [
    [
      "1.1"
    ]
  ],
  "BASE10NUM": [
    [
      "1.1",
      "200",
      "124",
      "0.070"
    ]
  ],
  "rawrequest": [
    [
      null
    ]
  ],
  "response": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "124"
    ]
  ],
  "referrer": [
    [
      ""http://test.com/dist/myroomcard.html""
    ]
  ],
  "QUOTEDSTRING": [
    [
      ""http://test.com/dist/myroomcard.html"",
      ""Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/044113 Mobile Safari/537.36 MicroMessenger/6.6.7.1320(0x26060739) NetType/4G Language/zh_CN"",
      ""117.136.40.160""
    ]
  ],
  "agent": [
    [
      ""Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/044113 Mobile Safari/537.36 MicroMessenger/6.6.7.1320(0x26060739) NetType/4G Language/zh_CN""
    ]
  ],
  "http_x_forwarded_for": [
    [
      ""117.136.40.160""
    ]
  ],
  "upstream_time": [
    [
      "0.070"
    ]
  ]
}

 

三、整理我们所需要的字段

以上通过正则表达式匹配的结果,可以获取到

字段名称说明
clientip请求的IP来源
timestamp请求时间
verb请求方式
request请求地址
responsehttp响应码

这里就不一一列举了

四、后续修饰

通过grok正则表达式,我们已经获取到我们想要的数据了,但是数据类型有些不对,比如upstream_time发送到es是String格式,我们需要转换为float。

这里需要使用到filter中的date和mutate插件,最终的config配置文件如下:

input {
  file {
        path => "/data0/log_receiver/nginx/access.log"
  }
}
filter {
  grok {
        match => { "message" => "%{COMBINEDAPACHELOG} %{QS:http_x_forwarded_for} %{NUMBER:upstream_time}"}
  }
  mutate {convert => ["upstream_time", "float"]}
}
output {
  elasticsearch {
    hosts => "logview-es.yunshanpp.com:80"
    index => "xinghuo-nginx-%{+YYYY.MM.dd}"
  }
}

ELK日志系统开发,Kibana简单实用Discover(三)

 

  • 1
    点赞
  • 5
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值