nginx grok 正则错误的输出情况

原创 2016年08月29日 09:54:16
nginx 配置:
http {
    include       mime.types;
    default_type  application/octet-stream;
     log_format  main  '$http_host $server_addr $remote_addr [$time_local] "$request" '
                     '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
                     '$request_time $upstream_response_time';
      #send the log to syslog and file.
      access_log  /var/log/nginx/access.log main;

      # pre 1.5.x
      error_log /var/log/nginx/error.log;
	  

nginx 服务器rsyslog配置:
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
module(load="imfile" PollingInterval="5")
$ModLoad imtcp
$InputTCPServerRun 514
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none;local5.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
input(type="imfile"
File="/var/log/nginx/access.log"
Tag="uat-frontend01-access"
Severity="info"
Facility="local5")
input(type="imfile"
File="/var/log/nginx/error.log"
Tag="uat-frontend01-error"
Severity="info"
Facility="local5")
local5.* @@xx:514


logstash 配置;
zjtest7-frontend:/usr/local/logstash-2.3.4/config# cat loguat.cof 
input {
        file {
                type => "uat_nginx_access"
                path => ["/rsyslog/data/nginx/uat/nginx_access0*_log.*"]
        }
}
filter {
    grok {
        match => {
            "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\""
        }
    }   
}

output {
      elasticsearch {
                hosts => "192.168.32.80:9200"
                index => "logstash-uat-test"
        }
		stdout {
			codec => rubydebug
		}
        }

		
logstash 输出;
zjtest7-frontend:/usr/local/logstash-2.3.4/config# ../bin/logstash -f loguat.cof 
Settings: Default pipeline workers: 1
Pipeline main started
{
       "message" => " uatest.winfae.com 121.40.189.90 121.40.205.143 [29/Aug/2016:09:42:25 +0800] \"GET /wechat/css/wechat.2a00a782.css HTTP/1.1\" - 304 0 \"https://uatest.winfae.com/wechat/account.html\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\" 0.000 -",
      "@version" => "1",
    "@timestamp" => "2016-08-29T01:45:09.748Z",
          "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
          "host" => "0.0.0.0",
          "type" => "uat_nginx_access",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

elasticsearch 输出;


{

    "_index": "logstash-uat-test",
    "_type": "uat_nginx_access",
    "_id": "AVbT-JPMEY-onx06xYf_",
    "_version": 1,
    "_score": 1,
    "_source": {
        "message": " uatest.winfae.com 121.40.189.90 121.40.205.143 [29/Aug/2016:09:42:25 +0800] "GET /wechat/js/libs/dialog-min.88247f5e.js HTTP/1.1" - 304 0 "https://uatest.winfae.com/wechat/account.html" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN" 0.000 -",
        "@version": "1",
        "@timestamp": "2016-08-29T01:45:10.220Z",
        "path": "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
        "host": "0.0.0.0",
        "type": "uat_nginx_access",
        "tags": [
            "_grokparsefailure"
        ]
    }
	
<img src="http://img.blog.csdn.net/20160829100135123?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" />


ELK错误1_Kafka->Logstash->Elasticsearch过程,Elasticsearch报grokparsefailure错误

实验背景: Kafka->Logstash->Elasticsearch实验环境安装完成; Logstash的启动文件设置有过滤机制; Logstash启动文件: input { kafka {   ...

logstash使用grok正则解析日志

http://xiaorui.cc/2015/01/27/logstash%E4%BD%BF%E7%94%A8grok%E6%AD%A3%E5%88%99%E8%A7%A3%E6%9E%90%E6%9...
  • earbao
  • earbao
  • 2015年10月21日 16:38
  • 29111

grok 正则解析日志例子<1>

下面是日志的样子 55.3.244.1 GET /index.html 15824 0.043 正则的例子 %{IP:client} %{WORD:method} %{URIPATHPARAM:re...

Logstash 常用正则(grok-patterns)

USERNAME [a-zA-Z0-9._-]+ USER %{USERNAME} INT (?:[+-]?(?:[0-9]+)) BASE10NUM (?[+-]?(?:(?:[0-9]+(?:\....

logstash + grok 正则语法

logstash + grok 正则语法
  • signmem
  • signmem
  • 2017年02月28日 17:58
  • 696

logstash,nginx日志,grok pattern调试

#Nginx日志格式定义 log_format combine '$remote_addr - $remote_user [$time_local] "$request" $h...

ELK收集Nginx日志,使用grok正则表达式(二)

一、Nginx日志例子 Nginx日志例子 nginx日志默认配置: log_format main '$remote_addr - $remote_user [$time_local]...

UNIX环境高级编程__针对apue.h找不到的情况以及log错误信息输出的问题

今天开始进行UNIX环境高级编程的额学习之旅,刚开始的额代码中就发现apue.h找不到,本来还以为他是系统文件,但后来看了下格式,确定不是系统文件,是原作者写的头文件,目的是为了避免以后重复的头文件书...
  • VCCTor
  • VCCTor
  • 2016年01月02日 01:57
  • 756

Logstash学习5_[logstash/patterns/grok-patterns]Logstash grok 内置正则

USERNAME [a-zA-Z0-9._-]+ USER %{USERNAME} INT (?:[+-]?(?:[0-9]+)) BASE10NUM (?[+-]?(?:(?:[0-9]+(?:\...

nginx php-fpm 输出php错误日志

nginx是一个web服务器,因此nginx的access日志只有对访问页面的记录,不会有php 的 error log信息。 nginx把对php的请求发给php-fpm fastcgi进程来...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:nginx grok 正则错误的输出情况
举报原因:
原因补充:

(最多只允许输入30个字)