rsyslog模块 -- mmfields

最新推荐文章于 2022-04-19 12:13:26 发布

weixin_33888907

最新推荐文章于 2022-04-19 12:13:26 发布

阅读量247

点赞数

文章标签：运维 json python

原文链接：https://my.oschina.net/MrYx3en/blog/495385

版权

2019独角兽企业重金招聘Python工程师标准>>>

官方示例

下面简单的示例的每个msg都按照指定格式被解析。逗号被作为默认分隔符。

module(load="mmfields")
template(name="ftpl"
         type=string
         string="%$!%\\n")
action(type="mmfields")
action(type="omfile"
       file="/path/to/logfile"
       template="ftpl")

下面的示例和前一个有点类似，但是这次指定使用 ':' 冒号作为分隔符，而且数据被写入以 '$!mmfields' 开始的 json路径

module(load="mmfields")
template(name="ftpl"
         type=string
         string="%$!%\\n")
action(type="mmfields"
       separator=":"
       jsonRoot="!mmfields")
       action(type="omfile"
       file="/path/to/logfile"
       template="ftpl")

实际解析

nginx.conf定义的日志格式

log_format              proxyformat    "$remote_addr`$request_time_usec`$http_x_readtime`[$time_local]`$host`$request_method`$scheme://$host$request_uri`$status`$body_bytes_sent`$http_referer`$upstream_addr`$upstream_response_time`$http_user_agent";

结合mmfields和omelasticsearch模块，收集nginx日志，并发送给ES：

module(load='mmfileds')
module(load="omelasticsearch")
module(load='imfile')

template( name="jsonmesgTemplate" type="string" string="%jsonmesg%\n" )
template( name="linetemplate" type="list" )
{
    constant(vaule="{")
    property(name="$!line" position.from="2")
    constant(value="\n")
}

Ruleset( name='mmfieldsruleset' )
{
    action (
        type="mmfields"
        separator="`"
        jsonRoot="!mmfields"
    )
    set $!line!remote_addr = $!mmfields!f1;
    set $!line!request_time_usec = $!mmfields!f2;
    set $!line!http_x_readtime = $!mmfields!f3;
    set $!line!time_local = $!mmfields!f4;
    set $!line!host = $!mmfields!f5;
    set $!line!request_method = $!mmfields!f6;
    set $!line!request = $!mmfields!f7;
    set $!line!status = $!mmfields!f8;
    set $!line!body_bytes_sent = $!mmfields!f9;
    set $!line!http_referer = $!mmfields!f10;
    set $!line!upstream_addr = $!mmfields!f11;
    set $!line!upstream_response_time = $!mmfields!f12;
    set $!line!http_user_agent = $!mmfields!f13;

    action (
        type="omelasticsearch"
        template="linetemplate"
        ...
    )
    action (
        type="omfile"
        name="action_omfile-mmfields"
        File="/opt/nginx-access-mmfields.log"
#        template="jsonmesgTemplate"
        template="linetemplate"
        FileCreateMode="0644"
        flushOnTXEnd="off" 
        ioBufferSize="8192k"
        flushInterval="5"
    )
}

input( type='imfile' Flie='/var/log/nginx/access.log' Tag='nginx-access' Facility='local6' persiststateinterval='1000' )

if( $programname == "nginx-access" and $syslogfacility-text == "local6" ) then {
    call mmfieldsruleset
    stop
}

nginx-access-mmfields.log

{
    "remote_addr": "ip.ip.ip.ip",
    "request_time_usec": "1921",
    "http_x_readtime": "-",
    "time_local": "[08/Dec/2015:14:56:41 +0800]",
    "host": "a.b.com",
    "request_method": "HEAD",
    "request": "https://urlpath/globalConfig.json",
    "status": "200",
    "body_bytes_sent": "0",
    "http_referer": "-",
    "upstream_addr": "ip.ip.ip.ip:80",
    "upstream_response_time": "0.002",
    "http_user_agent": "Dalvik/2.1.0 (Linux; U; Android 5.1.1; OPPO A53m Build/LMY47V)"
}

转载于:https://my.oschina.net/MrYx3en/blog/495385