logstash之filter-Geoip寻找日志ip的经纬度和城市及Key-value拆分

geoip查询

logstash可以将nginx的非格式化日志进行格式化(参考https://blog.csdn.net/weixin_44062339/article/details/103221269)，那么在nginx的日志中有IP；往往会根据ip定位当前的地理位置，
然后在kibana上以高德地图做展示；
Vim /conf/template/geoip.conf
启动：bin/logstash -f /usr/local/elk/logstash-5.5.2/conf/template/geoip.conf
向控制台输入nginx日志：

119.151.192.24 - - [10/May/2018:12:12:40 +0800] "GET /plugins/ml/ml.svg HTTP/1.1" 304 0 "http://hadoop01/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" "-"

截图展示：

为了更准确的定位ip的经纬度，可以下载GeoLite2-City.mmdb的ip-经纬度库
下载地址：
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

例子1

input{
        stdin{}
}

filter{
        grok{
                match=>{
                        "message" => "%{IPORHOST:remote_addr} - %{NGUSER:remote_addr} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:request} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NOTSPACE:http_x_forwarded_for}"
                }
        }
        geoip{
                source => "remote_addr"
                database => "/export/servers/elk/logstash-5.5.2/GeoLite2-City.mmdb"
        }
}

output{
        stdout{
                codec=> rubydebug
        }
}

例子2

input{
        stdin{}
}

filter{
        grok{
                match=>{
                        "message" => "%{IPORHOST:remote_addr} - %{NGUSER:remote_addr} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:request} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NOTSPACE:http_x_forwarded_for}"
                }
        }
        geoip{
                source => "remote_addr"
                database => "/export/servers/elk/logstash-5.5.2/GeoLite2-City.mmdb"
                target => "geoip"
                add_field => ["[geoip][coordinates]" , "&{[geoip][latitude]}"]
                add_field => ["[geoip][coordinates]" , "&{[geoip][longitude]}"]
                fields => ["country_name" , "region_name" , "city_name" , "latitude" , "longitude"]
                #remove_field => {[geoip][latitude] , []}   
     }
}

output{
        stdout{
                codec=> rubydebug
        }
}

Key-value拆分

在采集的日志中，往往出现类似于这样的URL：

https://www.baidu.com/s?wd=哈哈，这就是测试&a=1&b=2&c=3&d=4&e=5

类似这种url，字段的信息是按照&拼接而成的，所以需要把这些url进行拆分
Vim k_v_split.conf

input {
    stdin {
    }
}
filter {
         kv {
                prefix => "key_"
                source => "message"
                field_split => "&" 
                value_split => "="
            }
}
output {
    stdout{codec=>rubydebug}
}

启动：bin/logstash -f /usr/local/elk/logstash-5.5.2/conf/template/k_v_split.conf
向控制台输入：

https://www.baidu.com/s?wd=哈哈，这就是测试&a=1&b=2&c=3&d=4&e=5

结果截图：