logstash利用geoip获取IP地理位置信息

Logstash模块之geoip

我们在用logstash收集日志时，有时需要将日志中的IP地址映射出具体的地理位置信息，logstash中geoip过滤器提供了这个功能.

logstash中默认的geoip插件位于
logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.9-java/vendor

GeoLite2-City.mmdb 可在 https://www.maxmind.com/en/accounts/307701/geoip/downloads进行下载，如下图：

logstash中geoip配置

filter {
   geoip{
        source => "src_ip" //原始事件中src_ip字段为源IP
        fields => ["country_name", "region_name", "city_name", "latitude", "longitude"] //通过geoip库将源IP这些字段的信息返回出来
        target => "location"  //目标字段存在 location 字段下
     }  
}

使用示例

input {
        udp{
           port => 15100
       }
}

filter {
     json{
        source => "message"
     }
     
     geoip{
        source => "src_ip"
        fields => ["country_name", "region_name", "city_name", "latitude", "longitude"]
        target => "location"    
     }  
}

output{
     stdout{}
}

向logstash的udp 15100端口发送以下测试事件：

{"src_ip":"125.33.203.100","logid":"1000"}
{"src_ip":"101.227.131.20","logid":"1001"}
{"src_ip":"36.49.111.56","logid":"1002"}
{"src_ip":"120.35.215.100","logid":"1003"}
{"src_ip":"183.242.45.14","logid":"1004"}

logstash处理后输出如下：

{
      "location" => {
         "region_name" => "Beijing",
            "latitude" => 39.9143,
        "country_name" => "China",
           "longitude" => 116.3861,
           "city_name" => "Beijing"
    },
          "host" => "10.7.2.235",
        "src_ip" => "125.33.203.100",
      "@version" => "1",
    "@timestamp" => 2023-07-04T06:49:33.333Z,
       "message" => "{\"src_ip\":\"125.33.203.100\",\"logid\":\"1000\"}\n",
         "logid" => "1000"
}
{
      "location" => {
         "region_name" => "Beijing",
            "latitude" => 39.9143,
        "country_name" => "China",
           "longitude" => 116.3861,
           "city_name" => "Beijing"
    },
          "host" => "10.7.2.235",
        "src_ip" => "183.242.45.14",
      "@version" => "1",
    "@timestamp" => 2023-07-04T06:49:33.881Z,
       "message" => "{\"src_ip\":\"183.242.45.14\",\"logid\":\"1004\"}\n",
         "logid" => "1004"
}
{
      "location" => {
            "latitude" => 34.7732,
        "country_name" => "China",
           "longitude" => 113.722
    },
          "host" => "10.7.2.235",
        "src_ip" => "36.49.111.56",
      "@version" => "1",
    "@timestamp" => 2023-07-04T06:49:33.365Z,
       "message" => "{\"src_ip\":\"36.49.111.56\",\"logid\":\"1002\"}\n",
         "logid" => "1002"
}
{
      "location" => {
         "region_name" => "Fujian",
            "latitude" => 24.4798,
        "country_name" => "China",
           "longitude" => 118.0819,
           "city_name" => "Xiamen"
    },
          "host" => "10.7.2.235",
        "src_ip" => "120.35.215.100",
      "@version" => "1",
    "@timestamp" => 2023-07-04T06:49:33.365Z,
       "message" => "{\"src_ip\":\"120.35.215.100\",\"logid\":\"1003\"}\n",
         "logid" => "1003"
}
{
      "location" => {
            "latitude" => 34.7732,
        "country_name" => "China",
           "longitude" => 113.722
    },
          "host" => "10.7.2.235",
        "src_ip" => "101.227.131.20",
      "@version" => "1",
    "@timestamp" => 2023-07-04T06:49:33.333Z,
       "message" => "{\"src_ip\":\"101.227.131.20\",\"logid\":\"1001\"}\n",
         "logid" => "1001"
}

由于geoip为免费IP库，存在部分IP无法获取到省份城市信息