logstash7.15.1 导入ngnix日志在kibana根据ip在地图进行统计展示

1. 配置logstash

input {
  file {
    start_position => beginning 
    path => "E:/logstash-test/access.log"
	type => "ngnix" ### 用去输出到es时判断存入哪个索引	
  } 
}


filter{
	grok{
	  match => { "message" => "%{HTTPD_COMBINEDLOG}" }
	} ### 通过grok匹配内容并将
	date {        
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
		target => "@timestamp"
    }
	geoip {
        source => "clientip"
        # 指定需要的字段
        # fields => ["country_name", "continent_code", "region_name", "city_name", "latitude", "longitude"]
		target => "geoip"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]		
    }
	
	mutate {
               convert => [ "[geoip][coordinates]", "float"]
           }
	
}


output {
  stdout{}
  elasticsearch {
   #es地址，可多个
   hosts => ["localhost:9200"]
    action => "index"
	#获取输出参数"indexname"值当做索引，如果没有则会自动创建对应索引(需要es开启自动创建索引)
    index => "logstash-nginxlogs-geoip-%{+YYYY-MM}"
   }
}

2. 执行logstash，导入数据

logstash -f logstash3-ngnix-geoip.conf

控制台输出如下：

{
       "@version" => "1",
       "clientip" => "39.144.0.219",
        "request" => "/app-cweb/api/cpn/v1/districts",
          "agent" => "\"Mozilla/5.0 (Linux; U; Android 11; zh-CN; Redmi K30 5G Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.18 Mobile Safari/537.36 UCBS/3.22.2.18_210803145558 ChannelId(5) NebulaSDK/1.8.100112 Nebula Bankabc/Portal BankabcAndroid/7.1.0 SDKVersion/30  mPaaSClient\"",
           "type" => "type1",
          "bytes" => "0",
    "httpversion" => "1.1",
          "ident" => "-",
           "host" => "DESKTOP",
           "path" => "E:/logstash-test/access.log",
           "auth" => "-",
           "verb" => "POST",
       "referrer" => "\"https://www.appserver.com/mnt/qd-mall/\"",
       "response" => "499",
     "@timestamp" => 2022-02-28T22:55:00.000Z,
      "timestamp" => "01/Mar/2022:06:55:00 +0800",
        "message" => "39.144.0.219 - - [01/Mar/2022:06:55:00 +0800] \"POST /app-cweb/api/cpn/v1/districts HTTP/1.1\" 499 0 \"https://www.appserver.com/mnt/qd-mall/\" \"Mozilla/5.0 (Linux; U; Android 11; zh-CN; Redmi K30 5G Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.18 Mobile Safari/537.36 UCBS/3.22.2.18_210803145558 ChannelId(5) NebulaSDK/1.8.100112 Nebula Bankabc/Portal BankabcAndroid/7.1.0 SDKVersion/30  mPaaSClient\" \"-\"",
          "geoip" => {
           "region_code" => "SD",
             "longitude" => 115.4719,
         "country_code2" => "CN",
           "coordinates" => [
            [0] 115.4719,
            [1] 35.2394
        ],
              "timezone" => "Asia/Shanghai",
              "location" => {
            "lon" => 115.4719,
            "lat" => 35.2394
        },
                    "ip" => "39.144.0.219",
        "continent_code" => "AS",
         "country_code3" => "CN",
             "city_name" => "Heze",
          "country_name" => "China",
              "latitude" => 35.2394,
           "region_name" => "Shandong"
    }
}

3. 在kibana创建可视化

3.  选择index pattern

4.  设置统计方式

5. 点击应用按钮， 查看统计图

图上的圆点就是访问数据统计。可以看出主要在亚洲和北美。

通过放大地图，可以进一步看到更小范围的统计。

 6. jdk版本问题

在logstash 7.15.1使用geoip插件时，该插件使用jdk11编译，所以使用jdk8启动时，会出现报错，提示jdk版本不匹配。

7. 索引名称

当索引名称不是以logstash-开头的时候，统计的时候会出现以下错误：

这个错误提示很难理解。

The index pattern nginxlogs-geoiptest-* does not contain any of the following compatible field types: geo_point

将索引名称修改为logstash-开头以后，问题就解决了。