ELK之获取用户地理位置

获取用户地理位置

---

• 利用GeoIP映射用户地理位置
• 直接处理日志中的经纬度信息

在Elasticsearch中，所有的数据都有一个类型，什么样的类型，就可以在其上做一些对应类型的特殊操作。geo信息中的location字段是经纬度，我们需要使用经纬度来定位地理位置；在Elasticsearch中，对于经纬度来说，要想使用Elasticsearch提供的地理位置查询相关的功能，就需要构造一个结构，并且将其类型属性设置为geo_point

利用GeoIP映射用户地理位置

之前在整理logstsh时提到过通过Geoip插件来获取日志中IP地址的相关地理位置信息（logstsh整理：https://blog.csdn.net/wfs1994/article/details/80071815#t1），以apache为例，这里贴一下一个完整的收集实例：

input {
        file {
                path => "/home/test/access_log"
                type => "access_log"
        }
}
filter {
        grok {
                match => {"message" => "%{COMBINEDAPACHELOG}"}
        }
        geoip {
                source => "clientip"     
                database => "/home/test/GeoLite2-City.mmdb"
                add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
                convert => [ "[geoip][coordinates]", "float"]
        }
}
output {
       stdout { codec => rubydebug }
       elasticsearch {
                hosts => [ "192.168.20.60:9200" ]
                index => "logstash-apache-access-%{+YYY.MM.dd}"
       }
}   

geoip插件相关配置说明：

source:   必须设置 
database: 如果未指定，则默认为Logstash附带的GeoLite2城市数据库。
default_database_type: 默认值是city。可接受参数：City和ASN。
target: 默认值是“geoip”，指定Logstash应该存储geoip数据的字段。 例如，如果您有src_ip和dst_ip字段，并希望两个IP的GeoIP信息，这可能是有用的。

默认预定义的模板必须只有匹配 logstash-* 的索引才会应用这个模板

kibana可视化：

直接处理日志中的经纬度信息

日志格式如下所示：

  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302411,863010030035393,39.2055473,117.1409302,10,66.1,1,12,2,9,1070,14.4,65,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302412,863010030039387,38.9934654,116.9013672,36,178.3,1,12,2,36,2628,14.1,93,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302413,863010030043751,39.2268219,117.0283508,36,152.6,1,12,2,37,769,14.2,69,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302412,863010030031244,,,0,-1.0,0,0,2,27,1113,14.4,87,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302413,863010030031269,38.9425621,116.9591980,48,71.7,1,12,2,48,1519,14.8,83,0,0,0,0.0,0.0,
  [com.test.nlbs.filter.MsgDecodeFilter] - SoutheastMessage:#,1521302414,863010030039387,38.9932823,116.9013748,35,178.6,1,12,2,35,2626,14.1,91,0,0,0,0.0,0.0,

首先先通过grok,csv插件获取到相关地理位置信息：

filter {
        grok {
                match => { "message" => "\[com\.test\.nlbs\.filter\.MsgDecodeFilter\] \- \SoutheastMessage\:(?<SoutheastMessa>(.*\n){0,}.*)" }
                remove_field => ["message"]
                add_field => {"head" =>"SoutheastMessage"}
        }
        csv {
                source => "SoutheastMessa"
                separator => ","
                columns => [ "#","time", "imei", "latitude", "longitude" ]
                remove_field => ["column6","column7","column8","column9","column10","column11","column12","column13","column14","column15","column16","column17","column18","column19","column20"]
        }
}

columns: 如果 columns未配置，或者没有指定足够的列，则缺省列名称为 “column1”， “column2”等。
remove_field: 移除指定的字段

获取信息如下图所示：

可以看出通过插件我们过滤出了日志中的经纬度信息,但是是由 “longitude” 和 “latitude” 两个字段表示的。如果想要使用Elasticsearch提供的地理位置查询相关的功能，就需要构造一个类型属性为geo_point的结构：

filter {
        grok {
                match => { "message" => "\[com\.test\.nlbs\.filter\.MsgDecodeFilter\] \- \SoutheastMessage\:(?<SoutheastMessa>(.*\n){0,}.*)" }
                remove_field => ["message"]
                add_field => {"head" =>"SoutheastMessage"}
        }
        csv {
                source => "SoutheastMessa"
                separator => ","
                columns => [ "#","time", "imei", "latitude", "longitude" ]
                remove_field => ["column6","column7","column8","column9","column10","column11","column12","column13","column14","column15","column16","column17","column18","column19","column20"]
                add_field => [ "location", "%{longitude}" ]
                add_field => [ "location", "%{latitude}" ]
        }

        mutate {
                convert => [ "location", "float"]
        }
}

获取的结果如下所示：

要构造一个类型属性为geo_point的结构需要定义mapping：

PUT /myindex 
{
  "mappings": {
    "doc": { 
      "properties": { 
        "location":  { "type": "geo_point"  }
        }
      }
    }
}

查看映射：

GET /myindex/_mapping
...
          "location": {
            "type": "geo_point"
          },
...

注意：这里要有先后顺序，先定义mapping

如果没有定义mapping，默认的话会是“text”类型，在可视化创建Coordinate Map时就会报“No Compatible Fields”错误：

最后使用高德地图将地图地名显示成中文：
在kibana配置文件中添加如下内容并重启kibana服务：

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

最后成果如下：

参考

http://www.51niux.com/?id=212
http://www.51niux.com/?id=213
http://blog.csdn.net/yanggd1987/article/details/50469113