hive UDF 根据ip解析地理位置信息

hive UDF 根据ip查询对应地理位置信息

最终效果

具体可返回信息：洲,国家,省,市，区,运营商,行政编码,国家英文名,国家英文缩写,经纬度，以下为按需求进行选择性显示，可更改打印信息为全部。
输入：

202.203.78.210 

返回:

{continent='亚洲', country='中国', province='云南', city='昆明', District=''}

基本原理：
ip地址为四段字符串组成，一般同一个子网的ip同属于一个区域。如202.203.78.210/24 和202.203.78.220/24同属于202.203.78.0这个子网 属于云南昆明这个区域。所以为了方便查找我们就可以将ip字符串转为long类型整数来进行查找。
我们会将ip地址库加载进内存，然后使用二分查找来进行搜素并返回结果。

基本要求
1.hive依赖
2.ip地址库

具体实现

1. 依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cn.spj</groupId>
    <artifactId>com.jiayeli.hiveUDF</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>

</project>

2. ip2long

	/**
     * ip to long
     * @param ip
     * @return
     */
    private synchronized static Long ip2long(String ip) {
        Long result = 0l;
        String[] ipSegment = ip.split("\\.");
        //ip格式校验
        if (ipSegment.length != 4) {
            System.out.println("ip 格式错误！！！");
        }
        for (int i =0; i<ipSegment.length; i++) {
            result += Long.valueOf(ipSegment[i].isEmpty() ? "0" : ipSegment[i]) << (8*(3-i)) ;
        }
        return result;
    }

4. 二分查找

/**
     * 查找ip对应的地理位置信息
     * @param ip
     * @return
     */
    public String searchIpInfo(final String ip) {
        IPInfo startPoint = locationInfos.get(0);
        IPInfo endPoint = locationInfos.get(locationInfos.size()-1);
        IPInfo midItem;
        Long desIP = ip2long(ip);
        String info = "空";
        int mindIndex;
        int startIndex = 0;
        int endIndex = locationInfos.size()-1;

        if (desIP < startPoint.getStartIPMapLong() || desIP > endPoint.getEndIPMapLong())
            return "empty";
        int count = 1;
        while (startPoint.getEndIPMapLong() < endPoint.getEndIPMapLong()) {
            System.out.println(count++);

            if (startPoint.getStartIPMapLong() < desIP && startPoint.getEndIPMapLong() > desIP) {
                info =  startPoint.getLocationInfo().toString();
                return info;
            } else if (endPoint.getStartIPMapLong() < desIP && endPoint.getEndIPMapLong() > desIP) {
                info = endPoint.getLocationInfo().toString();
                return info;
            } else {
                mindIndex = startIndex +  (endIndex - startIndex)/2;
                midItem = locationInfos.get(mindIndex);
               if (midItem.getStartIPMapLong() > desIP){
                    endPoint = midItem;
                    endIndex = mindIndex;
                } else {
                    startIndex = mindIndex;
                    startPoint = midItem;
                }
            }

        }

        return info;
    }

5. ip地址库加载

 /**
     * 解析地理位置信息为模型
     * @param locationInfoStr
     * @return
     */
    private static synchronized IPInfo getIPInfo(String locationInfoStr) {

        String[] info = locationInfoStr.split(",");
        //错误信息处理
        if (info.length != 15) {
            System.out.println("不完整的地理位置信息，将被忽略解析：" + locationInfoStr);
            return null;
        }

        //映射为model
        return new IPInfo(
                info[0].isEmpty() ?  "" : info[0],
                info[1].isEmpty() ?  "" : info[0],
                Long.valueOf(info[2].isEmpty() ? "0" : info[2]),
                Long.valueOf(info[3].isEmpty() ? "0" : info[3]),
                new LocationInfo(info[4].isEmpty() ? "" : info[4],
                        info[5].isEmpty() ? "" : info[5],
                        info[6].isEmpty() ? "" : info[6],
                        info[7].isEmpty() ? "" : info[7],
                        info[8].isEmpty() ? "" : info[8]
                ),
                info[9].isEmpty() ? "" : info[9],
                info[11].isEmpty() ? "" : info[11],
                Long.valueOf(info[10].isEmpty() ? "0" : info[10]),
                info[12].isEmpty() ? "" : info[12],
                Float.valueOf(info[13].isEmpty() ? "0" : info[13]),
                Float.valueOf(info[14].isEmpty() ? "0" : info[14])
        );

    }

 /**
     * 加载ip信息文件
     * @param path
     */
    public static void loadIPInfo(String path) {

        File file = new File(path);
        BufferedReader br = null;

        try {
            br = new BufferedReader(new InputStreamReader(new FileInputStream(file), "UTF-8"));
            String line = null;
            while ((line = br.readLine()) != null) {
                locationInfos.add(getIPInfo(line));
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                br.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

源码

gitee.