mongo 数据映射到 hive 表中全过程

服务器环境:

        CDH 5.8 

        mongo 3.6.1 

1.下载依赖:

(1)http://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core/2.0.2

(2)http://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-hive/2.0.2

(3)https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver/3.6.1

2.将三个依赖分别放到服务器上

        /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib

        /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib

3.给jar包权限

        chmod 777 + (jar包名称)

4. 启动并进入hive 

         nohup hiveserver2 >/etc/hive-hcatalog/log/hiveServer2.log 2>&1 &

        beeline -u jdbc:hive2://localhost:10000 -n hadoop

5.在hive中创建外部表

mongo 数据为 :{ "_id": 0,"road_type": "bridleway"}

CREATE EXTERNAL TABLE roadnet_type_temp(id  INT , `road_type` STRING ) STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id"}') TBLPROPERTIES('mongo.uri'='mongodb://localhost:20000/basicdata.roadnet_type') ;

 注:WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id"}'):字段一样时可以不加,

hive的列自动修改为小写,mapping时要区分大小写

6.测试

        select * from roadnet_type_temp;

        select min(id) from roadnet_type_temp;

        如果跑mr时报错,在hive中执行如下命令

        add jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib/mongo-hadoop-core-2.0.2.jar;
        add jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib/mongo-hadoop-hive-2.0.2.jar;
         add jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib/mongo-java-driver-3.6.1.jar;

        (如果解决了hive中没加载jar包,每次进入hive客户端都要加载jar包,可以把jar包放在配置文件中,让它启动时自动加载)

[若有侵权，请及时告知随及修正或删除文章！]