hive导入es6.x

这是在开发环境中测试的hive到es的数据导入方法

[root@sage-bigdata-01 es6]# pwd

/opt/es6

1、上传到hdfs中

hdfs> hdfs dfs -mkdir /hive

hdfs> hdfs dfs -chown hive:hive /hive

hive> hdfs dfs -put commons-httpclient-3.1.jar /hive

hive> hdfs dfs -put elasticsearch-hadoop-6.3.0.jar /hive

2、将jar添加到hive中

hive> add jar hdfs://bigdata-01:8020/hive/elasticsearch-hadoop-6.3.0.jar;

hive> add jar hdfs://bigdata-01:8020/hive/commons-httpclient-3.1.jar;

3、创建外部表

hive> CREATE EXTERNAL TABLE persons_es (id STRING, age INT, name STRING) 

 STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

 TBLPROPERTIES('es.nodes'='172.16.2.115', 'es.resource' = 'hive/persons');

4、执行导入

hive> INSERT OVERWRITE TABLE persons_es select * from bigdata.persons;

Logs like this:

Query ID = hive_20180625172002_058cad13-89a3-4bad-aabd-abad5e105fb3

Total jobs = 1

Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1526267537875_0112)

--------------------------------------------------------------------------------

        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

--------------------------------------------------------------------------------

Map 1 ..........   SUCCEEDED      1          1        0        0       2       0

--------------------------------------------------------------------------------

VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 16.94 s    

--------------------------------------------------------------------------------

OK

Time taken: 19.925 seconds

5、help

TBLPROPERTIES Settings: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html