hive操作elasticsearch

一、es添加数据

curl -XPUT 'masterhadoop:9600/customer/user/00002' -d '{"name":"lisi","sex":"M","phone":"13379434343","age":20}';

二、进入到hive cli命令控制台，添加依赖的jar

add jar file:home/elasticsearch-hadoop-5.3.1/dist/elasticsearch-hadoop-hive-5.3.1.jar;

三、创建es数据对应的hive关联表

1）es无searchguard插件的

CREATE EXTERNAL TABLE es_customer_user(

 id STRING,

 name STRING,

 phone STRING,

 sex STRING,

 age BIGINT

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

 'es.nodes'='masterhadoop:9600',

 'es.index.auto.create'='true',

 'es.resource'='customer/user',

 'es.read.metadata' = 'true',

 'es.mapping.names'='id:_metadata._id,name:name,phone:phone,sex:sex,age:age'

);

2）es集成searchgurad插件的情况：

CREATE EXTERNAL TABLE es_customer_user_2(

 id STRING,

 name STRING,

 phone STRING,

 sex STRING,

 age BIGINT

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

 'es.nodes'='masterhadoop:9200',

 'es.net.ssl' = 'true',

 'es.net.ssl.keystore.location' = 'file:usr/local/test/elasticsearch-5.3.0/config/CN=node-1-keystore.jks',

 'es.net.ssl.keystore.pass' = 'changeit',

 'es.net.ssl.keystore.type' = 'JKS',

 'es.net.ssl.truststore.location' = 'file:usr/local/test/elasticsearch-5.3.0/config/truststore.jks',

 'es.net.ssl.truststore.pass' = 'changeit',

 'es.net.http.auth.user' = 'admin',
 'es.net.http.auth.pass' = 'admin',

 'es.index.auto.create'='false',

 'es.resource'='customer/user',

 'es.read.metadata' = 'true',

 'es.mapping.names'='id:_metadata._id,name:name,phone:phone,sex:sex,age:age'

);

三、验证

1) select * from es_customer_user;

2) select * from es_customer_user_2;

三、备注

如果在同一个hive cli上下文，映射二种不同环境的es（es和es+searchguard），其中会有一个查询会报错的，貌似是由于有些变量是共用的，所以最好开二个cli进行操作，当然实际情况中肯定是同样的es环境。