【DataX】将hive表数据导入ES

最新推荐文章于 2024-05-16 19:06:18 发布

郝少

最新推荐文章于 2024-05-16 19:06:18 发布

阅读量1.2k

点赞数

分类专栏： DataX 文章标签： hive elasticsearch 大数据

本文链接：https://blog.csdn.net/hyj_king/article/details/130706510

版权

DataX 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

一、环境

DataX：windows安装

Es版本：7.9.0

二、创建hive测试表

CREATE TABLE teacher(
 name string,
 age int
)row format delimited fields terminated by ','
stored as orc;

insert into teacher(name,age) values("zhangsn",22);
insert into teacher(name,age) values("lisi",30);
insert into teacher(name,age) values("wangwu",66);
insert into teacher(name,age) values("lihua",15);

三、Es写入插件包

下载地址

将es写入插件包放进plugin\writer文件夹下。

四、配置json

hive_es.json:

{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            }
        },
        "content": [
            {
                "reader": {
                    "name": "hdfsreader",
                    "parameter": {
                        "path": "/user/hive/warehouse/teacher/*",
                        "defaultFS": "hdfs://192.168.xx.xx:8020",
                        "column": [
                               {"index": 0,   "type": "string" },
                               {"index": 1,   "type": "long" }
                        ],
                        "fileType": "orc",
                        "encoding": "UTF-8",
                        "fieldDelimiter": ","
                    }
                },
                "writer": {
                  "name": "elasticsearchwriter",
                  "parameter": {
                    "endpoint": "http://192.168.xx.xx:9200/",
                    "accessId": "123",
                    "accessKey": "123",
                    "cleanup": true,
                    "index":"teacher",
                    "type":"_doc",
                    "settings": {
                        "settings":{
                            "index":{
                                "mapping":{"total_fields":{"limit":2000}},
                                "number_of_replicas":2,
                                "number_of_shards":10
                            }
                        }
                    },
                    "batchSize": 1000,
                    "splitter": ",",
                    "column": [
                      {"name":"name","type":"string"},
                      {"name":"age","type":"long"}
                    ]
                  }
                }
            }
        ]
    }
}

注意：

1、将配置好的hive_es.json文件放到job文件夹下；

2、hive的读取配置是hdfsreader，在hive中使用show create table teacher命令可以查看teacher表在hdfs中的位置，得到这个位置可以配置hdfsreader的defaultFs和path；

3、hive表的int类型对应hdfsreader字段配置的long,否则数据同步失败；ES也是对应为long;

4、可以使用下面命令查找读取、写入的标准模板（前提是对应reader和writer文件夹中的插件文件夹中要有plugin_job_template.json文件模板）：

D:\workSoftWare\datax\datax\bin>datax.py -r mysqlreader -w elasticsearchwriter

五、数据同步

1、执行命令

D:\workSoftWare\datax\datax\bin>python  datax.py  ../job/hive_es.json

2、查看es结果

这里使用kibana查询：

GET /teacher/_search

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "teacher",
        "_type" : "_doc",
        "_id" : "pnhlI4gBUNFFJN_x_IxO",
        "_score" : 1.0,
        "_source" : {
          "name" : "wangwu",
          "age" : 66
        }
      },
      {
        "_index" : "teacher",
        "_type" : "_doc",
        "_id" : "o3hlI4gBUNFFJN_x9Yw7",
        "_score" : 1.0,
        "_source" : {
          "name" : "zhangsn",
          "age" : 22
        }
      },
      {
        "_index" : "teacher",
        "_type" : "_doc",
        "_id" : "onhlI4gBUNFFJN_x74zU",
        "_score" : 1.0,
        "_source" : {
          "name" : "lisi",
          "age" : 30
        }
      },
      {
        "_index" : "teacher",
        "_type" : "_doc",
        "_id" : "pHhlI4gBUNFFJN_x-IyH",
        "_score" : 1.0,
        "_source" : {
          "name" : "lihua",
          "age" : 15
        }
      }
    ]
  }
}

同步成功！！！