Hbase协处理器实践总结（hbase数据同步）

最新推荐文章于 2021-12-10 16:57:24 发布

☞空白页

最新推荐文章于 2021-12-10 16:57:24 发布

阅读量190

点赞数

分类专栏： Hbase

本文链接：https://blog.csdn.net/weixin_44441757/article/details/113174423

版权

本文总结了使用Hbase协处理器将数据实时同步到Elasticsearch的实践过程，包括版本介绍、协处理器代码示例、动态加载部署方法及遇到的问题与解决策略。在Hbase 1.2.0、ES 6.8.5和Hadoop 2.6.0环境下，通过编写并部署Coprocessor实现数据同步，特别强调了版本匹配、JDK一致性以及特定配置的重要性。

摘要由CSDN通过智能技术生成

Hbase协处理器实践总结（hbase数据同步）

前景提示：

使用背景

使用hbase同步数据到es，每次hbase客户端发送put请求后，触发协处理器将数据同步到es。

版本介绍

hbase版本：1.2.0
es版本：6.8.5
hadoop版本：2.6.0

话不多说，直接上代码。有关协处理器了解请参考底部官网连接和相关博客。

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>Hbase-Observer-ES</groupId>
    <artifactId>Hbase-Observer-ES</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       
        <!--<hadoop.version>2.7.3</hadoop.version>-->
        <hadoop.version>2.6.0</hadoop.version>
        <fastjson.version>1.2.70</fastjson.version>
        <elasticsearch.version>6.8.5</elasticsearch.version>
        <hbase.version>1.0.0</hbase.version>
        <!--<hbase.version>1.2.6</hbase.version>-->
        <http.version>4.5.7</http.version>
        <log4j2.version>2.10.0</log4j2.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>${fastjson.version}</version>
        </dependency>



        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId> elasticsearch-cli</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-hbase-handler</artifactId>
            <version>2.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>${http.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.13</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-common -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>${hbase.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>${hbase.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            &