【HBase-Coprocessor】HBase协处理器同步数据至Elasticsearch

已于 2024-01-03 15:20:32 修改
阅读量358 收藏 1
点赞数 2
分类专栏： NoSql 文章标签： elasticsearch hbase java
于 2023-02-24 17:33:15 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_53543905/article/details/129167121
版权

【HBase-Coprocessor】HBase协处理器同步数据至Elasticsearch

1）代码编写

1.1.Pom依赖导入

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>hbase-observer-es</groupId>
    <artifactId>langya</artifactId>
    <version>20190921</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <java.version>1.8</java.version>
        <spark.cloudera.version>2.10.cloudera2</spark.cloudera.version>
        <hbase.cloudera.version>1.2.0-cdh5.13.3</hbase.cloudera.version>
        <maven-assembly-plugin.version>2.6</maven-assembly-plugin.version>


    </properties>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.40</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.6.1</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>7.6.1</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.2.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.2.0</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>${maven-assembly-plugin.version}</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass></mainClass>
                        </manifest>
                    </archive>
                    <descriptors>
                        <descriptor>assembly.xml</descriptor>
                    </descriptors>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

1.2.assembly.xml

<?xml version="1.0" encoding="UTF-8"?>
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
    <id>test</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <unpack>true</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <directory>${project.build.outputDirectory}</directory>
            <outputDirectory>./</outputDirectory>
        </fileSet>
    </fileSets>
</assembly>

1.3.Elasticsearch相关代码

1.3.1.ElasticsearchPoolUtil

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;

import java.net.InetAddress;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;

public class ElasticsearchPoolUtil {
    private static final Log LOGGER = LogFactory.getLog(ElasticsearchPoolUtil.class);

    private static String ID = "id";
    private static String INDEX = "index";
    private static String TYPE = "type";
    private static String DATAJSON = "dataJson";
    public static Integer MAX_CONNECT_SIZE=1;//默认最小连接数
    private static Integer MAX_RETRY_SIZE=600;//默认重试次数

    //静态的Connection队列
    private static LinkedList<TransportClient> clientQueue = null;

    public static String CLUSTER_NAME="itcast-es";

    private static String ES_NODE_ONE="192.168.32.61";
    private static String ES_PORT_ONE="9200";
    public static List<String> esHostTcpList = new ArrayList<>();

    public synchronized static TransportClient getClient() throws Exception{

        if(clientQueue ==null ){
            clientQueue = new LinkedList<>();
            for(int i=0;i<MAX_CONNECT_SIZE;i++){
                clientQueue.push(clientPush());
            }
        }else if(clientQueue.size() == 0){
            clientQueue.push(clientPush());
        }
        return clientQueue.poll();

    }

    public static TransportAddress[] initTranSportAddress(){
        TransportAddress[] transportAddresses = new TransportAddress[esHostTcpList.size()];
        int offset = 0;
        for(int i=0;i<esHostTcpList.size();i++){
            String[] ipHost = esHostTcpList.get(i).split(":");
            try{
                transportAddresses[offset] = new TransportAddress(InetAddress.getByName(ipHost[0].trim()),Integer.valueOf(ipHost[1].trim()));
                offset++;

            }catch(Exception e){
                LOGGER.error("exec init transport address error:",e);
            }
        }
        return transportAddresses;
    }

    public static void pilotConnection(){
        synchronized (clientQueue){
            long startTime = System.currentTimeMillis();
            LOGGER.warn("正在啟動控制連接，目前連接數量為："+clientQueue.size());
            if(clientQueue.size()>MAX_CONNECT_SIZE){
                clientQueue.getLast().close();
                clientQueue.removeLast();
                LOGGER.warn("關閉連接耗時："+(System.currentTimeMillis()-startTime));
                pilotConnection();
            }else{
                return;
            }
        }
    }

    public static void destoryAllConnection(){
        LOGGER.warn("正在銷毀連接，目前連接數為"+clientQueue.size());
        synchronized (clientQueue){
            long startTime = System.currentTimeMillis();
            if(clientQueue.size()>0){
                clientQueue.getLast().close();
                clientQueue.removeLast();
                LOGGER.warn("關閉連接耗時："+(System.currentTimeMillis()-startTime));
                pilotConnection();
            }else{
                return;
            }
        }
    }

    private synchronized static TransportClient clientPush() throws Exception{

        TransportClient client = null;
        int upCount = 0;
        while(clientQueue.size()<MAX_CONNECT_SIZE && client == null && upCount < MAX_RETRY_SIZE){
            client = init();
            Thread.sleep(100);
            upCount++;
        }
        if(client == null){
            throw new Exception("Es client init failed wait for 60s");
        }
        return client;
    }

    public static TransportClient init()throws Exception{
        System.setProperty("es.set.netty.runtime.available.processors","true");
        Settings esSettings = Settings.builder().put("cluster.name",CLUSTER_NAME).put("client.transport.sniff",false).build();
        TransportClient client = new PreBuiltTransportClient(esSettings);
        client.addTransportAddresses(initTranSportAddress());
        return client;
    }

    public synchronized static void returnClient(TransportClient client){
        if(clientQueue ==null){
            clientQueue  = new LinkedList<>();
        }
        clientQueue.push(client);
    }
}

1.3.2.ElasticsearchUtil

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;

import java.net.InetAddress;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;

public class ElasticsearchUtil {
    private static final Log LOGGER = LogFactory.getLog(ElasticsearchUtil.class);

    private static String ID = "id";
    private static String INDEX = "index";
    private static String TYPE = "type";
    private static String DATAJSON = "dataJson";
    public static Integer MAX_CONNECT_SIZE=1;//默认最小连接数
    private static Integer MAX_RETRY_SIZE=600;//默认重试次数

    //静态的Connection队列
    private static LinkedList<TransportClient> clientQueue = null;

    public static String CLUSTER_NAME="itcast-es";

    private static String ES_NODE_ONE="192.168.32.61";
    private static String ES_PORT_ONE="9200";
    public static List<String> esHostTcpList = new ArrayList<>();

    public synchronized static TransportClient getClient() throws Exception{

        if(clientQueue ==null ){
            clientQueue = new LinkedList<>();
            for(int i=0;i<MAX_CONNECT_SIZE;i++){
                clientQueue.push(clientPush());
            }
        }else if(clientQueue.size() == 0){
            clientQueue.push(clientPush());
        }
        return clientQueue.poll();

    }

    public static TransportAddress[] initTranSportAddress(){
        TransportAddress[] transportAddresses = new TransportAddress[esHostTcpList.size()];
        int offset = 0;
        for(int i=0;i<esHostTcpList.size();i++){
            String[] ipHost = esHostTcpList.get(i).split(":");
            try{
                transportAddresses[offset] = new TransportAddress(InetAddress.getByName(ipHost[0].trim()),Integer.valueOf(ipHost[1].trim()));
                offset++;

            }catch(Exception e){
                LOGGER.error("exec init transport address error:",e);
            }
        }
        return transportAddresses;
    }

    public static void pilotConnection(){
        synchronized (clientQueue){
            long startTime = System.currentTimeMillis();
            LOGGER.warn("正在啟動控制連接，目前連接數量為："+clientQueue.size());
            if(clientQueue.size()>MAX_CONNECT_SIZE){
                clientQueue.getLast().close();
                clientQueue.removeLast();
                LOGGER.warn("關閉連接耗時："+(System.currentTimeMillis()-startTime));
                pilotConnection();
            }else{
                return;
            }
        }
    }

    public static void destoryAllConnection(){
        LOGGER.warn("正在銷毀連接，目前連接數為"+clientQueue.size());
        synchronized (clientQueue){
            long startTime = System.currentTimeMillis();
            if(clientQueue.size()>0){
                clientQueue.getLast().close();
                clientQueue.removeLast();
                LOGGER.warn("關閉連接耗時："+(System.currentTimeMillis()-startTime));
                pilotConnection();
            }else{
                return;
            }
        }
    }

    private synchronized static TransportClient clientPush() throws Exception{

        TransportClient client = null;
        int upCount = 0;
        while(clientQueue.size()<MAX_CONNECT_SIZE && client == null && upCount < MAX_RETRY_SIZE){
            client = init();
            Thread.sleep(100);
            upCount++;
        }
        if(client == null){
            throw new Exception("Es client init failed wait for 60s");
        }
        return client;
    }

    public static TransportClient init()throws Exception{
        System.setProperty("es.set.netty.runtime.available.processors","true");
        Settings esSettings = Settings.builder().put("cluster.name",CLUSTER_NAME).put("client.transport.sniff",false).build();
        TransportClient client = new PreBuiltTransportClient(esSettings);
        client.addTransportAddresses(initTranSportAddress());
        return client;
    }

    public synchronized static void returnClient(TransportClient client){
        if(clientQueue ==null){
            clientQueue  = new LinkedList<>();
        }
        clientQueue.push(client);
    }
}

1.3.3.ElasticsearchBulkOperator

import com.langya.elasticsearch.util.*;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.elasticsearch.action.DocWriteRequest;
import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequestBuilder;
import org.elasticsearch.action.update.UpdateRequestBuilder;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.unit.TimeValue;

import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

public class ElasticsearchBulkOperator {
    private static final Log LOGGER = LogFactory.getLog(ElasticsearchBulkOperator.class);
    private static final Integer BULK_TIME_OUT = 5;//批次请求超时时间，单位：分钟

    private static final Integer CORE_POOL_SIZE = 3;//线程池大小

    private static final Integer PUT_INITIAL_DELAY = 10;//单位：秒
    private static final Integer PUT_PREIOD = 10;

    private static Integer idleIimeAccumulation = 0;//协处理器空闲时间积累，达到半小时就会断掉所有的连接，将所有的连接释放，减轻Elasticsearch的和HBase的压力

    private static final Integer SERVICE_MAXIMUM_IDLE_TIME = 1800;//服务最大空闲时间，单位：秒，默认半小时

    private static final Integer DESTOR_CONNECTION_INITIAL_DELAY = 10;//单位：秒

    private static final Integer DESTOR_CONNECTION_PERIOD = 3;//单位：秒，销毁多余连接检查间隔

    public static int MAX_BULK_COUNT = 10000;//最大批次提交，单位：条，用来限制不断写入的时候请求队列大小（通过安装协处理器的时候传递参数可以改变此默认值）

    public static int MIN_BULK_COUNT = 0;//最小批次提交大小，单位：条，用在周期性提交请求（周期性任务执行时，只要数据量 > 0条，就会提交），为了避免长时间不进数据，又没有达到最大请求队列限制，导致数据延迟问题（通过安装协处理器传递参数可以改变此默认值）

    private static BulkRequestBuilder bulkRequestBuilder = null;

    private static final Lock commitLock = new ReentrantLock();

    private static ScheduledExecutorService scheduledExecutorService;

    static {
        bulkRequestBuilder();
        scheduledExecutorService = Executors.newScheduledThreadPool(CORE_POOL_SIZE);
        final Runnable beeper = () -> run();
        final Runnable destoryRedundantConnection = () -> destoryRedundantConnections();
        scheduledExecutorService.scheduleAtFixedRate(beeper, PUT_INITIAL_DELAY, PUT_PREIOD, TimeUnit.SECONDS);//定时刷新数据到Elasticsearch中
        scheduledExecutorService.scheduleAtFixedRate(destoryRedundantConnection, DESTOR_CONNECTION_INITIAL_DELAY, DESTOR_CONNECTION_PERIOD, TimeUnit.SECONDS);//定时清除多余Elasticsearch连接

    }

    private static void serviceIdleCheck() {
        synchronized (bulkRequestBuilder) {
            System.out.println("写入队列当前大小：" + bulkRequestBuilder.numberOfActions() + "   服务空闲时间累积:" + idleIimeAccumulation + "秒");
            if (bulkRequestBuilder.numberOfActions() == 0) {
                if (idleIimeAccumulation < SERVICE_MAXIMUM_IDLE_TIME) {
                    idleIimeAccumulation += DESTOR_CONNECTION_PERIOD;
                } else {
                    ElasticsearchPoolUtil.destoryAllConnection();
                    idleIimeAccumulation = 0;
                }
            } else {
                idleIimeAccumulation = 0;
            }
        }
    }

    private static void bulkRequestBuilder() {
        try {

            TransportClient client = ElasticsearchPoolUtil.getClient();
            bulkRequestBuilder = client.prepareBulk();
            ElasticsearchPoolUtil.returnClient(client);

        } catch (Exception e) {
            LOGGER.error(e);
        }
        bulkRequestBuilder.setTimeout(TimeValue.timeValueMinutes(BULK_TIME_OUT));
    }

    private static void run() {
        try {
            commitLock.lock();
            bulkRequest(MIN_BULK_COUNT);
        } catch (Exception ex) {
            LOGGER.error("Time Bulk index error:" + ex.getMessage());
        } finally {
            commitLock.unlock();
        }

    }


    private static void destoryRedundantConnections() {
        ElasticsearchPoolUtil.pilotConnection();
        serviceIdleCheck();
    }

    public static void shutdownScheduEx() {
        if (null != scheduledExecutorService && !scheduledExecutorService.isShutdown()) {
            scheduledExecutorService.shutdown();
        }
    }

    private static void bulkRequest(int threshold) {
        if (bulkRequestBuilder.numberOfActions() > threshold) {
            try {
                BulkResponse bulkItemResponses = bulkRequestBuilder.execute().actionGet();
                if (!bulkItemResponses.hasFailures()) {
                    TransportClient client = ElasticsearchPoolUtil.getClient();
                    bulkRequestBuilder = client.prepareBulk();
                    LOGGER.info("批次提交");
                    ElasticsearchPoolUtil.returnClient(client);
                }
            } catch (Exception ex) {
                try {
                    TransportClient client = ElasticsearchPoolUtil.getClient();
                    List<DocWriteRequest<?>> tempRequests = bulkRequestBuilder.request().requests();
                    bulkRequestBuilder = client.prepareBulk();
                    bulkRequestBuilder.request().add(tempRequests);
                    ElasticsearchPoolUtil.returnClient(client);
                } catch (Exception es) {
                    LOGGER.error(es);
                }
                LOGGER.error(ex);
            }
        }
    }

    public static void addUpdateBuilderToBulk(UpdateRequestBuilder builder) {
        commitLock.lock();
        try {
            bulkRequestBuilder.add(builder);
            bulkRequest(MAX_BULK_COUNT);
        } catch (Exception e) {
            LOGGER.error(e);
        } finally {
            commitLock.unlock();
        }
    }

    public static void addDeleteBuilderToBulk(DeleteRequestBuilder builder) {
        commitLock.lock();
        try {
            bulkRequestBuilder.add(builder);
            bulkRequest(MAX_BULK_COUNT);
        } catch (Exception e) {
            LOGGER.error(e);
        } finally {
            commitLock.unlock();
        }
    }
}

1.4.HBase相关代码

1.4.1.HbaseDataSyncEsObserver

import org.apache.commons.collections.map.HashedMap;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.CoprocessorEnvironment;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
import org.apache.hadoop.hbase.coprocessor.ObserverContext;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
import org.apache.hadoop.hbase.util.Bytes;
import org.elasticsearch.action.delete.DeleteRequestBuilder;
import org.elasticsearch.action.update.UpdateRequestBuilder;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.transport.client.PreBuiltTransportClient;

import java.io.IOException;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;

public class HbaseDataSyncEsObserver extends BaseRegionObserver {
    private static final Log LOGGER = LogFactory.getLog(HbaseDataSyncEsObserver.class);

    private String indexName;
    private String indexType;

    @Override
    public void start(CoprocessorEnvironment e) throws IOException {
        super.start(e);
        initIndexConfiguraction(e);
    }

    private void initIndexConfiguraction(CoprocessorEnvironment e){
        loadEsClientInfo(e);
    }

    private void loadEsClientInfo(CoprocessorEnvironment e){

        //集群名称
        ElasticsearchPoolUtil.CLUSTER_NAME = e.getConfiguration().get("cluster.name","itcast-es");
        this.setIndexName(e.getConfiguration().get("indexName",""));
        this.setIndexType(e.getConfiguration().get("indexType",""));

        //只需要改这里即可
//        String esClientInfo = e.getConfiguration().get("esClientInfo","192.168.1.1:9300-192.168.1.2:9300");
        String esClientInfo = e.getConfiguration().get("esClientInfo","10.70.71.52:9300");
        String[] esClientInfoList = esClientInfo.split("-");
        for(String esClientInfoTemp : esClientInfoList){
            ElasticsearchPoolUtil.esHostTcpList.add(esClientInfoTemp);
        }

    }

    @Override
    public void stop(CoprocessorEnvironment e) throws IOException {
        super.stop(e);
        ElasticsearchBulkOperator.shutdownScheduEx();
        ElasticsearchPoolUtil.destoryAllConnection();
    }

    @Override
    public void postPut(ObserverContext<RegionCoprocessorEnvironment> e, Put put, WALEdit edit, Durability durability) throws IOException {
        super.postPut(e, put, edit, durability);
        String rowKey = new String(put.getRow());
        NavigableMap<byte[],List<Cell>> familys = put.getFamilyCellMap();
        Map<String,Object> result = new HashedMap();
        for(Map.Entry<byte[],List<Cell>> entry : familys.entrySet()){
            for(Cell cell : entry.getValue()){
                String key = Bytes.toString(CellUtil.cloneQualifier(cell));
                String value = Bytes.toString(CellUtil.cloneValue(cell));
                result.put(key,value);
            }
        }

        try{

            TransportClient transportClient = ElasticsearchPoolUtil.getClient();
            UpdateRequestBuilder updateRequestBuilder = transportClient.prepareUpdate(this.indexName,this.indexType,rowKey);
            updateRequestBuilder.setDoc(result);
            updateRequestBuilder.setDocAsUpsert(true);
            ElasticsearchPoolUtil.returnClient(transportClient);
            ElasticsearchBulkOperator.addUpdateBuilderToBulk(updateRequestBuilder);
        }catch (Exception e1){
            LOGGER.error(e1);
        }

    }

    @Override
    public void postDelete(ObserverContext<RegionCoprocessorEnvironment> e, Delete delete, WALEdit edit, Durability durability) throws IOException {
        super.postDelete(e, delete, edit, durability);
        String rowKey = new String(delete.getRow());
        try{
            TransportClient transportClient = ElasticsearchPoolUtil.getClient();
            DeleteRequestBuilder deleteRequestBuilder = transportClient.prepareDelete(this.indexName,this.indexType,rowKey);
            ElasticsearchPoolUtil.returnClient(transportClient);
            ElasticsearchBulkOperator.addDeleteBuilderToBulk(deleteRequestBuilder);
        }catch (Exception e1){
            LOGGER.error(e1);
        }
    }

    public void setIndexName(String indexName) {
        this.indexName = indexName;
    }

    public void setIndexType(String indexType) {
        this.indexType = indexType;
    }

    public String getIndexName() {
        return indexName;
    }

    public String getIndexType() {
        return indexType;
    }
}

1.5.windows(cmd)打包

mvn assembly:assembly

2）使用步骤

2.1.准备工作

#进入HBase控制台
hbase shell

#创建HBase表
create 'tbl_profile','user'

2.2.开启协处理器

（1）停用表

disable 'tbl_profile'

（2）添加协处理器

参数说明：

  • table_att：任务名称
  • coprocessor：使用协处理器
  • hdfs:///user/hbase/lib/test.jar：HDFS上jar包位置
  • com.test.hbase.observer.util.HbaseDataSyncEsObserver：HbaseDataSyncEsObserver所在jar包路径
  • 1001：任务id
  • cluster.name：es集群名称
  • indexName：es索引名称
  • indexType：es索引类型
alter 'tbl_profile',METHOD=> 'table_att','coprocessor'=>'hdfs:///user/hbase/lib/langya-20190926-test.jar|com.langya.hbase.observer.util.HbaseDataSyncEsObserver|1001|cluster.name=itcast-es,indexName=tbl_profile,indexType=user'

（3）启用表

enable 'tbl_profile'

（4）查看表详细信息

desc 'test'

3）注意

1、如果成功了，就会显示用时多少秒。

2、如果替换的程序有问题或者替换语句有问题，这一步就会造成集群挂掉，如果集群挂掉，hbase是无法重启的，需要进行以下步骤。

① 在hbase-site.xml文件添加

<property>
	   <name>hbase.coprocessor.abortonerror</name>
	   <value>false</value>
</property>

② 重启HBase，报错会被抑制

3、注意修改代码中的ip地址。

4、jar包要上传到HDFS。

4）测试

put 'test','1','info:name','zhangsan'

然后查看es中出现同步后的数据，成功。

5）解绑协处理器

（1）禁用表

disable 'test'

（2）解绑

alter 'tbl_profile', METHOD => 'table_att_unset',NAME => 'coprocessor$1'

（3）启用表并查看是否解绑成功，desc后查看没有协处理器了就是解绑成功了

enable 'test'
desc 'test'
bmyyyyyy
关注
  • 2
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
hbase 导入到es_HBase 2.0 协处理器实现 ES 数据同步
weixin_27603469的博客
12-31 576
标签：hbase 2.0、elasticsearch、Coprocessor、协处理器在正式进行讲述实现之前，我觉得有必要说一下出发点。团队期初数据都是基于 HBase+Phoenix 这样架构进行持久化。随着业务的复杂性增加，对部分表的查询效率和查询条件多样性，提出了更高的要求。HBase+Phoenix 就会出现索引滥用。变更索引变的特别的频繁，同时一些数据客观的表，变更索引的代价是非常大的。...
使用Hbase协作器(Coprocessor)同步数据到ElasticSearch（hbase 版本 1.2.0-cdh5.8.0, es 2.4.0 版本）
11-28
使用Hbase协作器(Coprocessor)同步数据到ElasticSearch（hbase 版本 1.2.0-cdh5.8.0, es 2.4.0 版本）源代码
Elasticsearch-HBase-replication同步
05-11
用于hbase同步数据到es(replication机制)
HBase数据同步到ElasticSearch的方案
热门推荐
横云断岭的专栏
11-15 3万+
ElasticSearch的River机制 ElasticSearch自身提供了一个River机制，用于同步数据。 这里可以找到官方目前推荐的River： http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/ 但是官方没有提供HBase的River。 其实ES的River非常简单，就是一个用户打包好的j
hbase使用协处理器同步es
weixin_42348946的博客
07-31 2186
hbase同步到es hbase版本:1.1.2 es版本:6.3.2 一开始从网上找的也能用,但是发现运行一段时间发现报错,丢失数据,后来又从网上搜索加上自己整理的,终于成功了 不多说上代码 public class ElasticSearchBulkOperator { private static final Log LOG = LogFactory.getLog(ElasticS...
HBase-Elasticsearch 数据同步工具
最新发布
07-01
jdk timer定时全量同步 org.frameworkset.elasticsearch.imp.HBase2ESFullDemo jdk timer定时增量同步 org.frameworkset.elasticsearch.imp.HBase2ESScrollTimestampDemo ...HBase 到elasticsearch数据同步
HBase-coprocessor
04-28
协处理器通过宏和概括的方式，我们可以将协处理器定义为一个框架，该框架提供了一种在HBase中执行自定义代码的简便方法。 代表协处理器的最常用类比是“触发器/存储过程”和AOP。 协处理器可以开发为： 观察者协处理...
hbase-coprocessor-example
07-01
hbase-协处理器-示例此示例演示如何使用 HBase 协处理器和 Algebird monoid 实现分组聚合。 我们在这里使用的 HBase 版本是 0.94.18，与 AWS EMR 上可用的版本完全相同。在本地 HBase 应用程序中创建演示表下载并...
HBase-coprocessor.pptx
01-12
HBase Coprocessor 提供了丰富的扩展性，使得开发者可以根据业务需求定制数据处理逻辑，提高了系统的灵活性和效率。无论是Observer还是Endpoint，都提供了对HBase操作的强大干预能力，从而满足各种复杂的业务场景。...
HBase-Elasticsearch数据同步1
08-08
【HBase-Elasticsearch数据同步】是大数据领域中常见的数据迁移和整合操作，它涉及到两个重要的分布式存储系统：HBase和Elasticsearch。HBase是一个基于Google Bigtable设计的高可靠性、高性能、分布式的行存数据库...
Elasticsearch-HBase-River同步
05-11
用于hbase同步数据到es(river机制)
HBase协处理器
08-12 558
试着写了个协处理器。感觉协处理器不好使，一旦出错，搞得整个Hbase集群都挂了… 后来选择用Spark建索引放ES，来改善查询这块儿的效率 具体步骤有点忘，后面补充。 1. option java_package = "com.cloud.coprocessor"; option java_outer_classname = "RAProto"; option java_generic...
HBase-6.hbase 协处理器
艾文
01-26 2477
引入Hbase中的Coprocessor的原因 HBase作为列族数据库无法建立“二级索引”，难以执行求和、计数、排序等操作。为解决这些问题，HBase0.92 之后引入协处理器（Coprocessor），实现一些新特性，能够轻易建立二次索引、复杂过滤器、以及访问控制。 参考： http://blog.csdn.net/lifuxiangcaohui/article/details/3
使用datax将Hbase导入数据到ES
weixin_47303949的博客
12-16 828
{ "job": { "setting": { "speed": { "channel": 3 } }, "content": [ { "reader": { "name": "hbase20xsqlreader", "parameter": { "queryServer
Sqoop导入HBase，并借助Coprocessor协处理器同步索引到ES
weixin_30788731的博客
08-22 176
1.环境 Mysql 5.6 Sqoop 1.4.6 Hadoop 2.5.2 HBase 0.98 Elasticsearch 2.3.5 2.安装（略过） 3.HBase Coprocessor实现 HBase Observer import org.apache.commons.logging.Log; import org.apache.commons.lo...
面向高稳定，高性能之-Hbase数据实时同步到ElasticSearch(之二)
zhulangfly的专栏
06-22 7829
我在这篇focus在两个主题：如何支持多表同步共用一个jar包，如何持续稳定的与ES交互写入数据。 在 《 使用Hbase协作器(Coprocessor)同步数据到ElasticSearch》中作者把两个关键组件中的属性和方法都声明为static，这意味什么？类方法和属性在所有的线程中共享，源代码请参考该博客。 问题出来了，当你用如下传参数的方式绑定到多个表： alter 'tes
hbase 导入到es_ES+Hbase对接方案概述
weixin_39809584的博客
12-22 903
方案背景Hbase的索引方案有很多，越来越多的人开始选择ES+Hbase的方案，其实该方案并没有想象中那么完美，ES并发低，同时查询速度相对Hbase也慢很多,那为什么会选择他呢，它的写入比较快，如果一个宽表需要建20个索引，在数据导入时，hbase每秒导入20W，那么ES压力就是每秒400W，solr和hindex都不能解决该问题。所以对并发高的业务场景，还是使用华为HIndex这种方案，也可以...
【实战解析】基于HBase的大数据存储在京东的应用场景
开涛的博客
04-15 820
HBase是一个高可靠性、高性能、面向列、可伸缩的分布式存储系统，适用于结构化的存储，底层依赖于Hadoop的HDFS，利用HBase技术可在廉价PCServer上搭建起...
hbase 导入到es_HBase数据同步到ElasticSearch的方案
weixin_39883091的博客
12-22 832
ElasticSearch的River机制ElasticSearch自身提供了一个River机制，用于同步数据。这里能够找到官方眼下推荐的River：可是官方没有提供HBase的River。事实上ES的River很easy，就是一个用户打包好的jar包，ES负责找到一个node，并启动这个River。假设node失效了。会自己主动找另外一个node来启动这个River。public interfa...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值