Flink HBaseLookupFunction

最新推荐文章于 2024-05-16 10:36:54 发布
最新推荐文章于 2024-05-16 10:36:54 发布
阅读量726 收藏
点赞数
分类专栏： Flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_44865574/article/details/115550535
版权

Flink HBaseLookupFunction

Flink版本：1.12.2

Flink源码内有HbaseLookupFuncation类，最近想试试kafka 流数据实时关联hbase维表数据，看看使用HbaseLookupFuncation能否成功使用，于是稍微研究：

1. Flink源码:

HBaseLookupFunction

package org.apache.flink.connector.hbase.source;

import org.apache.flink.annotation.Internal;
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.connector.hbase.util.HBaseConfigurationUtil;
import org.apache.flink.connector.hbase.util.HBaseReadWriteHelper;
import org.apache.flink.connector.hbase.util.HBaseTableSchema;
import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.types.Row;
import org.apache.flink.util.StringUtils;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

/**
 * The HBaseLookupFunction is a standard user-defined table function, it can be used in tableAPI and
 * also useful for temporal table join plan in SQL. It looks up the result as {@link Row}.
 */
@Internal
public class HBaseLookupFunction extends TableFunction<Row> {
    private static final Logger LOG = LoggerFactory.getLogger(HBaseLookupFunction.class);
    private static final long serialVersionUID = 1L;

    private final String hTableName;
    private final byte[] serializedConfig;
    private final HBaseTableSchema hbaseTableSchema;

    private transient HBaseReadWriteHelper readHelper;
    private transient Connection hConnection;
    private transient HTable table;

    public HBaseLookupFunction(
            Configuration configuration, String hTableName, HBaseTableSchema hbaseTableSchema) {
        this.serializedConfig = HBaseConfigurationUtil.serializeConfiguration(configuration);
        this.hTableName = hTableName;
        this.hbaseTableSchema = hbaseTableSchema;
    }

    /**
     * The invoke entry point of lookup function.
     *
     * @param rowKey the lookup key. Currently only support single rowkey.
     */
    public void eval(Object rowKey) throws IOException {
        // fetch result
        Result result = table.get(readHelper.createGet(rowKey));
        if (!result.isEmpty()) {
            // parse and collect
            collect(readHelper.parseToRow(result, rowKey));
        }
    }

    @Override
    public TypeInformation<Row> getResultType() {
        return hbaseTableSchema.convertsToTableSchema().toRowType();
    }

    private org.apache.hadoop.conf.Configuration prepareRuntimeConfiguration() {
        // create default configuration from current runtime env (`hbase-site.xml` in classpath)
        // first,
        // and overwrite configuration using serialized configuration from client-side env
        // (`hbase-site.xml` in classpath).
        // user params from client-side have the highest priority
        org.apache.hadoop.conf.Configuration runtimeConfig =
                HBaseConfigurationUtil.deserializeConfiguration(
                        serializedConfig, HBaseConfigurationUtil.getHBaseConfiguration());

        // do validation: check key option(s) in final runtime configuration
        if (StringUtils.isNullOrWhitespaceOnly(runtimeConfig.get(HConstants.ZOOKEEPER_QUORUM))) {
            LOG.error(
                    "can not connect to HBase without {} configuration",
                    HConstants.ZOOKEEPER_QUORUM);
            throw new IllegalArgumentException(
                    "check HBase configuration failed, lost: '"
                            + HConstants.ZOOKEEPER_QUORUM
                            + "'!");
        }

        return runtimeConfig;
    }

    @Override
    public void open(FunctionContext context) {
        LOG.info("start open ...");
        org.apache.hadoop.conf.Configuration config = prepareRuntimeConfiguration();
        try {
            hConnection = ConnectionFactory.createConnection(config);
            table = (HTable) hConnection.getTable(TableName.valueOf(hTableName));
        } catch (TableNotFoundException tnfe) {
            LOG.error("Table '{}' not found ", hTableName, tnfe);
            throw new RuntimeException("HBase table '" + hTableName + "' not found.", tnfe);
        } catch (IOException ioe) {
            LOG.error("Exception while creating connection to HBase.", ioe);
            throw new RuntimeException("Cannot create connection to HBase.", ioe);
        }
        this.readHelper = new HBaseReadWriteHelper(hbaseTableSchema);
        LOG.info("end open.");
    }

    @Override
    public void close() {
        LOG.info("start close ...");
        if (null != table) {
            try {
                table.close();
                table = null;
            } catch (IOException e) {
                // ignore exception when close.
                LOG.warn("exception when close table", e);
            }
        }
        if (null != hConnection) {
            try {
                hConnection.close();
                hConnection = null;
            } catch (IOException e) {
                // ignore exception when close.
                LOG.warn("exception when close connection", e);
            }
        }
        LOG.info("end close.");
    }

    @VisibleForTesting
    public String getHTableName() {
        return hTableName;
    }
}

2. 简单版:

之前由于在不熟悉代码的情况下测试，使用的源码的返回值Row会报错，所以我在基于HBaseLookupFunction的逻辑上简化了操作，具体代码如下：

MyHbaseLookupFuncation

public static class MyHbaseLookupFuncation extends TableFunction<String> {
        private static final Logger LOG = LoggerFactory.getLogger(MyHbaseLookupFuncation.class);

        private final String hTableName;
        private Configuration configuration;

        private transient Connection hConnection;
        private transient HTable table;

        public MyHbaseLookupFuncation(String hTableName) {
            this.hTableName = hTableName;
        }
		
    	// 修改了返回值，从Row 改成 String
        public void eval(Object rowKey) throws IOException {
            Result result = table.get(new Get(Bytes.toBytes(String.valueOf(rowKey))));
            if (!result.isEmpty()) {
                collect(parseColumn(result));
            }
        }


        public String parseColumn(Result result) {
            StringBuffer sb = new StringBuffer();

            //行键
            String rowkey = Bytes.toString(result.getRow());
            sb.append(rowkey).append(",");
            KeyValue[] kvs = result.raw();

            for (KeyValue kv : kvs) {
                //列族名
                String family = Bytes.toString(kv.getFamily());
                //列名
                String qualifier = Bytes.toString(kv.getQualifier());

                String value = Bytes.toString(result.getValue(Bytes.toBytes(family), Bytes.toBytes(qualifier)));
                sb.append(qualifier).append(":").append(value).append(" , ");
            }
            return sb.toString();
        }


        @Override
        public void open(FunctionContext context) {
            LOG.info("start open ...");
            try {
                Configuration configuration = new Configuration();
                configuration.set("hbase.zookeeper.quorum", "192.168.0.115");
                hConnection = ConnectionFactory.createConnection(configuration);
                table = (HTable) hConnection.getTable(TableName.valueOf(hTableName));
            } catch (TableNotFoundException tnfe) {
                LOG.error("Table '{}' not found ", hTableName, tnfe);
                throw new RuntimeException("HBase table '" + hTableName + "' not found.", tnfe);
            } catch (IOException ioe) {
                LOG.error("Exception while creating connection to HBase.", ioe);
                throw new RuntimeException("Cannot create connection to HBase.", ioe);
            }
            LOG.info("end open.");
        }

        @Override
        public void close() {
            LOG.info("start close ...");
            if (null != table) {
                try {
                    table.close();
                    table = null;
                } catch (IOException e) {
                    // ignore exception when close.
                    LOG.warn("exception when close table", e);
                }
            }
            if (null != hConnection) {
                try {
                    hConnection.close();
                    hConnection = null;
                } catch (IOException e) {
                    // ignore exception when close.
                    LOG.warn("exception when close connection", e);
                }
            }
            LOG.info("end close.");
        }
    }
最终代码：
1. Main
package cn.xhjava.flink.table.connector;

import cn.xhjava.domain.Student2;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.connector.hbase.source.HBaseLookupFunction;
import org.apache.flink.connector.hbase.util.HBaseTableSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.types.Row;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.io.Serializable;

/**
 * @author Xiahu
 * @create 2021/4/6
 */
public class HbaseLookupFuncationDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);


        DataStreamSource<String> sourceStream = env.readTextFile("student");
        DataStream<Student2> map = sourceStream.map(new MapFunction<String, Student2>() {
            @Override
            public Student2 map(String s) throws Exception {
                String[] fields = s.split(",");
                return new Student2(Integer.valueOf(fields[0]), fields[1], fields[2]);
            }
        });

        Table student = tableEnv.fromDataStream(map, "id,name,sex");
        tableEnv.createTemporaryView("student", student);

		// 实例化之前修改的Function
        MyHbaseLookupFuncation baseLookupFunction = new MyHbaseLookupFuncation("test:hbase_user_behavior");

        //注册函数
        tableEnv.registerFunction("hbaseLookup", baseLookupFunction);
        System.out.println("函数注册成功~~~");

        Table table = tableEnv.sqlQuery("select id,name,sex,info from student,LATERAL TABLE(hbaseLookup(id)) as T(info)");


        tableEnv.toAppendStream(table, Row.class).print();

        env.execute();

    }
}
2. student 文件
1,mike,男
2,jimmy,女
3,jerry,女
3. Student2.java
public class Student2 {
    private Integer id;
    private String name;
    private String sex;



    public Student2() {
    }



    public Student2(Integer id, String name, String sex) {
        this.id = id;
        this.name = name;
        this.sex = sex;
    }

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSex() {
        return sex;
    }

    public void setSex(String sex) {
        this.sex = sex;
    }
}
hbase 表内数据

执行结果：
14:52:07,647 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x346643ca connecting to ZooKeeper ensemble=192.168.0.115:2181
14:52:07,647 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x3c6b1620 connecting to ZooKeeper ensemble=192.168.0.115:2181
14:52:07,647 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x8dc76f connecting to ZooKeeper ensemble=192.168.0.115:2181
14:52:07,647 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x23388e54 connecting to ZooKeeper ensemble=192.168.0.115:2181
14:52:10,274 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e824dc6
2> 2,jimmy,女,2,create_time,test,dt,test,hr,test,mm,test,mt_wm_poi_id,test,platform,test,shop_name,test,source,test,user_id,test,
3> 3,jerry,女,3,create_time,test,dt,test,hr,test,mm,test,mt_wm_poi_id,test,platform,test,shop_name,test,source,test,user_id,test,
1> 1,mike,男,1,create_time,test,dt,test,hr,test,mm,test,mt_wm_poi_id,test,platform,test,shop_name,test,source,test,user_id,test,
14:52:10,998 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e824dc3
14:52:10,998 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e824dc4
14:52:10,998 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e824dc5

student 文件内有三条数据，lookup hbase 内的四条数据，最后打印3条数据，结果没问题

注意：通过日志显示，程序与hbase服务端 连接共计4次，断开连接4次；因为Flink程序默认的并行度为4，所以与habse服务端建立4次连接，如果需要修改，env.setParallelism(1);即可

3. 源码版

在对HBaseLookupFunction进行一定的了解后，使用TableFunction 进行look up 查询

源码

package org.apache.flink.connector.hbase.source;

import org.apache.flink.annotation.Internal;
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.connector.hbase.util.HBaseConfigurationUtil;
import org.apache.flink.connector.hbase.util.HBaseReadWriteHelper;
import org.apache.flink.connector.hbase.util.HBaseTableSchema;
import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.types.Row;
import org.apache.flink.util.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

@Internal
public class HBaseLookupFunction extends TableFunction<Row> {
    private static final Logger LOG = LoggerFactory.getLogger(HBaseLookupFunction.class);
    private static final long serialVersionUID = 1L;

    private final String hTableName;
    private final byte[] serializedConfig;
    private final HBaseTableSchema hbaseTableSchema;

    private transient HBaseReadWriteHelper readHelper;
    private transient Connection hConnection;
    private transient HTable table;

    public HBaseLookupFunction(
            Configuration configuration, String hTableName, HBaseTableSchema hbaseTableSchema) {
        this.serializedConfig = HBaseConfigurationUtil.serializeConfiguration(configuration);
        this.hTableName = hTableName;
        this.hbaseTableSchema = hbaseTableSchema;
    }

	
    public void eval(Object rowKey) throws IOException {
        Result result = table.get(readHelper.createGet(rowKey));
        if (!result.isEmpty()) {
            collect(readHelper.parseToRow(result, rowKey));
        }
    }

    //改方法有关返回值参数列表，比如:
    // 返回两个参数: return new RowTypeInfo(Types.STRING, Types.INT);
    // 返回三个参数: return new RowTypeInfo(Types.STRING, Types.INT);
    // 由于我们是去lookUp hbase的数据，我们只能去操作：HBaseTableSchema,来修改返回参数
    @Override
    public TypeInformation<Row> getResultType() {
        return hbaseTableSchema.convertsToTableSchema().toRowType();
    }

    private org.apache.hadoop.conf.Configuration prepareRuntimeConfiguration() {
        org.apache.hadoop.conf.Configuration runtimeConfig =
                HBaseConfigurationUtil.deserializeConfiguration(
                        serializedConfig, HBaseConfigurationUtil.getHBaseConfiguration());

        // do validation: check key option(s) in final runtime configuration
        if (StringUtils.isNullOrWhitespaceOnly(runtimeConfig.get(HConstants.ZOOKEEPER_QUORUM))) {
            LOG.error(
                    "can not connect to HBase without {} configuration",
                    HConstants.ZOOKEEPER_QUORUM);
            throw new IllegalArgumentException(
                    "check HBase configuration failed, lost: '"
                            + HConstants.ZOOKEEPER_QUORUM
                            + "'!");
        }

        return runtimeConfig;
    }

    @Override
    public void open(FunctionContext context) {
        LOG.info("start open ...");
        org.apache.hadoop.conf.Configuration config = prepareRuntimeConfiguration();
        try {
            hConnection = ConnectionFactory.createConnection(config);
            table = (HTable) hConnection.getTable(TableName.valueOf(hTableName));
        } catch (TableNotFoundException tnfe) {
            LOG.error("Table '{}' not found ", hTableName, tnfe);
            throw new RuntimeException("HBase table '" + hTableName + "' not found.", tnfe);
        } catch (IOException ioe) {
            LOG.error("Exception while creating connection to HBase.", ioe);
            throw new RuntimeException("Cannot create connection to HBase.", ioe);
        }
        this.readHelper = new HBaseReadWriteHelper(hbaseTableSchema);
        LOG.info("end open.");
    }

    @Override
    public void close() {
        LOG.info("start close ...");
        if (null != table) {
            try {
                table.close();
                table = null;
            } catch (IOException e) {
                // ignore exception when close.
                LOG.warn("exception when close table", e);
            }
        }
        if (null != hConnection) {
            try {
                hConnection.close();
                hConnection = null;
            } catch (IOException e) {
                // ignore exception when close.
                LOG.warn("exception when close connection", e);
            }
        }
        LOG.info("end close.");
    }

    @VisibleForTesting
    public String getHTableName() {
        return hTableName;
    }
}
1. 实时数据lookup 单张Hbase表
1. Main
public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        env.setParallelism(1);
        DataStreamSource<String> sourceStream = env.readTextFile("D:\\git\\study\\Flink-Learning\\flink-table\\src\\main\\resources\\student");
        DataStream<Student3> map = sourceStream.map(new MapFunction<String, Student3>() {
            @Override
            public Student3 map(String s) throws Exception {
                String[] fields = s.split(",");
                return new Student3(String.valueOf(fields[0]), fields[1], fields[2]);
            }
        });

        Table student = tableEnv.fromDataStream(map, "id,name,sex");
        tableEnv.createTemporaryView("student", student);

        Configuration configuration = new Configuration();
        configuration.set("hbase.zookeeper.quorum", "192.168.0.115");
    	// 注册封装HBaseTableSchema，关于后续参数返回
        HBaseTableSchema baseTableSchema = new HBaseTableSchema();
   		// 设置主键
        baseTableSchema.setRowKey("rowkey", String.class);
    	// 设置列族
        baseTableSchema.addColumn("info","small",String.class);
        baseTableSchema.addColumn("info","yellow",String.class);
        baseTableSchema.addColumn("info","man",String.class);
        HBaseLookupFunction baseLookupFunction = new HBaseLookupFunction(configuration, "test:hbase_user_behavior", baseTableSchema);

        //注册函数
        tableEnv.registerFunction("hbaseLookup", baseLookupFunction);
        System.out.println("函数注册成功~~~");

        Table table = tableEnv.sqlQuery("select id,name,sex,info.small,info.yellow,info.man from student,LATERAL TABLE(hbaseLookup(id)) as T(rowkey,info)");
    	/*
    	 * 在封装HBaseTableSchema阶段，我设置了RowKey,列族，所以在SQL中可以使用：LATERAL TABLE(hbaseLookup(id)) as T(rowkey,info)
    	 * 在上面代码设置列族时分别设置三个列:small,yellow,man
    	 * 所以在SQL中可以使用：info.small,info.yellow,info.man
    	 */

        tableEnv.toAppendStream(table, Row.class).print();
        env.execute();
    }
2. Student3.java
package cn.xhjava.domain;


/**
 * @author Xiahu
 * @create 2020/10/27
 */

public class Student3 {
    private String id;
    private String name;
    private String sex;

    public Student3() {
    }

    public Student3(String id, String name, String sex) {
        this.id = id;
        this.name = name;
        this.sex = sex;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSex() {
        return sex;
    }

    public void setSex(String sex) {
        this.sex = sex;
    }
}
3. student
1,tom1,男
4. hbase数据

test:hbase_user_behavior

5. 输出结果

函数注册成功~~~
14:32:14,046 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x82fa530 connecting to ZooKeeper ensemble=192.168.0.115:2181
1,tom1,男,jerry,basketball,1
14:32:18,952 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x1781a191e827f9e

2. 实时数据lookup 多张Hbase表
1. Main
package cn.xhjava.flink.table.demo;

import cn.xhjava.domain.Student3;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.connector.hbase.source.HBaseLookupFunction;
import org.apache.flink.connector.hbase.util.HBaseTableSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.apache.flink.types.Row;
import org.apache.hadoop.conf.Configuration;

import java.text.SimpleDateFormat;

/**
 * @author Xiahu
 * @create 2021/4/6
 * <p>
 * 流数据实时 look up hbase 双表表数据查询,数据类型 Row
 */
public class FlinkTable_02_HbaseLookupFuncation4 {
    private static SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        env.setParallelism(1);
        DataStreamSource<String> sourceStream = env.readTextFile("D:\\git\\study\\Flink-Learning\\flink-table\\src\\main\\resources\\student");
        DataStream<Student3> map = sourceStream.map(new MapFunction<String, Student3>() {
            @Override
            public Student3 map(String s) throws Exception {
                String[] fields = s.split(",");
                return new Student3(String.valueOf(fields[0]), fields[1], fields[2]);
            }
        });

        Table student = tableEnv.fromDataStream(map, "id,name,sex");
        tableEnv.createTemporaryView("student", student);

        Configuration configuration = new Configuration();
        configuration.set("hbase.zookeeper.quorum", "192.168.0.115");
        // 封装hbase_user_behavior HBaseTableSchema
        HBaseTableSchema baseTableSchema = new HBaseTableSchema();
        baseTableSchema.setRowKey("rowkey", String.class);
        baseTableSchema.addColumn("info","small",String.class);
        baseTableSchema.addColumn("info","yellow",String.class);
        baseTableSchema.addColumn("info","man",String.class);
        HBaseLookupFunction baseLookupFunction = new HBaseLookupFunction(configuration, "test:hbase_user_behavior", baseTableSchema);
		// 封装hbase_user_behavior2 HBaseTableSchema
        HBaseTableSchema baseTableSchema2 = new HBaseTableSchema();
        baseTableSchema2.setRowKey("rowkey", String.class);
        baseTableSchema2.addColumn("info","age",String.class);
        baseTableSchema2.addColumn("info","name",String.class);
        baseTableSchema2.addColumn("info","sex",String.class);
        HBaseLookupFunction baseLookupFunction2 = new HBaseLookupFunction(configuration, "test:hbase_user_behavior2", baseTableSchema2);

        //注册函数
        tableEnv.registerFunction("hbaseLookup", baseLookupFunction);
        tableEnv.registerFunction("hbaseLookup2", baseLookupFunction2);
        System.out.println("函数注册成功~~~");

        Table table = tableEnv.sqlQuery("select id,info1.small,info1.yellow,info1.man,info2 from student,LATERAL TABLE(hbaseLookup(id)) as T(rowkey1,info1),LATERAL TABLE(hbaseLookup2(info1.man)) as T2(rowkey2,info2)");
      	// 通过从表：hbase_user_behavior 查询hbase_user_behavior.man的值，  去 hbase_user_behavior2 查询结果，最终将查询结果返回

        tableEnv.toAppendStream(table, Row.class).print();
        env.execute();
    }
}
2. Student3.java
package cn.xhjava.domain;


/**
 * @author Xiahu
 * @create 2020/10/27
 */

public class Student3 {
    private String id;
    private String name;
    private String sex;

    public Student3() {
    }

    public Student3(String id, String name, String sex) {
        this.id = id;
        this.name = name;
        this.sex = sex;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSex() {
        return sex;
    }

    public void setSex(String sex) {
        this.sex = sex;
    }
}
3. student
1,tom1,男
4. hbase数据

test:hbase_user_behavior

test:hbase_user_behavior2

5. 结果
函数注册成功~~~
14:47:30,075 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x6a6c0d1d connecting to ZooKeeper ensemble=192.168.0.115:2181
14:47:32,585 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        - Process identifier=hconnection-0x51c734e8 connecting to ZooKeeper ensemble=192.168.0.115:2181
1,jerry,basketball,1,1,tom1,男
14:47:33,295 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e827fcd
14:47:33,308 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - Closing zookeeper sessionid=0x1781a191e827fcc

4. 性能测试

基于并行度=1测试

1.hbase 数据1000万；实时数据：1000
总耗时(flink程序启动)：
rowkey有序:
开始时间2021-04-06 15:33:33
结束时间2021-04-06 15:33:42

rowkey无序:
开始时间2021-04-06 15:38:49
结束时间2021-04-06 15:38:59


2.hbase 数据133631681；实时数据：1000
总耗时(flink程序启动)：
rowkey有序:
开始时间2021-04-06 18:07:55
结束时间2021-04-06 18:08:02

rowkey无序:
开始时间2021-04-06 18:05:56
结束时间2021-04-06 18:06:04

5. 实时

  1. 目前hbase内没有rowkey=200000数据；
  2. 实时数据id = 200000 暂时无法在hbase内查找到；
  3. hbase内插入rowkey = 200000 的数据；
  4. 实时数据id = 200000 再次进来，成功关联；

结论：实时数据可以实时的从hbase内look up ，真实实时性暂时还未测试

地球人是我哈
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
flink sql 知其所以然（十五）：改了改源码，实现了个 batch lookup join（附源码）
qq_34608620的博客
12-05 2739
看了那么多的技术文，你能明白作者想让你在读完文章后学到什么吗？大数据羊说的文章会让你明白
【SQLite数据库存储】CRUD操作
Errol's Blog
03-05 1114
CRUD C 代表添加 （Create） R代表查询（Retrieve） U代表更新（Update） D代表删除（Delete） 每一种操 作又各自对应了一种 SQL命令，如果你比较熟悉 SQL语言的话，一定会知道添加数据时使 用 insert，查询数据时使用 select，更新数据时使用 update，删除数据时使用 delete 但是开 发者的水平总会是参差不齐的，未必每一个人都...
Flink LookupJoin攒批查询
shirukai
05-16 630
使用Lookup Join进行维表关联时，流表数据需要实时与维表数据进行关联。使用Cache会导致数据关联不准确，不使用Cache会造成数据库服务压力。攒批查询是指攒够一定批数量的数据，相同的查询Key只查询一次，从而减少查询次数。对短时间Key重复率比较高的场景有不错的性能提升。
flink sql连接hbase
张不帅
11-02 1321
文章目录 创建hbase表可以设置参数 public static final ConfigOption<String> TABLE_NAME = ConfigOptions.key("table-name").stringType().noDefaultValue().withDescription("The name of HBase table to connect."); public static final ConfigOption<String> ZOO
在Spark Streaming中使用Hbase进行lookup实时读取
J.Tong
08-23 873
翻译自：HBase lookup in Spark Streaming 简介 在Spark Streaming应用中，HBase可用于处理流式数据过程中每个Batch进行lookup数据的高速缓存，对这个缓存的查询是在每条Spark Streaming新到的消息的基础上去进行的。由于Spark Streaming是微批次（micro-batching）架构，有两种方法去完成这个需求： 在每个微...
Flink异步IO查询Hbase数据，采用旁路缓存redis
qq_50408152的博客
08-01 1201
Flink异步IO查询Hbase数据，采用旁路缓存redis。
flink1.18.1安装包
最新发布
05-29
Flink 1.18.1 是一个高度可扩展的开源流处理框架，适用于实时数据流分析和批处理。在Linux环境下安装Flink 1.18.1，需要遵循以下步骤，确保系统具备必要的环境并正确配置。 首先，确保你的Linux系统上已经安装了...
flink cdc postgresql
04-07
标题中的“flink cdc postgresql”指的是一项技术整合，即使用Apache Flink的数据变更数据捕获（CDC，Change Data Capture）功能来实时同步PostgreSQL数据库的数据变化。这项技术在大数据实时处理和流计算场景中非常...
flink-sql集成rabbitmq
06-29
标题中的“flink-sql集成rabbitmq”指的是将Apache Flink的数据流处理能力与RabbitMQ消息队列系统相结合，实现数据的实时处理和传输。Flink是一个开源的流处理框架，提供低延迟、高吞吐量的数据处理，而RabbitMQ是一...
Flink剖析
01-27
Flink做一个简要的剖析认识以及Flink版本中的接口支持，希望对您的学习有所帮助。在如今数据爆炸的时代，企业的数据量与日俱增，大数据产品层出不穷。今天给大家分享一款产品—— ApacheFlink，目前，已是Apache顶级...
Flink Atlas 整合
09-09
Flink Atlas 整合 Flink Atlas 整合是指将 Apache Atlas 数据治理框架与 Apache Flink 集成，以追踪 Flink 作业的输入输出数据、所有权和数据血缘关系。Apache Atlas 是一个数据治理框架，用于存储元数据，追踪数据...
详解flink中Look up维表的使用
公众号[大数据技术与应用实战]，分享大数据实战案例
09-03 3768
在流式计算中，维表是一个很常见的概念，一般用于sql的join中，对流式数据进行数据补全，比如我们的source stream是来自日志的订单数据，但是日志中我们只是记录了订单商品的id，并没有其他的信息，但是我们把数据存入数仓进行数据分析的时候，却需要商品名称、价格等等其他的信息，这种问题我们可以在进行流处理的时候通过查询维表的方式对数据进行数据补全。 维表一般存储在外部存储中，比如mysql、hbase、redis等等，今天我们以mysql为例，讲讲flink中维表的使用。
flink 1.14 异步 join 基础分析
Hello World
12-16 1964
自定义AsyncTableFunction 顺序原理
Flink SQL 优化实战 - 维表 JOIN 优化
cloudbigdata的博客
05-27 6450
从源码层面分析维表 Join 原理及调优方案
flink-connector-hbase 任务提交yarn出现类冲突bugfix
TreeCode的博客
09-08 1524
第一种报错信息： java.lang.RuntimeException: Cannot create connection to HBase. at org.apache.flink.connector.hbase.sink.HBaseSinkFunction.open(HBaseSinkFunction.java:99) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java
flink启动报错
weixin_45230582的博客
12-16 2003
flink 启动报错 flink on yarn 启动报错： 连接不到xxx：8032 我的原因是flink.yaml中dfs的地址错误。 还有就是启动时进行内存检测，内存不足报错： 2018-05-19 15:30:10,456 WARN akka.remote.ReliableDeliverySupervisor - Association wi...
从Flink LookupJoin 入门 Flink Rule
zstraw的博客
03-03 6185
网上有一系列关于 Flink SQL、Calcite 的高大上的文章，看完却无法上手开发。 本文主要从 Flink 的 JOIN 基础开始，带新手走通 LookupJoin 的流程，教你自行 Debug，通过熟悉中间的 Rule 后，读者可以上手做一些自定义的修改或开发自己的 Rule。
Flink-安装部署与快速入门.docx
12-16
"Flink安装部署与快速入门" Apache Flink是一个开源流处理框架，它旨在提供低延迟、高性能和高吞吐量的数据处理能力。Flink的诞生背景是为了应对大数据实时处理的需求，尤其是在互联网和物联网(IoT)领域，实时数据...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值