Flink 两表关联流批处理消费kafka 数据写入hbase

最新推荐文章于 2024-03-13 22:45:02 发布

小小聪

最新推荐文章于 2024-03-13 22:45:02 发布

阅读量3.7k

点赞数 1

分类专栏： flink join

本文链接：https://blog.csdn.net/servletwjx/article/details/90260425

版权

该博客展示了如何使用Flink进行流批处理，消费Kafka中的两个主题数据，通过SQL Join操作进行数据关联，并将结果实时写入Hbase。详细步骤包括设置Flink环境、配置Kafka源、执行SQL查询以及Hbase数据写入。

摘要由CSDN通过智能技术生成

Flink流批处理消费kafka 数据写入hbase

通过flume将数据写入kafka topic

Kafka topic1 数据：

name, age, sexy, proctime.proctime

java,18,男,20190516

rose,28,女,20190516

tom,38,男,20190516

jack,18,男,20190516

luoli,19,女,20190516

Kafka topic2 数据：

name, age, sexy, city, proctime.proctime

java,18,男,hangzh,20190516

rose,28,女,xian,20190516

tom,38,男,shanghai,20190516

jack,18,男,beijin,20190516

luoli,19,女,baoji,20190516

具体实现代码：

package com.sitesh;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.runtime.state.filesystem.FsStateBackend;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.table.api.java.StreamTableEnvironment;
import org.apache.flink.types.Row;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;

import java.io.IOException;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

public class SqlJoinWithKafka {
public static void main(String[] args) throws Exception{

System.out.println("use command as: