Flink1.13中tableAPI对接数据源的source&sink

最新推荐文章于 2024-07-01 02:25:02 发布

Holdenxiao

最新推荐文章于 2024-07-01 02:25:02 发布

阅读量1k

点赞数 4

分类专栏： flink 文章标签： flink kafka big data

本文链接：https://blog.csdn.net/weixin_42430074/article/details/121753721

版权

flink 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

在学习的过程中遇到了file和kafka数据源读写（sink和source）的问题，我使用的flink是1.13.3的版本，发现connect方法已经过时了，于是就去官网查看了一下推荐的方式，下面我会用代码来实现一下：

参考官方文档：CSVhttps://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/formats/csv/Kafkahttps://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/kafka/

本地数据源：

id,timeStamp,temperature
sensor_1,1,1
sensor_1,2,2
sensor_1,3,32
sensor_1,4,4
sensor_1,5,5
sensor_1,22,6
sensor_1,23,99
sensor_2,25,13
sensor_2,32,40.2
sensor_3,40,42

过时的文件source

添加依赖

<dependency>
     <groupId>org.apache.flink</groupId>
     <artifactId>flink-csv</artifactId>
     <version>1.13.3</version>
</dependency>

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);


        //step-1 表的元数据
        Schema schema = new Schema()
                .field("id", DataTypes.STRING())
                .field("timeStamp", DataTypes.BIGINT())
                .field("temperature", DataTypes.FLOAT());

        //step-2 连接文件，并创建一个临时表，其实就是一个动态表
        tEnv.connect(new FileSystem()
                        .path("..\\resources\\sensor.txt"))
                .withSchema(schema)//定义表结构
                //fieldDelimiter是一行数据根据逗号划分字段
                //lineDelimiter每行数据根据换行符划分一条数据
                .withFormat(new Csv().fieldDelimiter(',').lineDelimiter("\n"))
                .createTemporaryTable("sensor");//创建临时表


        //step-3 做成表对象,然后对动态表进行查询
        Table resTable = tEnv.from("sensor")
                .groupBy($("id"))
                .select($("id"), $("id").count().as("cnt"));
        tEnv.toRetractStream(resTable, Row.class).print("cnt");

        env.execute();

过时的文件Sink

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);

        //step-1 建造数据源
        DataStreamSource<SensorReading> waterSensorStream =
                env.fromElements(new SensorReading("sensor_1", 1000L, 10D),
                        new SensorReading("sensor_1", 2000L, 20D),
                        new SensorReading("sensor_2", 3000L, 30D),
                        new SensorReading("sensor_1", 4000L, 40D),
                        new SensorReading("sensor_1", 5000L, 50D),
                        new SensorReading("sensor_2", 6000L, 60D));

        //step-2 处理数据
        Table resTable = tEnv.fromDataStream(waterSensorStream)
                .where($("id").isEqual("sensor_1"))
                .select($("id"), $("timeStamp"), $("temperature"));


        //step-3 配置元数据
        Schema schema = new Schema()
                .field("id", DataTypes.STRING())
                .field("timeStamp", DataTypes.BIGINT())
                .field("temperature", DataTypes.DOUBLE());

        tEnv.connect(new FileSystem().path("output/sink.txt"))
                .withFormat(new Csv().fieldDelimiter(','))
                .withSchema(schema)
                .createTemporaryTable("sensor_sink");

        resTable.executeInsert("sensor_sink");
       /*
        * explain
        * env.execute()方法会去分析代码，生成一些 graph，但是我们代码中没有调用算子，所以会报错，可以直接不用
        * */

过时的kafka-Source

添加依赖

<dependency>
       <groupId>org.apache.flink</groupId>
       <artifactId>flink-connector-kafka_2.12</artifactId>
       <version>1.13.3</version>
</dependency>

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.setParallelism(1);
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);


        //step-1 表的元数据
        Schema schema = new Schema()
                .field("id", DataTypes.STRING())
                .field("timeStamp", DataTypes.BIGINT())
                .field("temperature", DataTypes.FLOAT());

        //接收来自sensor主题的消息
        tEnv.connect(new Kafka()
                        .version("universal") //kafka通用版本
                        .topic("sensor")
                        .startFromLatest()
                        .property("group.id", "bigdata")
                        .property("bootstrap.servers", "hadoop102:9092,hadoop103:9092,hadoop104:9092"))
                .withFormat(new Json())
                .withSchema(schema)
                .createTemporaryTable("sensor");

        Table table = tEnv.from("sensor")
                .groupBy($("id"))
                .select($("id"), $("id").count().as("cnt"));

        tEnv.toRetractStream(table, Row.class).print();
        env.execute();

过时的kafka-Sink

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);

        //step-1 建造数据源
        DataStreamSource<SensorReading> waterSensorStream =
                env.fromElements(new SensorReading("sensor_1", 1000L, 10D),
                        new SensorReading("sensor_1", 2000L, 20D),
                        new SensorReading("sensor_2", 3000L, 30D),
                        new SensorReading("sensor_1", 4000L, 40D),
                        new SensorReading("sensor_1", 5000L, 50D),
                        new SensorReading("sensor_2", 6000L, 60D));

        //step-2 处理数据
        Table sensorTable = tEnv.fromDataStream(waterSensorStream);
        Table resultTable = sensorTable
                .where($("id").isEqual("sensor_1"))
                .select($("id"), $("timeStamp"), $("temperature"));

        //step-3 配置元数据
        Schema schema = new Schema()
                .field("id", DataTypes.STRING())
                .field("timeStamp", DataTypes.BIGINT())
                .field("temperature", DataTypes.DOUBLE());

        tEnv.connect(new Kafka()
                        .version("universal")
                        .topic("sensor")
                        .sinkPartitionerRoundRobin() // 分区轮询
                        .property("bootstrap.servers", "hadoop102:9092,hadoop103:9092,hadoop104:9092"))
                .withFormat(new Json())
                .withSchema(schema)
                .createTemporaryTable("kafka_sink");

        resultTable.executeInsert("kafka_sink");