Flink CDC2.2动态加载表，读取mysql数据，写入到mq

最新推荐文章于 2024-12-18 18:36:14 发布

LuciferWWP

最新推荐文章于 2024-12-18 18:36:14 发布

阅读量5.3k

点赞数

分类专栏： CDC数据同步文章标签： flink mysql java-rabbitmq

本文链接：https://blog.csdn.net/qq_34373761/article/details/125428278

版权

1、使用 Flink Stream API 开发一个 Flink CDC Demo

2、以Mysql为例，采集Mysql binlog数据并发送到mq

3、生成 checkpoint 数据，重启程序从执行的状态恢复数据

4、演示2.2版本动态加载表的新特性

新建一个maven项目，引入pom

<!-- flink-cdc-->
<dependency>
    <groupId>com.ververica</groupId>
    <artifactId>flink-connector-mysql-cdc</artifactId>
    <version>2.2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_2.12</artifactId>
    <version>1.13.6</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.12</artifactId>
    <version>1.13.6</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>1.13.6</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-table-planner-blink_2.12</artifactId>
    <version>1.13.6</version>
    <type>test-jar</type>
</dependency>
<!-- flink-cdc-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
    <version>2.0.9.RELEASE</version>
</dependency>
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.75</version>
</dependency>

新建mq工厂类

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import org.apache.commons.lang3.StringUtils;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;

import java.io.IOException;
import java.util.concurrent.TimeoutException;

/**
 * @description:
 * @author: wwp
 * @date: 2022-06-23 14:48
 */
public class HandlerSink extends RichSinkFunction<String>