8、flinkCDC

最新推荐文章于 2024-10-17 16:21:37 发布

Rainbow酱

最新推荐文章于 2024-10-17 16:21:37 发布

阅读量574

点赞数 29

分类专栏：大数据文章标签： flink

本文链接：https://blog.csdn.net/jusu123/article/details/140040236

版权

大数据专栏收录该内容

19 篇文章 0 订阅

订阅专栏

一、环境准备
1、mysql开启binlog
修改my.cnf，添加或修改
[mysqld]log-bin=mysql-bin_server_-id=1_binlog__format=ROW

检查binlog
重启mysql

之后运行以下语句，查看是否开启成功：

show variables like ‘%log_bin%’;
运行后应该显示 log_bin 是 ON

show variables like ‘binlog_format’;
运行后应该显示 binlog_format 是 ROW

如果运行结果正确，您可以继续下一步操作。如果不正确请先修正。

2、创建maven项目

修改maven配置

修改pom

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.stwc</groupId> <artifactId>flinkcdc-analysis</artifactId> <version>1.0</version> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>8</source> <target>8</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.22.2</version> <configuration> <skipTests>true</skipTests> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <mainClass>*.dsp.redispositive.Application</mainClass> <useUniqueVersions>false</useUniqueVersions> </manifest> </archive> </configuration> </plugin> </plugins> </build> <properties> <flink-version>1.17.0</flink-version> <flink-streaming-version>1.13.0</flink-streaming-version> </properties> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink-streaming-version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.12</artifactId> <version>${flink-streaming-version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.12</artifactId> <version>${flink-streaming-version}</version> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.44</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner-blink_2.12</artifactId> <version>${flink-streaming-version}</version> </dependency> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>2.0.2</version> </dependency>  <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.20</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.75</version> </dependency> </dependencies></project>

创建main类

import com.ververica.cdc.connectors.mysql.MySqlSource;
import com.ververica.cdc.debezium.StringDebeziumDeserializationSchema;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
/**
 * @Description
 *   * @Author 10870004
 *   * @Date 2023/3/27 14:53
 *   */
public class MySqlSourceExample {
    public static void main(String[] args) throws Exception {
        SourceFunction<String> sourceFunction = MySqlSource.<String>builder()
            .hostname("10.9.70.156")
            .port(3306)
            .databaseList("cdc_test") //订阅的库
            .username("root")
            .password("123456") 
            .serverTimeZone("CTT") 
            .deserializer(new StringDebeziumDeserializationSchema()) 
            .build(); 
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.addSource(sourceFunction)
            .print()
            .setParallelism(1);
        env.execute();
    }
}

run启动

修改mysql数据库
产生数据

SourceRecord {
	sourcePartition = {
			server = mysql_binlog_source
		},
		sourceOffset = {
			transaction_id = null,
			ts_sec = 1679906185,
			file = 108 - GRAD - ZQH - A - bin .000214,
			pos = 1491,
			row = 1,
			server_id = 1,
			event = 2
		}
}
ConnectRecord {
	topic = 'mysql_binlog_source.cdc_test.test',
		kafkaPartition = null,
		key = Struct {
			id = 1
		},
		keySchema = Schema {
			mysql_binlog_source.cdc_test.test.Key: STRUCT
		},
		value = Struct {
			before = Struct {
				id = 1, name = test, age = 43, create_time = 1679933796000
			}, after = Struct {
				id = 1, name = test, age = 25, create_time = 1679933796000
			}, source = Struct {
				version = 1.5 .2.Final, connector = mysql, name = mysql_binlog_source, ts_ms = 1679906185000, db = cdc_test, table = test, server_id = 1, file = 108 - GRAD - ZQH - A - bin .000214, pos = 1642, row = 0
			}, op = u, ts_ms = 1679906185278
		}, valueSchema = Schema {
			mysql_binlog_source.cdc_test.test.Envelope: STRUCT
		}, timestamp = null, headers = ConnectHeaders(headers = )
}