flink mysql cdc_Flink之Mysql数据CDC

最新推荐文章于 2024-04-16 08:14:10 发布

weixin_39834984

最新推荐文章于 2024-04-16 08:14:10 发布

阅读量425

点赞数

文章标签： flink mysql cdc

本文链接：https://blog.csdn.net/weixin_39834984/article/details/114995211

版权

本文介绍了如何使用Flink的MySQL CDC connector实时获取MySQL数据库的变更事件，包括插入、更新和删除操作，并展示了源码示例。通过自定义Sink，可以对这些数据进行ETL处理。

摘要由CSDN通过智能技术生成

知识点：

https://github.com/ververica/flink-cdc-connectors //官网地址

1、处理类

import org.apache.flink.configuration.Configuration;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import org.apache.flink.streaming.api.functions.source.SourceFunction;

import com.alibaba.ververica.cdc.debezium.StringDebeziumDeserializationSchema;

import com.alibaba.ververica.cdc.connectors.mysql.MySQLSource;

/**

* @program: Flink1.11

* @description:

* @author: yang

* @create: 2021-01-11 17:41

public class MySqlBinlogSourceExample {

public static void main(String[] args) throws Exception {

SourceFunction sourceFunction = MySQLSource.builder()

.hostname("localhost")

.port(3306)

.databaseList("test")

// monitor all tables under inventory database

.username("root")

.password("root")

.deserializer(new StringDebeziumDeserializationSchema())

// converts SourceRecord to String

.build();

StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());

env.addSource(sourceFunction).print().setParallelism(1);

// use parallelism 1 for sink to keep message ordering

env.execute("test");

}

2、binlog结果

修改:befor and after

SourceRecord{

sourcePartition={server=mysql-binlog-source},

sourceOffset={ts_sec=1610362335, file=mysql-bin.000004, pos=233445691, row=1, server_id=1, event=2}

}

ConnectRecord

{topic='mysql-binlog-source.test.weblog', kafkaPartition=null, key=Struct{id=5}, keySchema=Schema{mysql_binlog_source.test.weblog.Key:STRUCT}, value=Struct{before=Struct{id=5,url=5,method=5,ip=5,args=5,create_time=1610390670000},after=Struct{id=5,url=5555,method=5555,ip=5555,args=5555,create_time=1610390670000},source=Struct{version=1.2.0.Final,connector=mysql,name=mysql-binlog-source,ts_ms=1610362335000,db=test,table=weblog,server_id=1,file=mysql-bin.000004,pos=233445826,row=0,thread=944986},op=u,ts_ms=1610362335615}, valueSchema=Schema{mysql_binlog_source.test.weblog.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)

}

增加:只有after

SourceRecord{sourcePartition={server=mysql-binlog-source}, sourceOffset={file=mysql-bin.000004, pos=233455303}}

ConnectRecord

{topic='mysql-binlog-source.test.weblog', kafkaPartition=null, key=Struct{id=7}, keySchema=Schema{mysql_binlog_source.test.weblog.Key:STRUCT}, value=Struct{after=Struct{id=7,url=7,method=7,ip=7,args=7,create_time=1610391478000},source=Struct{version=1.2.0.Final,connector=mysql,name=mysql-binlog-source,ts_ms=0,snapshot=last,db=test,table=weblog,server_id=0,file=mysql-bin.000004,pos=233455303,row=0},op=c,ts_ms=1610362692061}, valueSchema=Schema{mysql_binlog_source.test.weblog.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

删除:只有before

SourceRecord{sourcePartition={server=mysql-binlog-source}, sourceOffset={ts_sec=1610362743, file=mysql-bin.000004, pos=233456891, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql-binlog-source.test.weblog', kafkaPartition=null, key=Struct{id=1}, keySchema=Schema{mysql_binlog_source.test.weblog.Key:STRUCT}, value=Struct{before=Struct{id=1,url=1,method=1,ip=1,args=1,create_time=1603115590000},source=Struct{version=1.2.0.Final,connector=mysql,name=mysql-binlog-source,ts_ms=1610362743000,db=test,table=weblog,server_id=1,file=mysql-bin.000004,pos=233457026,row=0,thread=944986},op=d,ts_ms=1610362744527}, valueSchema=Schema{mysql_binlog_source.test.weblog.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

3、如果需要将数据进行etl，可以自定义实现sink

标签：binlog,Flink,Struct,CDC,Mysql,source,mysql,test,id

来源： https://www.cnblogs.com/ywjfx/p/14263718.html