kafka mysql cdc_【Flink小试】Flink CDC DataStream API监听MySQL动态发往Kafka Topic

[toc]

一、背景

业务背景: MySQL增量数据实时更新同步到Kafka中供下游使用

查看了一下Flink CDC的官方文档，其中Features的描述中提到了SQL和DataStream API不同的支持程度。

Features

1. Supports reading database snapshot and continues to read binlogs with exactly-once processing even failures happen.

2. CDC connectors for DataStream API, users can consume changes on multiple databases and tables in a single job without Debezium and Kafka deployed.

3. CDC connectors for Table/SQL API, users can use SQL DDL to create a CDC source to monitor changes on a single table.

虽然SQL API使用很丝滑，也很简单。但是由于业务表较多，若是使用一个表的监听就开启一个Flink Job，会对资源消耗和运维操作带来很大的麻烦，所以笔者决定使用DataStream API实现单任务监听库级的MySQL CDC并根据表名将数据发往不同的Kafka Topic中。

二、代码实现

1. 关键maven依赖

com.alibaba.ververica

flink-connector-mysql-cdc

1.1.1

org.apache.flink

flink-connector-kafka_2.11

org.apache.kafka

kafka-clients

org.apache.kafka

kafka-clients

2.4.0

2. 自定义CDC数据反序列化器

Flink CDC定义了com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema接口用以对CDC数据进行反序列化。默认实现类com.alibaba.ververica.cdc.debezium.table.RowDataDebeziumDeserializeSchema和com.alibaba.ververica.cdc.debezium.StringDebeziumDeserializationSchema,由于我们需要自定义Schema，所以我们不采用这两周默认的实现类，自己实现该接口定义我们需要的Schema.

定义JsonDebeziumDeserializeSchema实现DebeziumDeserializationSchema接口方法

class JsonDebeziumDeserializeSchema extends DebeziumDeserializationSchema[String] {

private final val log: Logger = LoggerFactory.getLogger(classOf[JsonDebeziumDeserializeSchema])

override def deserialize(sourceRecord: SourceRecord, collector: Collector[String]): Unit = {

val op = Envelope.operationFor(sourceRecord)

val source = sourceRecord.topic()

val value = sourceRecord.value().asInstanceOf[Struct]

val valueSchema: Schema = sourceRecord.valueSchema()

if (op != Operation.CREATE && op != Operation.READ) {

if (op == Operation.DELETE) {

val data = extractBeforeData(value, valueSchema)

val record = new JSONObject()

.fluentPut("source", source)

.fluentPut("data", data)

.fluentPut("op", RowKind.DELETE.shortString())

.toJSONString

collector.collect(record)

} else {

val beforeData = extractBeforeData(value, valueSchema)

val beforeRecord = new JSONObject()

.fluentPut("source", source)

.fluentPut("data", beforeData)

.fluentPut("op", RowKind.UPDATE_BEFORE.shortString())

.toJSONString

collector.collect(beforeRecord)

val afterData = extractAfterData(value, valueSchema)

val afterRecord = new JSONObject()

.fluentPut("source", source)

.fluentPut("data", afterData)

.fluentPut("op", RowKind.UPDATE_AFTER.shortString())

.toJSONString

collector.collect(afterRecord)

}

} else {

val data = extractAfterData(value, valueSchema)

val record = new JSONObject()

.fluentPut("source", source)

.fluentPut("data", data)

.fluentPut("op", RowKind.INSERT.shortString())

.toJSONString

collector.collect(record)

}

}

override def getProducedType: TypeInformation[String] = BasicTypeInfo.STRING_TYPE_INFO

...

}

定义MySqlSource监听MySQL库数据变化：

val properties = new Properties()

properties.setProperty("snapshotMode", snapshotMode)

val mysqlCdcSource = MySQLSource.builder[String]()

.hostname(hostname)

.port(port)

.databaseList(database)

.tableList(tableName)

.username(username)

.password(password)

.deserializer(new JsonDebeziumDeserializeSchema)

.debeziumProperties(properties)

.serverId(serverId)

.build()

3. 数据动态发往Kafka不同的Topic

由上面自定义的Schema我们可以知道，source字段的构成为mysql_binlog_source.库名.表名。此时我们可以自定义KafkaSerializationSchema来实现将不同的数据发往不同的topic，即OverridingTopicSchema:

abstract class OverridingTopicSchema extends KafkaSerializationSchema[String] {

val topicPrefix: String

val topicSuffix: String

val topicKey: String

override def serialize(element: String, timestamp: lang.Long): ProducerRecord[Array[Byte], Array[Byte]] = {

val topic = if (element != null && element.contains(topicKey)) {

val topicStr = JSON.parseObject(element).getString(topicKey).replaceAll("\\.", "_")

topicPrefix.concat(topicStr).concat(topicSuffix)

} else null

new ProducerRecord[Array[Byte], Array[Byte]](topic, element.getBytes(StandardCharsets.UTF_8))

}

}

同时定义创建将数据动态发往不同topic的kafka生产者的方法

/**

* 创建将数据动态发往不同topic的kafka生产者

*

* @param boostrapServers kafka集群地址

* @param kafkaSerializationSchema kafka序列器

* @return

*/

def createDynamicFlinkProducer(boostrapServers: String, kafkaSerializationSchema: KafkaSerializationSchema[String]): FlinkKafkaProducer[String] = {

if (StringUtils.isEmpty(boostrapServers))

throw new IllegalArgumentException("boostrapServers is necessary")

val properties = initDefaultKafkaProducerConfig(boostrapServers)

properties.put(ACKS_CONFIG, "all")

new FlinkKafkaProducer[String](DEFAULT_TOPIC, kafkaSerializationSchema,

properties, FlinkKafkaProducer.Semantic.EXACTLY_ONCE)

}

4. 主类完整实现

object Cdc2KafkaByStream {

def main(args: Array[String]): Unit = {

val parameterTool = ParameterTool.fromArgs(args)

//cdc config

val hostname = parameterTool.get("hostname")

val port = parameterTool.getInt("port", 3306)

val username = parameterTool.get("username")

val password = parameterTool.get("password")

val database = parameterTool.get("database")

val tableName = parameterTool.get("tableName")

val serverId = parameterTool.getInt("serverId")

val snapshotMode = parameterTool.get("snapshotMode", "initial")

//kafka config

val kafkaBrokers = parameterTool.get("kafkaBrokers")

val kafkaTopicPrefix = parameterTool.get("kafkaTopicPrefix", "topic_")

val kafkaTopicSuffix = parameterTool.get("kafkaTopicSuffix", "")

val kafkaTopicKey = parameterTool.get("kafkaTopicKey", "source")

val env = StreamExecutionEnvironment.getExecutionEnvironment

ExecutionEnvUtils.configStreamExecutionEnv(env, parameterTool)

ExecutionEnvUtils.parameterPrint(parameterTool)

val properties = new Properties()

properties.setProperty("snapshotMode", snapshotMode)

val mysqlCdcSource = MySQLSource.builder[String]()

.hostname(hostname)

.port(port)

.databaseList(database)

.tableList(tableName)

.username(username)

.password(password)

.deserializer(new JsonDebeziumDeserializeSchema)

.debeziumProperties(properties)

.serverId(serverId)

.build()

val kafkaSink = KafkaUtils.createDynamicFlinkProducer(kafkaBrokers, new OverridingTopicSchema() {

override val topicPrefix: String = kafkaTopicPrefix

override val topicSuffix: String = kafkaTopicSuffix

override val topicKey: String = kafkaTopicKey

})

env.addSource(mysqlCdcSource).addSink(kafkaSink).setParallelism(1)

env.execute()

}

}

启动任务后可以看到kakfa中根据表名创建了不同的topic，并保存了不同表里的数据。

至此，实现了使用DataStream API单任务监听库级的MySQL CDC并根据表名将数据发往不同的Kafka Topic的功能。

三、小结

本文主要介绍了通过Flink CDC DataStream API实现监听MySQL库数据发往kafka不同Topic的功能，其中运用到自定义DebeziumDeserializationSchema实现CDC Schema自定义反序列化解析以及自定义KafkaSerializationSchema实现根据数据内容将消息发送到不同的topic等功能。