Flink CDC 笔记

数据格式

{ "before": null, "after": { "source_table": "11", "source_type": "11", "sink_table": "111", "sink_type": "11", "sink_columns": "1", "sink_pk": null, "sink_extend": null }, "source": { "version": "1.5.4.Final", "connector": "mysql", "name": "mysql_binlog_source", "ts_ms": 1675604674000, "snapshot": "false", "db": "gmall_config", "sequence": null, "table": "table_process", "server_id": 1, "gtid": null, "file": "mysql-bin.000045", "pos": 7772, "row": 0, "thread": null, "query": null }, "op": "c", "ts_ms": 1675653938345, "transaction": null }

 { "before": { "source_table": "11", "source_type": "11", "sink_table": "111", "sink_type": "11", "sink_columns": "1", "sink_pk": null, "sink_extend": null }, "after": { "source_table": "11", "source_type": "22", "sink_table": "111", "sink_type": "11", "sink_columns": "1", "sink_pk": null, "sink_extend": null }, "source": { "version": "1.5.4.Final", "connector": "mysql", "name": "mysql_binlog_source", "ts_ms": 1675604693000, "snapshot": "false", "db": "gmall_config", "sequence": null, "table": "table_process", "server_id": 1, "gtid": null, "file": "mysql-bin.000045", "pos": 8117, "row": 0, "thread": null, "query": null }, "op": "u", "ts_ms": 1675653955883, "transaction": null } 

{ "before": { "source_table": "11", "source_type": "22", "sink_table": "111", "sink_type": "11", "sink_columns": "1", "sink_pk": null, "sink_extend": null }, "after": null, "source": { "version": "1.5.4.Final", "connector": "mysql", "name": "mysql_binlog_source", "ts_ms": 1675604720000, "snapshot": "false", "db": "gmall_config", "sequence": null, "table": "table_process", "server_id": 1, "gtid": null, "file": "mysql-bin.000045", "pos": 8475, "row": 0, "thread": null, "query": null }, "op": "d", "ts_ms": 1675653980108, "transaction": null } 

flink1.13+mysqlcdc2.1导入依赖(DataStream)

<dependency>

        <groupId>com.ververica</groupId>

        <artifactId>flink-connector-mysql-cdc</artifactId>

        <version>2.1.0</version>

</dependency>

<!-- 如果不引入 flink-table 相关依赖，则会报错：

Caused by: java.lang.ClassNotFoundException:

org.apache.flink.connector.base.source.reader.RecordEmitter

引入如下依赖可以解决这个问题（引入某些其它的 flink-table 相关依赖也可）

-->

<dependency>

<groupId>org.apache.flink</groupId>

    <artifactId>flink-table-api-java-bridge_2.12</artifactId>

    <version>1.13.0</version>

</dependency>

导入依赖(FlinkSQL)

<dependency>

    <groupId>org.apache.flink</groupId>

    <artifactId>flink-table-planner-blink_2.12</artifactId>

    <version>1.13.0</version>

</dependency>

flink1.16+mysqlcdc2.3导入依赖(DataStream)

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>1.16.0</version>
</dependency>

<dependency>
    <groupId>com.ververica</groupId>
    <artifactId>flink-connector-mysql-cdc</artifactId>
    <version>2.3.0</version>
</dependency>

<!-- 高于1.14的版本各种包后面就不再有scala的版本了

我在删除各依赖名称后面的scala版本后并升级为1.16.0后，运行还是报了这个错

Caused by: java.lang.ClassNotFoundException:

org.apache.flink.connector.base.source.reader.RecordEmitter

引入如下依赖可以解决这个问题（引入某些其它的 flink-table 相关依赖也可）

-->

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-base</artifactId>
    <version>1.16.0</version>
</dependency>

flink1.16+mysqlcdc2.3 NoSuchMethodError

报错如下 
ERROR (org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager:accept) - Received uncaught exception.
java.lang.NoSuchMethodError: com.mysql.cj.CharsetMapping.getStaticJavaEncodingForMysqlCharset(Ljava/lang/String;)Ljava/lang/String;

用的是这个版本的依赖

<dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.21</version>

 </dependency>

解决方法：换成8.0.30，27也行