1.安装flink,启动flink
文档地址:Apache Flink 1.3-SNAPSHOT 中文文档: Apache Flink 中文文档
代码:GitHub - apache/flink: Apache Flink
jar下载 Index of /dist/flink
1。开启mysql binlog
2. flink代码参考(flink1.16.0+flinkcdc 2.3.0)问题1已经采用方案1解决
3.执行sql
4. 开放端口: 8081(flink 默认端口),直接解压linux-flink-1.16.0.rar启动即可
下载地址:百度网盘微信
注意点: 执行多个job配置:参考下面问题2
5.关于运行job方法
方法1. 打成jar, 上传到flink服务器
方法二:main方法运行
问题1. flink启动出现如下问题:
Caused by: java.lang.NoSuchMethodError: io.debezium.connector.mysql.MySqlConnection$MySqlConnectionConfiguration.<init>(Lio/debezium/config/Configuration;Ljava/util/Properties;)V
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.createMySqlConnection(DebeziumUtils.java:85)
erica.cdc.connectors.mysql.source.reader.MySqlSplitReader.getSnapshotSplitRea
方案1,访问地址:
方案2. 访问fix review · ververica/flink-cdc-connectors@5415bcf · GitHub
把flink-cdc-connectors 源码修改(源码地址:GitHub - ververica/flink-cdc-connectors: CDC Connectors for Apache Flink®)
2.1.flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/MySqlValidator.java
2.2.flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/debezium/DebeziumUtils.java
2.3 flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlConnection.java
* Copyright Debezium Authors.
*
* Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
*/
package io.debezium.connector.mysql;
import io.debezium.DebeziumException;
import io.debezium.config.CommonConnectorConfig;
import io.debezium.config.CommonConnectorConfig.EventProcessingFailureHandlingMode;
import io.debezium.config.Configuration;
import io.debezium.config.Configuration.Builder;
import io.debezium.config.Field;
import io.debezium.connector.mysql.MySqlConnectorConfig.SecureConnectionMode;
import io.debezium.connector.mysql.legacy.MySqlJdbcContext.DatabaseLocales;
import io.debezium.jdbc.JdbcConnection;
import io.debezium.relational.Column;
import io.debezium.relational.Table;
import io.debezium.relational.TableId;
import io.debezium.relational.history.DatabaseHistory;
import io.debezium.schema.DatabaseSchema;
import io.debezium.util.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.OptionalLong;
import java.util.Properties;
/**
* Copied from Debezium project(1.6.4.final) to add custom jdbc properties in the jdbc url. The new
* parameter {@code jdbcProperties} in the constructor of {@link MySqlConnectionConfiguration} will
* be used to generate the jdbc url pattern, and may overwrite the default value.
*
* <p>Line 80: Add field {@code urlPattern} in {@link MySqlConnection} and remove old pattern.
*
* <p>Line 93: Init {@code urlPattern} using the url pattern from {@link
* MySqlConnectionConfiguration}.
*
* <p>Line 561: Generate the connection string by the new field {@code urlPattern}.
*
* <p>Line 574 ~ 581: Add new constant and field {@code urlPattern} to {@link
* MySqlConnectionConfiguration}.
*
* <p>Line 626 ~ 629: Init new field {@code urlPattern} in {@link MySqlConnectionConfiguration}.
*
* <p>Line 690 ~ 720: Add some methods helping to generate the url pattern and add default values.
*/
public class MySqlConnection extends JdbcConnection {
private static Logger LOGGER = LoggerFactory.getLogger(MySqlConnection.class);
private static final String SQL_SHOW_SYSTEM_VARIABLES = "SHOW VARIABLES";
private static final String SQL_SHOW_SYSTEM_VARIABLES_CHARACTER_SET =
"SHOW VARIABLES WHERE Variable_name IN ('character_set_server','collation_server')";
private static final String SQL_SHOW_SESSION_VARIABLE_SSL_VERSION =
"SHOW SESSION STATUS LIKE 'Ssl_version'";
private final Map<String, String> originalSystemProperties = new HashMap<>();
private final MySqlConnectionConfiguration connectionConfig;
private final MysqlFieldReader mysqlFieldReader;
private final String urlPattern;
/**
* Creates a new connection using the supplied configuration.
*
* @param connectionConfig {@link MySqlConnectionConfiguration} instance, may not be null.
* @param fieldReader binary or text protocol based readers
*/
public MySqlConnection(
MySqlConnectionConfiguration connectionConfig, MysqlFieldReader fieldReader) {
super(connection