Flink doris connector 整合Flink mysql CDC 实现实时数据入库

  Apache Doris 代码仓库地址：apache/incubator-doris 欢迎大家关注加星
 

---

通过flink cdc实现mysql binlog日志数据的消费，然后通过flink doris connector sql实时导入mysql数据到doris表数据中

 org.apache.doris.demo.flink.FlinkConnectorMysqlCDCDemo

注意: 由于Flink doris connector jar包不在Maven中央仓库中，需要单独编译并添加到你项目的classpath中。 参考Flink doris connector的编译和使用: Flink doris connector

1. 首先启用Mysql binlog
   具体如何打开binlog请自行搜索或到Mysql官方文档查询
2. 安装Flink
   Flink的安装和使用这里不做介绍，只是在开发环境中给出代码示例
3. 创建Mysql数据库表

 CREATE TABLE `test` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
 ) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

4. 创建doris表

CREATE TABLE `doris_test` (
  `id` int NULL COMMENT "",
  `name` varchar(100) NULL COMMENT ""
 ) ENGINE=OLAP
 DUPLICATE KEY(`id`)
 COMMENT "OLAP"
 DISTRIBUTED BY HASH(`id`) BUCKETS 1
 PROPERTIES (
 "replication_num" = "3",
 "in_memory" = "false",
 "storage_format" = "V2"
 );

5. 创建Flink Mysql CDC

tEnv.executeSql(
  "CREATE TABLE orders (\n" +
  "  id INT,\n" +
  "  name STRING\n" +
  ") WITH (\n" +
  "  'connector' = 'mysql-cdc',\n" +
  "  'hostname' = 'localhost',\n" +
  "  'port' = '3306',\n" +
  "  'username' = 'root',\n" +
  "  'password' = 'zhangfeng',\n" +
  "  'database-name' = 'demo',\n" +
  "  'table-name' = 'test'\n" +
  ")");

6. 创建Flink Doris Table 映射表

 tEnv.executeSql(
  "CREATE TABLE doris_test_sink (" +
  "id INT," +
  "name STRING" +
  ") " +
  "WITH (\n" +
  "  'connector' = 'doris',\n" +
  "  'fenodes' = '10.220.146.10:8030',\n" +
  "  'table.identifier' = 'test_2.doris_test',\n" +
  "  'sink.batch.size' = '2',\n" +
  "  'username' = 'root',\n" +
  "  'password' = ''\n" +
  ")");

7. 执行插入操作

 tEnv.executeSql("INSERT INTO doris_test_sink select id,name from orders");