Flink CDC实践

使用 Flink CDC 3.0 ELT 从 MySQL 流传输到 Doris

配置环境

参考https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/mysql-doris-pipeline-tutorial.html

前提：安装JDK环境！！！
参考：Centos7下安装java8

1. 下载flink安装包
   从官网https://flink.apache.org/downloads/下载flink安装包，网速较慢，可以尝试在国内镜像网址https://mirrors.cloud.tencent.com/apache/flink下载，上传到服务器
   

2. 解压flink安装包

   mkdir /usr/local/flink
   tar -zxvf /opt/flink-1.16.0-bin-scala_2.12.tgz -C /usr/local/flink
   

3. 配置flink环境变量

   sudo vi /etc/profile.d/myprofile.sh
   export FLINK_HOME=/usr/local/flink/flink-1.16.0
   

   source /etc/profile
   

4. 关闭防火墙，配置

   systemctl status firewalld
   systemctl stop firewalld
   systemctl disable firewalld //禁止开机启动
   

   cd /usr/local/flink/flink-1.16.0/conf
   vi flink-conf.yaml
   配置Rest & web frontend项：rest.bind-address: 0.0.0.0
   

5. 启动flink，查看启动是否成功

   cd /usr/local/flink/flink-1.16.0/bin
   ./start-cluster.sh
   jps  //查看是否启动成功
   

   访问http://虚拟机ip:8081，查看flink webUI是否正常访问

   ifconfig虚拟机ip查看：
   
   备注：如果输入ifconfig没有找到命令执行：

   yum search ifconfig
   yum install net-tools.x86_64
   

   

使用 FlinkCDC cli 提交作业

1. 解压flink-cdc安装包

   tar -zxvf /opt/flink-cdc-3.0.0-bin.tar.gz -C /usr/local/flink
   

2. 将mysql和doris的驱动包移动到lib目录下

   mv /opt/flink-cdc-pipeline-connector-mysql-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/lib
   mv /opt/flink-cdc-pipeline-connector-doris-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/lib
   

3. 配置yaml文件

   cd /usr/local/flink/flink-cdc-3.0.0/
   mkdir pipeline
   vi ./pipeline/mysqlcdc_to_doris.yaml
   

   	################################################################################
   	# Description: Sync MySQL all tables to Doris
   	################################################################################
   	source:
   	  type: mysql
   	  hostname: localhost
   	  port: 3306
   	  username: xxx
   	  password: xxx
   	  tables: app_db.\.*
   	  server-id: 5400-5404
   	  server-time-zone: UTC
   	
   	sink:
   	  type: doris
   	  fenodes: 127.0.0.1:8030
   	  username: xxx
   	  password: xxx
   	  table.create.properties.light_schema_change: true
   	  table.create.properties.replication_num: 1
   	
   	pipeline:
   	  name: Sync MySQL Database to Doris
   	  parallelism: 2
   

4. 启动

   cd /usr/local/flink/flink-cdc-3.0.0/
   bash ./bin/flink-cdc.sh ./pipeline/mysqlcdc_to_doris.yaml
   

出现的问题

1. flinksql-Could not acquire the minimum required resources
   无法获取所需的最低资源，其实是可用任务槽太少，任务过多时，任务槽就不够用了
   解决方法：修改配置文件 flink-conf.yaml，重新设置任务槽的数量

   cd /usr/local/flink/flink-1.16.0/conf
   vi flink-conf.yaml
   taskmanager.numberOfTaskSlots: 30