深入理解 MySQL —— canal 安装和配置

1. mysql 开启 bin log

1. 修改 /etc/my.cnf

   $ vim /etc/my.cnf
   

   添加内容：

   [mysqld] 
   log-bin=mysql-bin
   binlog-format=ROW
   server-id=102
   
   

2. 重启 mysql

   $ service mysqld restart
   

3. 创建 canal 用户，并授权

   CREATE USER canal IDENTIFIED BY 'canal';    
   GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';  
   -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;  
   FLUSH PRIVILEGES; 
   

   查看权限：

   show grants for 'canal' 
   

2 安装 canal

1. 下载 canal 源码

   $ wget https://github.com/alibaba/canal/archive/canal-1.1.4.tar.gz
   

2. 解压 & 修改名称

   $ tar -zvxf canal-canal-1.1.4.tar.gz
   $ mv canal-canal-1.1.4/ canal
   

3. 编译

   $ mvn clean install -Denv=release -Dmaven.test.skip=true
   

   编译完成后在 target 目录得到：

   canal.adapter-1.1.4.tar.gz
   canal.admin-1.1.4.tar.gz
   canal.deployer-1.1.4.tar.gz
   canal.example-1.1.4.tar.gz
   

4. 解压 canal.adapter-1.1.4.tar.gz

   $ mkdir /opt/module/canal
   $ mkdir /opt/module/canal/canal-deployer
   $ tar -zvxf canal.deployer-1.1.4.tar.gz -C /opt/module/canal/canal-deployer
   

5. 配置 instance

   $ cd /opt/module/canal/canal-deployer/conf/example
   $ vim instance.properties
   canal.instance.master.address=192.168.248.102:3306
   

6. 修改 canal.properties

   $ vim canal.properties
   canal.user = canal
   canal.passwd = canal
   

7. 启动

   $ ./bin/startup.sh
   

   启动时如果报

   Caused by: java.io.IOException: Error When doing Client Authentication:ErrorPacket [errorNumber=1045, fieldCount=-1, message=Access denied for user 'canal'@'hadoop102' (using password: YES), sqlState=28000, sqlStateMarker=#]
   	at com.alibaba.otter.canal.parse.driver.mysql.MysqlConnector.negotiate(MysqlConnector.java:271)
   	at com.alibaba.otter.canal.parse.driver.mysql.MysqlConnector.connect(MysqlConnector.java:80)
   	... 4 more
   

   在 mysql 中执行

   delete from user where user=''; 
   flush privileges; 
   

8. 查看启动日志

   2021-01-16 00:28:14.972 [destination = example , address = /192.168.248.102:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
   2021-01-16 00:28:14.973 [destination = example , address = /192.168.248.102:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
   2021-01-16 00:28:16.497 [destination = example , address = /192.168.248.102:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000003,position=2665,serverId=102,gtid=,timestamp=1610728089000] cost : 1494ms , the next step is binlog dump
   

   说明启动成功

3. HA 模式配置

1. 复制 canal-canal-deployer 到 hadoop103

2. 修改 canal.properties

   canal.zkServers=hadoop102:2181,hadoop103:2181,hadoop104:2181
   canal.instance.global.spring.xml = classpath:spring/default-instance.xml
   

3. 修改 instance.properties

   hadoop102：

   canal.instance.mysql.slaveId=102
   

   hadoop103：

   canal.instance.mysql.slaveId=103
   

   注意：两台机器上的 instance 目录名字要保证完全一致，HA 模式依赖于instance name 进行管理，同时必须都选择default-instance.xml配置

4. 编写启动脚本 canal.sh

   case $1 in
   "start"){
           for i in hadoop102 hadoop103 
           do
                   ssh $i "/opt/module/canal/canal-deployer/bin/startup.sh"
           done
   };;
   "stop"){
           for i in hadoop102 hadoop103 
           do
                   ssh $i "/opt/module/canal/canal-deployer/bin/stop.sh"
           done
   };;
   esac
   

5. 启动 canal

   启动之前需要确保 zookeeper 已经启动

   [dwjf321@hadoop102 bin]$ ./canal.sh start
   

6. 查看 canal 在 zookeeper 中的信息

   [zk: localhost:2181(CONNECTED) 12] get /otter/canal/destinations/example/running
   {"active":true,"address":"192.168.122.1:11111"}
   

4. canal-admin 安装

1. 解压 canal.admin-1.1.4.tar.gz

   $ tar zxvf canal.admin-1.1.4.tar.gz -C /opt/module/canal/canal-admin
   

2. 修改 application.yml

   $ cd /opt/module/canal/canal-admin/conf
   $ vim application.yml
   

   内容如下：

   server:
     port: 8089
   spring:
     jackson:
       date-format: yyyy-MM-dd HH:mm:ss
       time-zone: GMT+8
   
   spring.datasource:
     address: hadoop102:3306
     database: canal_manager
     username: canal
     password: canal
     driver-class-name: com.mysql.jdbc.Driver
     url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
     hikari:
       maximum-pool-size: 30
       minimum-idle: 1
   
   canal:
     adminUser: admin
     adminPasswd: admin
   

3. 初始化元数据库

   $ mysql -ucanal -p
   

   导入初始化 sql

   mysql> source canal_manager.sql
   

4. 启动

   $ bin/startup.sh
   

5. 访问 http://hadoop102:8089/，默认密码是 admin/123456。

6. 关闭

   $ bin/stop.sh
   

7. canal-server 端配置

   使用 canal_local.properties的配置覆盖 canal.properties

   $ vim canal_local.properties
   

   hadoop102 和 hadoop103 都添加如下内容

   canal.register.ip =
   
   # canal admin config
   canal.admin.manager = hadoop102:8089
   canal.admin.port = 11110
   canal.admin.user = admin
   canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
   # admin auto register
   canal.admin.register.auto = true
   canal.admin.register.cluster =
   

   hadoop102:2181,hadoop103:2181,hadoop104:2181

参考文档：

https://github.com/alibaba/canal/wiki/AdminGuide