centos7上基于zookeeper完成canal1.1.5的高可用HA搭建 + mysql多节点解析自动切换

1. 服务器列表

服务器名	服务器地址
canal1	192.168.23.31
canal2	192.168.23.32
canal3	192.168.23.33

2. zookeeper的分布式安装

可以参考我的这篇zookeeper的分布式安装教程

这里我们在3台服务器上进行安装, 我们这里安装的版本是3.6.3版本

3. 安装mysql(master-master架构)

可以参考我的这篇centos7上mysql 8.0.25版本master-master架构的搭建

这里我们在canal1和canal2上进行安装, 我们这里安装的版本是8.0.25版本, 这样可以保证mysql的高可用

4. canal的安装配置

4.1 mysql创建canal用户

[root@canal1 ~]# mysql -h 192.168.23.31 -u root -pRoot_123
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14
Server version: 8.0.25 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER canal IDENTIFIED BY 'Canal_123';
Query OK, 0 rows affected (0.01 sec)

mysql> ALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY 'Canal_123';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
Query OK, 0 rows affected (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

这里有个问题：canal2上的mysql，binlog有操作的记录，表user里面竟然也有了canal用户的数据，但是我们设置的是不同步mysql数据库的数据的，这里搞不懂，先不管了，后面再解决

4.2 下载和配置

[root@canal1 ~]# 
[root@canal1 ~]# wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
[root@canal1 ~]# 
[root@canal1 ~]# mkdir canal.deployer-1.1.5
[root@canal1 ~]# tar -zxvf canal.deployer-1.1.5.tar.gz -C canal.deployer-1.1.5
[root@canal1 ~]# 

配置canal1和canal2的/etc/my.cnf，内容如下，然后重启mysql

# 用于mysql多节点解析自动切换
log_slave_updates=true

配置conf/example/instance.properties

# canal2(1235)、canal3(1236)
canal.instance.mysql.slaveId=1234  
canal.instance.master.address=canal1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=Canal_123
canal.instance.connectionCharset = UTF-8
canal.instance.filter.regex=.*\\..*

# >>>>>>>>>>>>>>>>>用于mysql多节点解析自动切换>>>>>>>>>>>>>>>
canal.instance.standby.address = canal2:3306
# 需要开启心跳检查
canal.instance.detecting.enable = true 
# 心跳检查sql
canal.instance.detecting.sql = select 1
# 心跳检查频率
canal.instance.detecting.interval.time = 3 
# 心跳检查失败次数阀值，超过该阀值后会触发mysql链接切换，比如切换到standby机器上继续消费binlog
canal.instance.detecting.retry.threshold = 3  
# 心跳检查超过失败次数阀值后，是否开启master/standby的切换
canal.instance.detecting.heartbeatHaEnable = true 

配置conf/canal.properties

# 如果系统是1个 cpu，需设置为 false
canal.instance.parser.parallel = false

canal.zkServers = canal1:2181,canal2:2181,canal3:2181
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

其中canal.properties配置canal和instance的公共属性

4.3 分发和修改配置

[root@canal1 ~]# scp -r canal.deployer-1.1.5 root@canal2:/root
[root@canal1 ~]# scp -r canal.deployer-1.1.5 root@canal3:/root

修改canal2和canal3的conf/example/instance.properties

# canal2
canal.instance.mysql.slaveId=1235

# canal3
canal.instance.mysql.slaveId=1236

5. 启动

分别在3台服务器上启动canal

canal.deployer-1.1.5/bin/startup.sh

6. 检验

从上面3张图片可以看出，已经启动成功了，且instance只有canal1提供服务; 这说明一个instance只能在一台服务器上保存active, 其它的服务器为standby

从zookeeper也可以看出只有canal1提供服务

[zk: canal1:2181,canal2:2181,canal3:2181(CONNECTED) 0] 
[zk: canal1:2181,canal2:2181,canal3:2181(CONNECTED) 0] get /otter/canal/destinations/example/running
{"active":true,"address":"192.168.23.31:11111"}
[zk: canal1:2181,canal2:2181,canal3:2181(CONNECTED) 1] 

7. 客户端连接

7.1 服务端的高可用

CanalConnector connector = CanalConnectors.newClusterConnector("192.168.23.31:2181,192.168.23.32:2181,192.168.23.33:2181", "example", "", "");

客户端通过zookeeper进行连接，然后获取当前active的instance服务器，来保证服务端的高可用

7.2 客户端的高可用

[zk: canal1:2181,canal2:2181,canal3:2181(CONNECTED) 4] get /otter/canal/destinations/example/1001/running
{"active":true,"address":"192.168.23.1:53750","clientId":1001}

[zk: canal1:2181,canal2:2181,canal3:2181(CONNECTED) 7] get /otter/canal/destinations/example/1001/cursor
{"@type":"com.alibaba.otter.canal.protocol.position.LogPosition","identity":{"slaveId":-1,"sourceAddress":{"address":"canal1","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mysql-bin.000004","position":235,"serverId":1,"timestamp":1624526107000}}

可以在不同的服务器启动多个client(都是1001), 只有一个client是active状态， 其它的为standby, 来保证client的高可用

client消费的时候，canal server也会将client最后消费成功的binlog位点保存到zookeeper, 下次重启客户端的时候，可以继续消费, 保证消费的一致性

7.3 重复的测试

如果测试的时候，也可以修改或删除zookeeper的clientexample/1001/cursor数据，然后重启canal server, 进行多次重复测试

这是因为消费的postion都是保存在example/1001/cursor，但如果不重启canal server, 不会从instance.properties里面设置的position开始消费，而是从最新的position

执行的脚本如下：

[root@canal1 ~]# cat example_repeat_test.sh 
#!/usr/bin/env bash

# 删除zk的canal client position
/root/apache-zookeeper-3.6.3-bin/bin/zkCli.sh -server canal1:2181,canal2:2181,canal3:2181 delete /otter/canal/destinations/example/1001/cursor

canal_servers=('canal1' 'canal2' 'canal3')

# 重启3台服务的canal server
for canal_server in ${canal_servers[@]}
do
ssh root@${canal_server} << begin2end
source /root/.bashrc
/root/canal.deployer-1.1.5/bin/restart.sh
exit
begin2end
done

[root@canal1 ~]# 

在active canal server重启的时候，因为zookeeper对此节点有默认5秒的抢占active等待时间，大概率此节点的canal server还会是active