中间件-canal

官网地址

一、mysql处理

1.修改配置文件

binlog_format=row

2.创建用户

CREATE USER canal IDENTIFIED BY 'passwd';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

二、kafka操作

创建对应topic

三、canal处理

1.下载

wget https://github.com/alibaba/canal/releases/download/canal-1.1.6/canal.deployer-1.1.6.tar.gz

2.解压

tar zxf canal.deployer-1.1.6.tar.gz

3.迁移目录

mv canal.deployer-1.1.6 /usr/local/canal

4.修改配置文件

vim /usr/local/canal/conf/example/instance.properties


canal.instance.master.address=aurora-host
canal.instance.dbUsername=aurora-user-name
canal.instance.dbPassword=aurora-user-passwd
canal.mq.topic=kafka-topic-name

vim /usr/local/canal/conf/canal.properties

canal.serverMode = kafka
kafka.bootstrap.servers = kafka_ip:kafka_port

5.启动

cd /usr/local/canal/bin
./startup.sh

四、注意事项

1.canal.instance.filter.regex不生效问题

首先确认一下是不是 binlog_rows_query_log_events=ON 这个开关打开了，如果不是的话就跳过
如果是的话，修改这一行配置 canal.instance.filter.query.dml = true （默认为 false，改为 true）试试

2.库表正则匹配规则

MySQL parses the table and Perl supports regex.
Use comma(,) for multiple regex, double backslash(\\) for escape characters.
Examples:
1. All the tables: .* or .*\\..*
2. All the tables in canal scheme: canal\\..*
3. Tables starting with canal in canal scheme: canal\\.canal.*
4. Access a table in canal scheme: canal.test1
5. A combination of multiple rules: canal\\..*, mysql.test1, mysql.test2 (with comma as delimeter)

3.重置binlog起始点

# 1.停止canal同步
./stop.sh
# 2.修改conf目录下对应同步任务的meta.dat文件
"postion":{"gtid":"","included":false,"journalName":"mysql-bin-changelog.001111","position":4
# 3.启动canal同步
./startup.sh