canal介绍
canal是阿里巴巴旗下的一款开源项目,纯Java开发。基于数据库增量日志解析,提供增量数据订阅&消费,目前主要支持了MySQL(也支持mariaDB)。简单来说就是用户监听数据库,只要数据库有增删改的操作canal都会监听到,届时我们就可以做我们对应的操作。
支持的业务
- 数据库镜像
- 数据库实时备份
- 多级索引 (卖家和买家各自分库索引)
- search build
- 业务cache刷新
- 价格变化等重要业务消息
运行原理
从上图可以看出一主从分库是如何做到数据同步的。而我们的canal基本与上面一致,基本就是将自己模拟成为一个从库,从而一直监听主库的数据变化然后写入外界(通过消息队列发送给外界、或者通过tcp连接canal获取增量数据)
搭建canal
简单的熟悉下canal后我们可以直接开始搭建canal,canal的搭建过程很简单,废话不多,说我们开始吧!
工具准备
- 数据库
- canal
- 一台linux服务器(本文暂时只介绍linux的安装方法)
数据库操作
canal监听最开始的前提就是读取binlog的数据来进行监控。一般数据库的binlog是默认关了的,所以需要我们打开来.
在my.cnf 加入如下:
[mysqld]
og-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式
server_id=1 #配置mysql replaction需要定义,不能和canal的slaveId重复
ps:binlog模式总共有三种
- Statement : 记录每一条修改数据的SQL语句(批量修改时,记录的不是单条SQL语句,而是批量修改的SQL语句事件)
- row:记录的方式是行,即如果批量修改数据,记录的不是批量修改的SQL语句事件,而是每条记录被更改的SQL语句,因此,ROW模式的binlog日志文件会变得很“重”。
- Mixed:实际上就是前两种模式的结合。在Mixed模式下,MySQL会根据执行的每一条具体的sql语句来区分对待记录的日志形式,也就是在Statement和Row之间选择一种。
以上三个模式的解释来自于https://blog.51cto.com/13691477/2150178该blog的大佬,如果想继续深入了解可以直接看下他的原帖.
改完以后咱们可以进入数据库输入show variables like "%bin%;这条sql查看咱们的binlog是否开启
开启完binlog以后咱们需要给从库(canal)一个管理员账户:
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
进入数据库输入上面这些sql回车即刻。创建了一个canal 密码为canal的一个账户。到这里,数据库的操作就已经做完了,咱们就可以开始做canal的相关操作了。
canal操作
开始我们需要下载canal并解压下来。(我们这里使用的1.1.3的版本。如果想用其他版本你可以访问官网https://github.com/alibaba/canal/releases 下载自己需要的版本。)
进入canal官方的github下载地址为: https://github.com/alibaba/canal/releases/download/canal-1.1.3/canal.deployer-1.1.3.tar.gz
如果下载速度慢可以试试复制这个连接去迅雷下载 可能会比浏览器下载稍快些。
下载后我们需要解压
mkdir ./canal
tar -zxvf ./canal.deployer-1.1.3.tar.gz -C ./canal
因为压缩包里面压缩的不是一个整的文件夹 我们需要指定压缩到一个文件夹里面 要不然会很杂乱。
进入canal文件夹里面后会有这么几个文件
bin:执行文件
conf: 配置文件
lib:canal所使用的java包(因为canal本身就是java开发出来的,所以用到了一些jar包)
logs:canal产生的日志
接下来就直接该配置文件就行了。
修改 conf下面example的instance.properties文件
#################################################
## mysql serverId
canal.instance.mysql.slaveId=1
# position info
canal.instance.master.address=127.0.0.1:3306 #输入自己数据库的ip地址
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
# table meta tsdb info
#canal.instance.tsdb.enable=false
#canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
#canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
# username/password
canal.instance.dbUsername=canal #这里就输入我们上面新增的数据库用户就可以了
canal.instance.dbPassword=canal
canal.instance.defaultDatabaseName= #去除test
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################
修改上面文件就可以直接进入bin目录执行启动文件
root@lvvyi:~/canal/bin ./startup.sh
cd to /root/canal/bin for workaround relative path
LOG CONFIGURATION : /root/canal/bin/../conf/logback.xml
canal conf : /root/canal/bin/../conf/canal.properties
CLASSPATH :/root/canal/bin/../conf:/root/canal/bin/../lib/zookeeper-3.4.5.jar:/r oot/canal/bin/../lib/zkclient-0.10.jar:/root/canal/bin/../lib/spring-tx-3.2.18.R ELEASE.jar:/root/canal/bin/../lib/spring-orm-3.2.18.RELEASE.jar:/root/canal/bin/ ../lib/spring-jdbc-3.2.18.RELEASE.jar:/root/canal/bin/../lib/spring-expression-3 .2.18.RELEASE.jar:/root/canal/bin/../lib/spring-core-3.2.18.RELEASE.jar:/root/ca nal/bin/../lib/spring-context-3.2.18.RELEASE.jar:/root/canal/bin/../lib/spring-b eans-3.2.18.RELEASE.jar:/root/canal/bin/../lib/spring-aop-3.2.18.RELEASE.jar:/ro ot/canal/bin/../lib/snappy-java-1.1.7.1.jar:/root/canal/bin/../lib/slf4j-api-1.7 .12.jar:/root/canal/bin/../lib/simpleclient_pushgateway-0.4.0.jar:/root/canal/bi n/../lib/simpleclient_httpserver-0.4.0.jar:/root/canal/bin/../lib/simpleclient_h otspot-0.4.0.jar:/root/canal/bin/../lib/simpleclient_common-0.4.0.jar:/root/cana l/bin/../lib/simpleclient-0.4.0.jar:/root/canal/bin/../lib/scala-reflect-2.11.12 .jar:/root/canal/bin/../lib/scala-logging_2.11-3.8.0.jar:/root/canal/bin/../lib/ scala-library-2.11.12.jar:/root/canal/bin/../lib/rocketmq-remoting-4.3.0.jar:/ro ot/canal/bin/../lib/rocketmq-logging-4.3.0.jar:/root/canal/bin/../lib/rocketmq-c ommon-4.3.0.jar:/root/canal/bin/../lib/rocketmq-client-4.3.0.jar:/root/canal/bin /../lib/protobuf-java-3.6.1.jar:/root/canal/bin/../lib/oro-2.0.8.jar:/root/canal /bin/../lib/netty-tcnative-boringssl-static-1.1.33.Fork26.jar:/root/canal/bin/.. /lib/netty-all-4.1.6.Final.jar:/root/canal/bin/../lib/netty-3.2.2.Final.jar:/roo t/canal/bin/../lib/mysql-connector-java-5.1.47.jar:/root/canal/bin/../lib/metric s-core-2.2.0.jar:/root/canal/bin/../lib/lz4-java-1.4.1.jar:/root/canal/bin/../li b/logback-core-1.1.3.jar:/root/canal/bin/../lib/logback-classic-1.1.3.jar:/root/ canal/bin/../lib/kafka-clients-1.1.1.jar:/root/canal/bin/../lib/kafka_2.11-1.1.1 .jar:/root/canal/bin/../lib/jsr305-3.0.2.jar:/root/canal/bin/../lib/jopt-simple- 5.0.4.jar:/root/canal/bin/../lib/jctools-core-2.1.2.jar:/root/canal/bin/../lib/j cl-over-slf4j-1.7.12.jar:/root/canal/bin/../lib/jackson-databind-2.9.6.jar:/root /canal/bin/../lib/jackson-core-2.9.6.jar:/root/canal/bin/../lib/jackson-annotati ons-2.9.0.jar:/root/canal/bin/../lib/ibatis-sqlmap-2.3.4.726.jar:/root/canal/bin /../lib/httpcore-4.4.3.jar:/root/canal/bin/../lib/httpclient-4.5.1.jar:/root/can al/bin/../lib/h2-1.4.196.jar:/root/canal/bin/../lib/guava-18.0.jar:/root/canal/b in/../lib/fastsql-2.0.0_preview_855.jar:/root/canal/bin/../lib/fastjson-1.2.28.j ar:/root/canal/bin/../lib/druid-1.1.9.jar:/root/canal/bin/../lib/disruptor-3.4.2 .jar:/root/canal/bin/../lib/commons-logging-1.1.3.jar:/root/canal/bin/../lib/com mons-lang3-3.4.jar:/root/canal/bin/../lib/commons-lang-2.6.jar:/root/canal/bin/. ./lib/commons-io-2.4.jar:/root/canal/bin/../lib/commons-compress-1.9.jar:/root/c anal/bin/../lib/commons-codec-1.9.jar:/root/canal/bin/../lib/commons-beanutils-1 .8.2.jar:/root/canal/bin/../lib/canal.store-1.1.3.jar:/root/canal/bin/../lib/can al.sink-1.1.3.jar:/root/canal/bin/../lib/canal.server-1.1.3.jar:/root/canal/bin/ ../lib/canal.protocol-1.1.3.jar:/root/canal/bin/../lib/canal.prometheus-1.1.3.ja r:/root/canal/bin/../lib/canal.parse.driver-1.1.3.jar:/root/canal/bin/../lib/can al.parse.dbsync-1.1.3.jar:/root/canal/bin/../lib/canal.parse-1.1.3.jar:/root/can al/bin/../lib/canal.meta-1.1.3.jar:/root/canal/bin/../lib/canal.instance.spring- 1.1.3.jar:/root/canal/bin/../lib/canal.instance.manager-1.1.3.jar:/root/canal/bi n/../lib/canal.instance.core-1.1.3.jar:/root/canal/bin/../lib/canal.filter-1.1.3 .jar:/root/canal/bin/../lib/canal.deployer-1.1.3.jar:/root/canal/bin/../lib/cana l.common-1.1.3.jar:/root/canal/bin/../lib/aviator-2.2.1.jar:/root/canal/bin/../l ib/aopalliance-1.0.jar:/root/canal/bin/../lib/aliware-apache-rocketmq-cloud-1.0. jar:
cd to /root/canal/bin for continue
root@lvvyi:~/canal/bin
然后去log里面查看下启动日志;
root@lvvyi:~/canal# ls
bin conf lib logs
root@lvvyi:~/canal# cd logs/
root@lvvyi:~/canal/logs# ls
canal canalQueue example metrics
root@lvvyi:~/canal/logs# cd canal
root@lvvyi:~/canal/logs/canal# ls
2020-08-17 2020-08-18 2020-08-19 2020-08-20 2020-08-24 2020-08-25 2020-08-26 2020-08-27 begin canal.log init
root@lvvyi:~/canal/logs/canal# tail -f canal.log
at com.alibaba.fastsql.sql.parser.SQLStatementParser.parseStatementList(SQLStatementParser.java:228) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.SQLUtils.parseStatements(SQLUtils.java:536) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.repository.SchemaRepository.console(SchemaRepository.java:439) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.MemoryTableMeta.apply(MemoryTableMeta.java:81) ~[canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.applyHistoryOnMemory(DatabaseTableMeta.java:463) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.rollback(DatabaseTableMeta.java:174) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.processTableMeta(AbstractMysqlEventParser.java:106) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:192) [canal.parse-1.1.3.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_261]
2020-08-31 16:14:29.624 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000113,position=58895318,serverId=123,gtid=,timestamp=1598861649000] cost : 651ms , the next step is binlog dump
the next step is binlog dump
出现上面这句话就说明是一家启动成功了 我这边是因为数据库有些问题上面才会报些waring,但是并不影响使用。
结束
以上就是canal的搭建方式 非常简单,但是也有很多的坑,如果你在搭建中遇到了些什么问题,欢迎在本文章下面留言,可以一起探讨下!