Canal使用篇

最新推荐文章于 2023-06-30 11:09:08 发布

扛枪蚂蚁

最新推荐文章于 2023-06-30 11:09:08 发布

阅读量593

点赞数

本文链接：https://blog.csdn.net/qq_30228707/article/details/103976143

版权

Java 专栏收录该内容

15 篇文章 1 订阅

订阅专栏

吐槽

Canal是一种新技术,对于新技术,大家都是懵逼的,最简单的是找准一个切入点,切进去,选择一个什么切入点呢?
一般选择官网,去官网看简易的Demo(吐槽:官网的demo很简易,只能满足你的一些低级需求,高级的就得看一些高手的研究文章)

官方文档

https://github.com/alibaba/canal

使用篇

linux安装canal server

#创建canal目录
mkdir /tmp/canal
#下载canal客户端
get https://github.com/alibaba/canal/releases/download/canal-1.1.0/canal.deployer-1.1.0.tar.gz
#解压客户端
tar -zxvf canal.deployer-1.1.0.tar.gz

canal.properties修改

#################################################
#########               common argument         #############
#################################################
canal.id= 1
canal.ip=xxx.xxx.xxx.xxx
canal.port=11111
canal.metrics.pull.port=11112
canal.zkServers=ip1:2181,ip2:2181,ip3:2181
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

# parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername=canal
canal.instance.tsdb.dbPassword=canal

# rds oss binlog account
canal.instance.rds.accesskey =
canal.instance.rds.secretkey =

#################################################
#########               destinations            #############
#################################################
canal.destinations= example
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.tsdb.spring.xml=classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml=classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring
canal.instance.global.lazy = false
#canal.instance.global.manager.address = 127.0.0.1:1099
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

备注:
canal.ip 当前虚拟机的地址
canal.zkServers 配置zk集群配置

instance.properties配置

#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=1234

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=xxx.xxx.xxx.xxx:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=123456
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=库名.表名
# table black regex
canal.instance.filter.black.regex=
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*

备注:
canal.instance.master.address=xxx.xxx.xxx.xxx:3306 数据库地址
canal.instance.dbUsername=root 用户名
canal.instance.dbPassword=123456 密码
canal.instance.filter.regex=库名.表名匹配表名

启动

sh startup.sh

在这里插入图片描述

验证

进入logs目录查看相应的日志校验是否启动成

在这里插入图片描述

进入example中查看运行异常

在这里插入图片描述

备注

客户端的使用请看官方文档,有各种语言的客户端调用；

小贴士

该篇描述的是Canal跟Zk集群的配置使用,使用之前需要开启Mysql binlog日志;还有很多的变种使用比如跟Kafka的搭配使用,官网基本没有这类的讲解
Canal搭建集群后,其实也没啥卵用,集群的作用只是容错
Canal注册到Zk上去了,node之间是不会互相通信的,一台挂了之后,客户端会连接另外一台Canal，读取的时候会从头到位再走一遍,因为binlog中的sql可以重复执行,所以全量走了一遍效果还是一样的.
同步的时候,Server会将binlog日志落地到meta文件,并且每天都会生成相应的meta文件日志,所以数据不会遗漏。
坑爹的事项:
Canal客户端调用的时候只能开启一台,如果要开启多台的话那对不起,需要再介入Kafka,
Binlog中sql如果过大会拆分多个sql，因此同步逻辑中如果拆分成多个sql对逻辑有影响的话需要额外注意了,需要你去处理,推荐一个方案,Canal中的数据结构中有一个executeTime,这个是binlog中带出来的,拆分成多个sql,这个时间戳是一致的
So，加油吧少年