Canal 同步 (集群版)

目录

一.部署准备

1.应用场景

2.主要安装包和说明

3.整体流程图

二.部署步骤

1.Mysql 部署

1.1 拉取镜像

1.2 挂载本地目录并启动镜像

2.RocketMQ 部署

2.1.NameServer 安装

2.3.控制台安装

3.Canal Service 部署

3.1.下载并解压

3.2.主要目录介绍

3.3.conf\canal.properties 配置

3.4.conf\yuncai\instance.properties 配置

3.5.启动

4.Canal Adapter 部署

4.1.下载 adapter 并解压

4.2.conf/application.yml 配置

4.2. conf/rds/**.xml 配置

4.3 启动

5.公司内网部说明

5.1.部署图

---

一.部署准备

1.应用场景

　　数据同步，适用垮服务器备份,本文是基于阿里云RDS>内网自部署Mysql。

2.主要安装包和说明

        Mysql:数据库

        Canal Service:canal 服务端用来解析binlog日志发送到MQ

        Canal Adapter:canal 消费端，消费MQ

        RocketMQ:消息中间件

        第一版的时候是基于tcp模式，由于RDS binlog有存储大小限制并且canal是顺序消费，当RDS数据库操作频繁同时Canal同步消费过慢会导致binlog丢失的情况，后改成MQ异步解耦。

3.整体流程图

二.部署步骤

1.Mysql 部署

1.1 拉取镜像

docker pull mysql:5.7

1.2 挂载本地目录并启动镜像

  docker run -p 3306:3306 --name   mysql   --restart=always --privileged=true \
  -v /***/log:/var/log/mysql \
  -v /***/data:/var/lib/mysql \
  -v /***/conf:/etc/mysql \
  -v /etc/localtime:/etc/localtime:ro   \
  -e MYSQL_ROOT_PASSWORD=*** -d   mysql:

        ***需要按照自己情况实际调整

2.RocketMQ 部署

2.1.NameServer 安装

docker run -d -p 9876:9876 \
-v /***/rocketmq/namesrv/logs:/root/logs \
-v /***/rocketmq/namesrv/store:/root/store \
-v /***/rocketmq/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf \
--name rmqnamesrv \
apache/rocketmq:4.9.3 sh mqnamesrv

        ***需要按照自己情况实际调整

2.2.Broker 安装

docker run -d  -p 10911:10911 -p 10909:10909 \
-v /***/rocketmq/broker/logs:/root/logs \
-v /***/rocketmq/broker/store:/root/store \
-v /***/rocketmq/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf \
--name rmqbroker \
-e "NAMESRV_ADDR=ip1:9876;ip2:9876" \
apache/rocketmq:4.9.3 \
sh mqbroker \
-c /opt/rocketmq-4.4.0/conf/broker.conf autoCreateTopicEnable=true

        ***和ip需要按照自己情况实际调整

2.3.控制台安装

docker run --name rocketmq-console \
-e "JAVA_OPTS=-Drocketmq.namesrv.addr=ip1:9876;ip2:9876 \
 -Duser.timezone='Asia/Shanghai' \
-Dcom.rocketmq.sendMessageWithVIPChannel=false" \
-v /etc/localtime:/etc/localtime \
-p 38888:8080 -t styletang/rocketmq-console-ng

        控制台访问地址：http://ip:38888/#/consumer

3.Canal Service 部署

        canal 支持自建 mysql 和 Aliyun RDS，下方主要讲解 Aliyun RDS 的部署方式，非 Aliyun RDS 可自行参考

        https://github.com/alibaba/canal 官方文档。

3.1.下载并解压

wget  https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
tar  -zxvf canal.deployer-1.1.5.tar.gz

　　建议使用 1.1.5 以上，小于 1.1.5 有 jdk 兼容问题。

3.2.主要目录介绍

        bin     执行文件目录
        conf   配置文件目录
        logs   日志存放目录

3.3.conf\canal.properties 配置

        注意标红部分

Properties
iivi#################################################
#########                 common argument                #############
#################################################
# tcp bind ip
canal.ip =
# register ip to zookeeper
canal.register.ip =
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
# canal.user = canal
# canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

# canal admin config
#canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 21110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
#canal.admin.register.auto = true
#canal.admin.register.cluster =
#canal.admin.register.name =

canal.zkServers =
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, rocketMQ, rabbitMQ, pulsarMQ
canal.serverMode = rocketMQ
rocketmq.producer.group = canal_group
#MQ NameServer地址
rocketmq.namesrv.addr = ip1:9876;ip2:9876

# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 32768
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 4096
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  2048
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 32768
canal.instance.network.sendBufferSize = 32768
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false
canal.instance.filter.dml.insert = false
canal.instance.filter.dml.update = false
canal.instance.filter.dml.delete = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

# parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

# table meta tsdb info
canal.instance.tsdb.enable = false
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = canal
canal.instance.tsdb.dbPassword = canal
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360

#################################################
#########                 destinations                #############
#################################################
canal.destinations = yuncai
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5
# set this value to 'true' means that when binlog pos not found, skip to latest.
# WARN: pls keep 'false' in production env, or if you know what you want.
canal.auto.reset.latest.pos.mode = false

canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

3.4.conf\yuncai\instance.properties 配置

#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=0

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=*.mysql.rds.aliyuncs.com:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=***
canal.instance.rds.secretkey=****
#RDS实例ID
canal.instance.rds.instanceId=***
# table meta tsdb info
canal.instance.tsdb.enable=false
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# RDS username/password
canal.instance.dbUsername=***
canal.instance.dbPassword=***
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
# 库很多的时候可以配置去掉一些
canal.instance.filter.regex=库名1\\..*,库名2\\..*
canal.instance.filter.black.regex = 库名1\\..*,库名2\\..
# MQ topic需要和Canal Adapter的配置对上
canal.mq.topic=***

3.5.启动

sh  canal.deployer-1.1.5/bin/startup.sh

4.Canal Adapter 部署

4.1.下载 adapter 并解压

wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.adapter-1.1.5.tar.gz
tar  -zxvf canal.adapter-1.1.5.tar.gz

        建议使用 1.1.5 以上，小于 1.1.5 有 jdk 兼容问题。

4.2.conf/application.yml 配置

zerver:
 zport: 8064
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: rocketMQ  #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: 0
  timeout: 600
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    rocketmq.namespace:
    rocketmq.namesrv.addr: ip1:9876;ip2:9876 #mq地址
    rocketmq.batch.size: 1000
    rocketmq.enable.message.trace: false
    rocketmq.customized.trace.topic: *** #topic名称需要跟service的instance.properties对应上
    rocketmq.access.channel:
    rocketmq.subscribe.filter:
 

  srcDataSources:
    sysDS:
      url: *** #生产数据地址地址和账号密码，理论上能去掉？
      username: ***
      password: ****
  canalAdapters:
  - instance: *** #需要和rdb配置对应上
    groups:
    - groupId: *** #需要和rdb配置对应上
      outerAdapters:
      - name: logger
      - name: rdb
        key: *** #需要和rdb配置对应上
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://内网数据库地址:3306/库名1?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
          jdbc.username: ***
          jdbc.password: ***
      - name: rdb
        key: mysqlPs
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: dbc:mysql://内网数据库地址:3306/库名2?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
          jdbc.username: ***
          jdbc.password: ***

4.2. conf/rds/**.xml 配置

dataSourceKey: ***
destination: ***#需要和application.ym的instance配置对应上
groupId: *** 要和application.ym的groupId配置对应上
outerAdapterKey: *** #需要和application.ym的groups.key配置对应上
concurrent: true
dbMapping:
  mirrorDb: true
  database: 库名

4.3 启动

sh  canal.adapter-1.1.5/bin/startup.sh

5.公司内网部说明

5.1.部署图

        （.1）代表192.168.168.1环境

        （.2）代表192.168.168.2环境

        由于实际项目数据库数据量比较多，执行ddl语句会时间较长，所以部署了两套，一套处理非ddl，一套处理ddl语句。