Canal环境搭建
1.1 下载canal.deployer
https://github.com/alibaba/canal/releases
1.2 文件目录结构
解压:canal.deployer-1.1.6.tar.gz
1.3 配置mysql数据库访问权限
创建canal的数据库访问账号
CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
GRANT SHOW VIEW, SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
1.4 修改配置文件
配置文件有2个:
\canal.deployer-1.1.6\conf\example\instance.properties
\canal.deployer-1.1.6\canal.properties
1.5 修改连接MySQL和RocketMq配置
canal.properties 修改以下配置项目:
# 配置模式tcp, kafka, rocketMQ, rabbitMQ
canal.serverMode = rocketMQ
# 指定实例目前包里自带example(可支持多个)
canal.destinations = example
# 配置mq的accessKey、secretKey,这里用原生rocektmq配置空
canal.aliyun.accessKey =
canal.aliyun.secretKey =
# 配置mq的group地址、group、topic及tag
rocketmq.producer.group = canal_group
rocketmq.enable.message.trace = false
rocketmq.customized.trace.topic = canal_topic
rocketmq.namespace =
rocketmq.namesrv.addr = 127.0.0.1:9876
rocketmq.retry.times.when.send.failed = 0
rocketmq.vip.channel.enabled = false
rocketmq.tag = canal_mysql_data_tag
conf\example\instance.properties 修改以下配置项目:
# 数据库连接
canal.instance.master.address=127.0.0.1:3306
# 数据库用户密码
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
# 匹配数据库的正则(库.表)
canal.instance.filter.regex=.*\\..*
# mq的topic
canal.mq.topic=canal_topic
1.6 启动RocketMq
手工创建canal_topic
1.7 启动Canal
1.8 查看启动日志
canal.deployer-1.1.6\logs\canal\canal.log
2023-05-04 16:10:57.126 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 16:10:57.135 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2023-05-04 16:10:57.147 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2023-05-04 16:10:57.406 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.0.102(192.168.0.102):11111]
2023-05-04 16:10:59.386 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
2023-05-04 18:04:13.901 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 18:04:13.909 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
......
2023-05-04 18:39:59.525 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 18:39:59.533 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2023-05-04 18:39:59.936 [main] INFO c.a.o.c.c.rocketmq.producer.CanalRocketMQProducer - ##Start RocketMQ producer##
2023-05-04 18:40:01.300 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2023-05-04 18:40:01.689 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.0.102(192.168.0.102):11111]
2023-05-04 18:40:04.093 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
canal.deployer-1.1.6\logs\example\example.log
2023-05-04 16:10:59.318 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2023-05-04 16:10:59.334 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2023-05-04 16:10:59.334 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2023-05-04 16:10:59.345 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2023-05-04 16:10:59.435 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2023-05-04 16:10:59.435 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2023-05-04 16:11:04.011 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000001,position=4,serverId=1,gtid=<null>,timestamp=1683169976000] cost : 4566ms , the next step is binlog dump
2023-05-04 18:40:03.978 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2023-05-04 18:40:04.016 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2023-05-04 18:40:04.016 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2023-05-04 18:40:04.042 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2023-05-04 18:40:04.504 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2023-05-04 18:40:04.506 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2023-05-04 18:40:05.329 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000002,position=4,serverId=1,gtid=<null>,timestamp=1683191796000] cost : 805ms , the next step is binlog dump
看到上述日志说明启动成功了!
1.9 启动报错解决方法
1.9.1 CanalMQProducer类找不到
报错信息如下:
java.lang.IllegalStateException: Extension instance(name: rocketmq, class: interface com.alibaba.otter.canal.connector.core.spi.CanalMQProducer) could not be instantiated: class could not be found
问题原因:插件目录不正确,将canal.deployer-1.1.6\plugin中的jar包拷贝到 canal.deployer-1.1.6\lib\canal\plugin中,在重新启动服务即可
2023-05-04 18:05:01.494 [main] ERROR com.alibaba.otter.canal.deployer.CanalLauncher - ## Something goes wrong when starting up the canal Server:
java.lang.IllegalStateException: Extension instance(name: rocketmq, class: interface com.alibaba.otter.canal.connector.core.spi.CanalMQProducer) could not be instantiated: class could not be found
at com.alibaba.otter.canal.connector.core.spi.ExtensionLoader.createExtension(ExtensionLoader.java:169) ~[connector.core-1.1.6.jar:na]
at com.alibaba.otter.canal.connector.core.spi.ExtensionLoader.getExtension(ExtensionLoader.java:121) ~[connector.core-1.1.6.jar:na]
at com.alibaba.otter.canal.deployer.CanalStarter.start(CanalStarter.java:68) ~[canal.deployer-1.1.6.jar:na]
at com.alibaba.otter.canal.deployer.CanalLauncher.main(CanalLauncher.java:124) ~[canal.deployer-1.1.6.jar:na]
1.9.2 TOPIC路由不存在
报错信息如下:
org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
问题原因:canal_topic 没有创建成功,可以在RocketMQ提前创建好topic
[included=false,journalName=mysql-bin.000002,position=4,serverId=1,gtid=<null>,timestamp=1683191796000] cost : 805ms , the next step is binlog dump
2023-05-04 18:42:03.682 [pool-4-thread-1] ERROR c.a.o.c.c.rocketmq.producer.CanalRocketMQProducer - org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
java.lang.RuntimeException: org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:303) ~[na:na]
at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:317) ~[na:na]
at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.send(CanalRocketMQProducer.java:284) ~[na:na]
at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.send(CanalRocketMQProducer.java:170) ~[na:na]
at com.alibaba.otter.canal.server.CanalMQStarter.worker(CanalMQStarter.java:181) ~[canal.server-1.1.6.jar:na]
at com.alibaba.otter.canal.server.CanalMQStarter.access$100(CanalMQStarter.java:24) ~[canal.server-1.1.6.jar:na]
at com.alibaba.otter.canal.server.CanalMQStarter$CanalMQRunnable.run(CanalMQStarter.java:223) ~[canal.server-1.1.6.jar:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendSelectImpl(DefaultMQProducerImpl.java:1133) ~[na:na]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1092) ~[na:na]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1087) ~[na:na]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:508) ~[na:na]
at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:291) ~[na:na]
... 9 common frames omitted