Canal1.1.6发送MySql的binlog日志到Rocketmq(2)

最新推荐文章于 2024-03-19 21:29:30 发布

黑暗行动

最新推荐文章于 2024-03-19 21:29:30 发布

阅读量635

点赞数 1

文章标签： rocketmq canal binlog采集

本文链接：https://blog.csdn.net/chy2z/article/details/130493222

版权

Canal环境搭建

1.1 下载canal.deployer

https://github.com/alibaba/canal/releases

1.2 文件目录结构

解压:canal.deployer-1.1.6.tar.gz

1.3 配置mysql数据库访问权限

创建canal的数据库访问账号

CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
GRANT SHOW VIEW, SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;

1.4 修改配置文件

配置文件有2个：

\canal.deployer-1.1.6\conf\example\instance.properties

\canal.deployer-1.1.6\canal.properties

1.5 修改连接MySQL和RocketMq配置

canal.properties 修改以下配置项目：

# 配置模式tcp, kafka, rocketMQ, rabbitMQ 
canal.serverMode = rocketMQ

# 指定实例目前包里自带example（可支持多个）
canal.destinations = example

# 配置mq的accessKey、secretKey,这里用原生rocektmq配置空
canal.aliyun.accessKey =
canal.aliyun.secretKey =

# 配置mq的group地址、group、topic及tag
rocketmq.producer.group = canal_group
rocketmq.enable.message.trace = false
rocketmq.customized.trace.topic = canal_topic
rocketmq.namespace =
rocketmq.namesrv.addr = 127.0.0.1:9876
rocketmq.retry.times.when.send.failed = 0
rocketmq.vip.channel.enabled = false
rocketmq.tag = canal_mysql_data_tag

conf\example\instance.properties 修改以下配置项目：

# 数据库连接
canal.instance.master.address=127.0.0.1:3306

# 数据库用户密码
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal

# 匹配数据库的正则(库.表)
canal.instance.filter.regex=.*\\..*

# mq的topic
canal.mq.topic=canal_topic

1.6 启动RocketMq

手工创建canal_topic

1.7 启动Canal

1.8 查看启动日志

canal.deployer-1.1.6\logs\canal\canal.log

2023-05-04 16:10:57.126 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 16:10:57.135 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2023-05-04 16:10:57.147 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2023-05-04 16:10:57.406 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.0.102(192.168.0.102):11111]
2023-05-04 16:10:59.386 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
2023-05-04 18:04:13.901 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 18:04:13.909 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
......
2023-05-04 18:39:59.525 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-05-04 18:39:59.533 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2023-05-04 18:39:59.936 [main] INFO  c.a.o.c.c.rocketmq.producer.CanalRocketMQProducer - ##Start RocketMQ producer##
2023-05-04 18:40:01.300 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2023-05-04 18:40:01.689 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.0.102(192.168.0.102):11111]
2023-05-04 18:40:04.093 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......

canal.deployer-1.1.6\logs\example\example.log

2023-05-04 16:10:59.318 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2023-05-04 16:10:59.334 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2023-05-04 16:10:59.334 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2023-05-04 16:10:59.345 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2023-05-04 16:10:59.435 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2023-05-04 16:10:59.435 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2023-05-04 16:11:04.011 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000001,position=4,serverId=1,gtid=<null>,timestamp=1683169976000] cost : 4566ms , the next step is binlog dump
2023-05-04 18:40:03.978 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2023-05-04 18:40:04.016 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2023-05-04 18:40:04.016 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2023-05-04 18:40:04.042 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2023-05-04 18:40:04.504 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2023-05-04 18:40:04.506 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2023-05-04 18:40:05.329 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000002,position=4,serverId=1,gtid=<null>,timestamp=1683191796000] cost : 805ms , the next step is binlog dump

看到上述日志说明启动成功了！

1.9 启动报错解决方法

1.9.1 CanalMQProducer类找不到

报错信息如下：

java.lang.IllegalStateException: Extension instance(name: rocketmq, class: interface com.alibaba.otter.canal.connector.core.spi.CanalMQProducer) could not be instantiated: class could not be found

问题原因：插件目录不正确，将canal.deployer-1.1.6\plugin中的jar包拷贝到 canal.deployer-1.1.6\lib\canal\plugin中，在重新启动服务即可

2023-05-04 18:05:01.494 [main] ERROR com.alibaba.otter.canal.deployer.CanalLauncher - ## Something goes wrong when starting up the canal Server:
java.lang.IllegalStateException: Extension instance(name: rocketmq, class: interface com.alibaba.otter.canal.connector.core.spi.CanalMQProducer)  could not be instantiated: class could not be found
	at com.alibaba.otter.canal.connector.core.spi.ExtensionLoader.createExtension(ExtensionLoader.java:169) ~[connector.core-1.1.6.jar:na]
	at com.alibaba.otter.canal.connector.core.spi.ExtensionLoader.getExtension(ExtensionLoader.java:121) ~[connector.core-1.1.6.jar:na]
	at com.alibaba.otter.canal.deployer.CanalStarter.start(CanalStarter.java:68) ~[canal.deployer-1.1.6.jar:na]
	at com.alibaba.otter.canal.deployer.CanalLauncher.main(CanalLauncher.java:124) ~[canal.deployer-1.1.6.jar:na]

1.9.2 TOPIC路由不存在

报错信息如下：

org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic

问题原因：canal_topic 没有创建成功，可以在RocketMQ提前创建好topic

[included=false,journalName=mysql-bin.000002,position=4,serverId=1,gtid=<null>,timestamp=1683191796000] cost : 805ms , the next step is binlog dump
2023-05-04 18:42:03.682 [pool-4-thread-1] ERROR c.a.o.c.c.rocketmq.producer.CanalRocketMQProducer - org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
java.lang.RuntimeException: org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
	at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:303) ~[na:na]
	at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:317) ~[na:na]
	at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.send(CanalRocketMQProducer.java:284) ~[na:na]
	at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.send(CanalRocketMQProducer.java:170) ~[na:na]
	at com.alibaba.otter.canal.server.CanalMQStarter.worker(CanalMQStarter.java:181) ~[canal.server-1.1.6.jar:na]
	at com.alibaba.otter.canal.server.CanalMQStarter.access$100(CanalMQStarter.java:24) ~[canal.server-1.1.6.jar:na]
	at com.alibaba.otter.canal.server.CanalMQStarter$CanalMQRunnable.run(CanalMQStarter.java:223) ~[canal.server-1.1.6.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.apache.rocketmq.client.exception.MQClientException: No route info for this topic, canal_topic
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
	at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendSelectImpl(DefaultMQProducerImpl.java:1133) ~[na:na]
	at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1092) ~[na:na]
	at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1087) ~[na:na]
	at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:508) ~[na:na]
	at com.alibaba.otter.canal.connector.rocketmq.producer.CanalRocketMQProducer.sendMessage(CanalRocketMQProducer.java:291) ~[na:na]
	... 9 common frames omitted