3.canal连接kafka传输binlog日志

默认值canal支持将MySQL订阅日志发送给kafka或者rocketMQ消息中间件,具体操作如下

1. 修改example下的 instance.properties

   #  按需修改成自己的数据库信息
   #################################################
   ...
   canal.instance.master.address=node1:3306
   # username/password,数据库的用户名和密码
   ...
   canal.instance.dbUsername = canal
   canal.instance.dbPassword = canal
   ...
   # mq config 对应MQ中具体是哪个主题 建议修改
   canal.mq.topic=example
   # 针对库名或者表名发送动态topic
   #canal.mq.dynamicTopic=mytest,.*,mytest.user,mytest\\..*,.*\\..*
   # 对应mq中的partition
   canal.mq.partition=0
   # hash partition config
   #canal.mq.partitionsNum=3
   #库名.表名: 唯一主键，多个表之间用逗号分隔
   #canal.mq.partitionHash=mytest.person:id,mytest.role:id
   #################################################
   

2. 修改canal.properties

   # ...
   # 可选项: tcp(默认), kafka,RocketMQ,rabbitmq,pulsarmq 将TCP修改成具体的MQ中间件
   canal.serverMode = kafka
   # ...
   
   # Canal的batch size, 默认50K, 由于kafka最大消息体限制请勿超过1M(900K以下)
   canal.mq.canalBatchSize = 50
   # Canal get数据的超时时间, 单位: 毫秒, 空为不限超时
   canal.mq.canalGetTimeout = 100
   # 是否为flat json格式对象
   canal.mq.flatMessage = false
   

3. 其他跟MQ相关的配置 详细参考官方文档 https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart 如果是>=1.1.5版本 canal将各个MQ配置做出了拆分 详细参考官网文档

   参数名	参数说明	默认值
   canal.mq.servers	kafka为bootstrap.servers rocketMQ中为nameserver列表	127.0.0.1:6667
   canal.mq.retries	发送失败重试次数	0
   canal.mq.batchSize	kafka为ProducerConfig.BATCH_SIZE_CONFIG rocketMQ无意义	16384
   canal.mq.maxRequestSize	kafka为ProducerConfig.MAX_REQUEST_SIZE_CONFIG rocketMQ无意义	1048576
   canal.mq.lingerMs	kafka为ProducerConfig.LINGER_MS_CONFIG , 如果是flatMessage格式建议将该值调大, 如: 200 rocketMQ无意义	1
   canal.mq.topic	mq里的topic名	无

启动kafka查看效果

./kafka-console-consumer.sh --bootstrap-server node1:9092 --topic example --from-beginning

{"data":[{"id":"1","name":"666"}],"database":"canal_test","es":1657417782000,"id":2,"isDdl":false,"mysqlType":{"id":"int(0)","name":"varchar(255)"},"old":[{"name":"44"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12},"table":"user","ts":1657417783117,"type":"UPDATE"}
{"data":[{"id":"4","name":"777"}],"database":"canal_test","es":1657417817000,"id":3,"isDdl":false,"mysqlType":{"id":"int(0)","name":"varchar(255)"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12},"table":"user","ts":1657417817639,"type":"INSERT"}
{"data":[{"id":"2","name":"555"}],"database":"canal_test","es":1657417820000,"id":4,"isDdl":false,"mysqlType":{"id":"int(0)","name":"varchar(255)"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12},"table":"user","ts":1657417820763,"type":"DELETE"}