canal.mq.flatMessage参数

canal.mq.flatMessage

是否为json格式
如果设置为false,对应MQ收到的消息为protobuf格式
需要通过CanalMessageDeserializer进行解码

canal.mq.flatMessage = true 生产到kafka的消息就是json的, 否则就是protobuf二进制的

$bin/kafka-console-consumer.sh --bootstrap-server 172.16.23.174:9092 --topic fanboshi.t1 --from-beginning
{"data":null,"database":"fanboshi","es":1583414454000,"id":2,"isDdl":true,"mysqlType":null,"old":null,"pkNames":null,"sql":"create table t1(id int not null auto_increment primary key,sname varchar(10))","sqlType":null,"table":"t1","ts":1583414454111,"type":"CREATE"}
{"data":[{"id":"1","sname":"fan"}],"database":"fanboshi","es":1583414520000,"id":3,"isDdl":false,"mysqlType":{"id":"int(11)","sname":"varchar(10)"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"sname":12},"table":"t1","ts":1583414520403,"type":"INSERT"}
{"data":[{"id":"1","sname":"fan"}],"database":"fanboshi","es":1583414520000,"id":1,"isDdl":false,"mysqlType":{"id":"int(11)","sname":"varchar(10)"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"sname":12},"table":"t1","ts":1583414603231,"type":"INSERT"}
{"data":[{"id":"2","sname":"bo"}],"database":"fanboshi","es":1583414621000,"id":2,"isDdl":false,"mysqlType":{"id":"int(11)","sname":"varchar(10)"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"sname":12},"table":"t1","ts":1583414621766,"type":"INSERT"}


*
mysql-bin.000002 *UTF-80֊.8fanboshiJt1P,Xb6
curtGtid*c30c6a02-4e32-11ea-84ec-fa163edcd14e:65460b

curtGtidSn64398b

curtGtidLct64397b
        _-+_C-+++1+,c30c6a02-4e32-11ea-84ec-fa163edcd14e:1-65460Pbid (0B1Ri++(11)#
                                                                                  _+a+e (0Bfa+R
                                                                                               +a_cha_(10)
+y_-+-bi+.000002  *UTF-80֊.8fa+b-_hiJ+1P+Xb6
c+_+G+id*c30c6a02-4e32-11ea-84ec-fa163edcd14e:65461b

c+_+G+idS+64399b

c+_+G+idLc+64398b
        _-+_C-+++1+,c30c6a02-4e32-11ea-84ec-fa163edcd14e:1-65461Pbid (0B2Ri++(11)"
                                                                                  _+a+e (0Bb-R
                                                                                              +a_cha_(10)

*

乱码部分就是protobuf的

protobuf性能要好很多. 1.1.5性能有很大提升. 即便canal.mq.flatMessage = true性能也比以前好了很多很多

canal.mq.flatMessage = true默认的格式没带GTID

https://github.com/alibaba/canal/wiki/Canal-MQ-Performance