git clone git@github.com:alibaba/canal.git
git co canal-$version #切换到对应的版本上
mvn clean install -Denv=release
show variables like 'log_%';
/var/lib/mysql
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
adapterLoader = new CanalAdapterLoader(adapterCanalConfig);
adapterLoader.init();
CanalAdapterLoader 会去初始化一个worker,这个worker应该是一个死循环
{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":null}],"database":"azkaban","es":1560762729000,"id":3,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762754852,"type":"INSERT"}
{"data":[{"project_id":"2","event_type":"6","event_time":"1544014976598","username":"azkaban","message":"Uploaded project files zip flow1.zip"}],"database":"azkaban","es":1560762751000,"id":4,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":[{"message":"Uploaded project files zip flow1.zip2"}],"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762776945,"type":"UPDATE"}
{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":"trans1"}],"database":"azkaban","es":1560762796000,"id":5,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762841819,"type":"INSERT"}
{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":"trans2"}],"database":"azkaban","es":1560763001000,"id":6,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560763027148,"type":"DELETE"}
AbstractCanalAdapterWorker 中有一个 process 方法, 是
CanalAdapterKafkaWorker实现的
writeOut 当中的
groupInnerExecutorService 大小和 outerAdapters 一致, 每次就是将 flatMessages 提交到adaptor 里面去
changeRow当中,
最关键就是这一句 Object finalValue = JdbcTypeUtil.typeConvert(table, columnName, columnValue, sqlType, mysqlType);
在这里面经过一系列的转换,把kafka里面的数据转成需要sync的数据
实际上就是RdbSyncService 当中的sync这个方法把数据刷到了目标数据库当中
===========================================================================
canal server
main 函数里面有这个方法
controller = new CanalController(properties);
controller.start();
这个真正server 启动的地方
autoScan = BooleanUtils.toBoolean(getProperty(properties, CanalConstants.CANAL_AUTO_SCAN));
if (autoScan) {
defaultAction = new InstanceAction() {
CanalStater
initGlobalConfig
初始化parser的时候使用的是这个函数 CanalInstanceWithManager
MysqlEventParser
RdsLocalBinlogEventParser 是 canalInstance的一个属性
AbstractCanalInstance 从这里开始Start的
beforeStartEventParser 当中的group这个说法主要是用来处理分库分表这种场景了的
public Message getWithoutAck(ClientIdentity clientIdentity, int batchSize, Long timeout, TimeUnit unit)
throws CanalServerException {
checkStart(clientIdentity.getDestination());
checkSubscribe(clientIdentity);
CanalInstance canalInstance = canalInstances.get(clientIdentity.getDestination());
synchronized (canalInstance) {
// 获取到流式数据中的最后一批获取的位置
PositionRange<LogPosition> positionRanges = canalInstance.getMetaManager().getLastestBatch(clientIdentity);
parser 获取到这个位点的过程如下:
findStartPositionInternal 尝试从内存当中获取 logPositionManager.getLatestIndexBy(destination)
这个logPosition 默认是一个failback型的,首先会去查内存,然后会去查这个项目上层目录的conf目录下面的 .dat文件,如果没有的化
会使用 findEndPositionWithMasterIdAndTimestamp 去数据库里面查
show master status
show binlog events
show slave hosts
select * from information_schema.processlist as p where p.command = 'Binlog Dump';
seek 就是从binlog里面4L 开始,