canal 相关问题

git clone git@github.com:alibaba/canal.git
git co canal-$version #切换到对应的版本上
mvn clean install -Denv=release

show variables like 'log_%';
/var/lib/mysql
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf

adapterLoader = new CanalAdapterLoader(adapterCanalConfig);
adapterLoader.init();

CanalAdapterLoader 会去初始化一个worker，这个worker应该是一个死循环

{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":null}],"database":"azkaban","es":1560762729000,"id":3,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762754852,"type":"INSERT"}
{"data":[{"project_id":"2","event_type":"6","event_time":"1544014976598","username":"azkaban","message":"Uploaded project files zip flow1.zip"}],"database":"azkaban","es":1560762751000,"id":4,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":[{"message":"Uploaded project files zip flow1.zip2"}],"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762776945,"type":"UPDATE"}
{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":"trans1"}],"database":"azkaban","es":1560762796000,"id":5,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560762841819,"type":"INSERT"}
{"data":[{"project_id":"3","event_type":"5","event_time":"1544078892117","username":"azkaban","message":"trans2"}],"database":"azkaban","es":1560763001000,"id":6,"isDdl":false,"mysqlType":{"project_id":"int(11)","event_type":"tinyint(4)","event_time":"bigint(20)","username":"varchar(64)","message":"varchar(512)"},"old":null,"pkNames":null,"sql":"","sqlType":{"project_id":4,"event_type":-6,"event_time":-5,"username":12,"message":12},"table":"project_events","ts":1560763027148,"type":"DELETE"}

AbstractCanalAdapterWorker  中有一个 process  方法， 是
CanalAdapterKafkaWorker实现的

writeOut  当中的
groupInnerExecutorService 大小和  outerAdapters 一致， 每次就是将 flatMessages 提交到adaptor 里面去

changeRow当中， 
最关键就是这一句 Object finalValue = JdbcTypeUtil.typeConvert(table, columnName, columnValue, sqlType, mysqlType);
在这里面经过一系列的转换，把kafka里面的数据转成需要sync的数据

实际上就是RdbSyncService 当中的sync这个方法把数据刷到了目标数据库当中

===========================================================================

 

canal server
main 函数里面有这个方法 

controller = new CanalController(properties);
controller.start();

这个真正server 启动的地方

autoScan = BooleanUtils.toBoolean(getProperty(properties, CanalConstants.CANAL_AUTO_SCAN));
        if (autoScan) {
            defaultAction = new InstanceAction() {
CanalStater

initGlobalConfig 
初始化parser的时候使用的是这个函数    CanalInstanceWithManager

MysqlEventParser    
RdsLocalBinlogEventParser  是 canalInstance的一个属性

AbstractCanalInstance   从这里开始Start的

beforeStartEventParser   当中的group这个说法主要是用来处理分库分表这种场景了的

public Message getWithoutAck(ClientIdentity clientIdentity, int batchSize, Long timeout, TimeUnit unit)
                                                                                                           throws CanalServerException {
        checkStart(clientIdentity.getDestination());
        checkSubscribe(clientIdentity);

        CanalInstance canalInstance = canalInstances.get(clientIdentity.getDestination());
        synchronized (canalInstance) {
            // 获取到流式数据中的最后一批获取的位置
            PositionRange<LogPosition> positionRanges = canalInstance.getMetaManager().getLastestBatch(clientIdentity);
            
            
parser 获取到这个位点的过程如下：
findStartPositionInternal   尝试从内存当中获取    logPositionManager.getLatestIndexBy(destination)
这个logPosition 默认是一个failback型的，首先会去查内存，然后会去查这个项目上层目录的conf目录下面的 .dat文件，如果没有的化
会使用 findEndPositionWithMasterIdAndTimestamp 去数据库里面查   

show master status
show binlog events

show slave hosts
select * from information_schema.processlist as p where p.command = 'Binlog Dump'; 

seek   就是从binlog里面4L 开始，