Canal之HA高可用

最新推荐文章于 2024-06-06 12:05:53 发布

码上中年

最新推荐文章于 2024-06-06 12:05:53 发布

阅读量435

点赞数 1

分类专栏：大数据组件学习 canal工具学习系统文章标签： mysql etl

本文链接：https://blog.csdn.net/q18729096963/article/details/114952933

版权

大数据组件学习同时被 3 个专栏收录

23 篇文章 1 订阅

订阅专栏

系统

20 篇文章 1 订阅

订阅专栏

canal工具学习

3 篇文章 0 订阅

订阅专栏

1. canal高可用

1.1 canal的HA分为两个部分：clientHA和serverHA
1.2 canal的HA需要借助于zookeeper来实现高可用
1.3 serverHA：针对每一个实例instance要做到同一时间只能有一个处于running状态，其他处于standby状态
1.4 clientHA：同一时间只能有一个client进行接收instance的数据，否则无法保证有序

2. canal serverHA实现

2.1 修改canal server的配置文件canal/conf/canal.properties
canal.zkServers = cdh-master:2181,cdh-worker1,cdh-worker2
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
2.2 分发整个canal文件夹到其他机器（可以是多台）
2.3 分发后修改canal server的实例配置canal/conf/example/instance.properties
canal.instance.mysql.slaveId=1235//其他机器也修改不同
2.4 启动各个机器的canal server： ./canal/bin/startup.sh
2.5 客户端client修改连接方式为：集群连接法
// 1. 创建连接(单节点canal server)
//        CanalConnector canalConnector = CanalConnectors.newSingleConnector(new InetSocketAddress("113.143.100.140", 11111),
//                "example", "", "");
        // 1.1 canal Server HA
        CanalConnector canalConnector = CanalConnectors.newClusterConnector("172.16.149.150:2181,172.16.149.151:2181,172.16.149.152:2181",
                "example", "", "");
2.6 测试效果

3. 附加serverHA完整代码

3.1 canal.properties

#################################################
#########               common argument         #############
#################################################
# tcp bind ip
canal.ip =
# register ip to zookeeper
canal.register.ip =
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
# canal.user = canal
# canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

# canal admin config
#canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441

canal.zkServers = cdh-master:2181,cdh-worker1,cdh-worker2
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = tcp
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024 
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED 
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

# parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

# table meta tsdb info
canal.instance.tsdb.enable = false
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = root
canal.instance.tsdb.dbPassword = 123456
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360

# aliyun ak/sk , support rds/mq
canal.aliyun.accessKey =
canal.aliyun.secretKey =

#################################################
#########               destinations            #############
#################################################
canal.destinations = example
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

##################################################
#########                    MQ                      #############
##################################################
canal.mq.servers = 127.0.0.1:6667
canal.mq.retries = 0
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 100
canal.mq.bufferMemory = 33554432
canal.mq.canalBatchSize = 50
canal.mq.canalGetTimeout = 100
canal.mq.flatMessage = true
canal.mq.compressionType = none
canal.mq.acks = all
#canal.mq.properties. =
canal.mq.producerGroup = test
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

##################################################
#########     Kafka Kerberos Info    #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = "../conf/kerberos/krb5.conf"
canal.mq.kafka.kerberos.jaasFilePath = "../conf/kerberos/jaas.conf"

3.2 instance.properties

#################################################
## mysql serverId , v1.0.26+ will autoGen
canal.instance.mysql.slaveId=1235

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=113.143.100.140:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=false
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=123456
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..

3.3 client完整代码（未做clientHA）

package com.qu.canal;

import com.alibaba.fastjson.JSON;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.alibaba.otter.canal.protocol.Message;
import com.google.protobuf.InvalidProtocolBufferException;
import com.qu.canal.proto.vo.CanalMessageModel;
import org.springframework.web.servlet.HandlerAdapter;

import java.net.InetSocketAddress;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * canal客户端程序： canal是cs架构，server端不需要编写代码 直接部署在Linux即可
 * 客户端程序需要开发 连接服务端 与canal服务端进行交互
 */
public class CanalClient {

    public static void main(String[] args) {
        /**
         * 实现步骤：
         *  1 创建连接
         *  2 建立连接
         *  3 订阅主题
         *  4 获取数据
         *  5 递交确认
         *  6 关闭连接
         */
        // 1. 创建连接(单节点canal server)
//        CanalConnector canalConnector = CanalConnectors.newSingleConnector(new InetSocketAddress("113.143.100.140", 11111),
//                "example", "", "");
        // 1.1 canal Server HA
        CanalConnector canalConnector = CanalConnectors.newClusterConnector("172.16.149.150:2181,172.16.149.151:2181,172.16.149.152:2181",
                "example", "", "");
        // 定义一个标记 不停的拉取数据
        boolean isRunning = true;
        try{
            // 2. 建立连接
            canalConnector.connect();
            //  宕机后重启 设置回滚上一次的请求，重新获取数据
            canalConnector.rollback();
            // 3. 订阅主题
            canalConnector.subscribe("test.person");
            // 4. 不停的拉取数据
            while (isRunning){
                // 获取数据
                Message message = canalConnector.getWithoutAck(1000);
                // 获取batch的ID
                long batchId= message.getId();
                // 获取binlog日志的数据总数
                List<CanalEntry.Entry> entries = message.getEntries();
                if(entries.size() > 0){
                    // 有数据再处理
                    System.out.println("转换成json字符串 ===> " + dealDatas(entries));
                    //将message转换成json字符串太占内存资源和网络资源，所以使用protobuf格式
                    //1. 将message对象转换成protobuf的字节码数据
                    byte[] data_bytes = entries2protobuf(entries);
                    System.out.println("转换成protobuf字节码 ===> " + data_bytes);
                    // 此处只是测试一下 反序列化
                    CanalMessageModel.RowData rowData = CanalMessageModel.RowData.parseFrom(data_bytes);
                    System.out.println("protobuf反序列化后的可是数据对象 ===> " + rowData);
                    //2. 将该字节码数据发送到Kafka集群，进行策略式存储（一般是7天）

                }
                // 5 递交确认
                canalConnector.ack(batchId);
            }
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            // 6. 关闭canal连接
            canalConnector.disconnect();
        }

    }

    public static byte[] entries2protobuf(List<CanalEntry.Entry> entries){


        // 构建protobuf的MessageModel.RowData实体对象
        CanalMessageModel.RowData.Builder rowDataBuilder = CanalMessageModel.RowData.newBuilder();

        for (CanalEntry.Entry entry: entries) {
            // entry 就是mysql的一条记录

            // 首先判断数据类型 当是binlog日志数据的开始和结尾 便不做任何处理
            // binlog日志数据的格式为： 事务头BEGIN｜数据ROWDATA｜事务尾END
            CanalEntry.EntryType entryType = entry.getEntryType();
            if(entryType == CanalEntry.EntryType.TRANSACTIONBEGIN || entryType == CanalEntry.EntryType.TRANSACTIONEND){
                continue;
            }

            // 获取数据的ROWDATA
            // 获取binlog文件名
            CanalEntry.Header header = entry.getHeader();
            String logfileName = header.getLogfileName();
            // 获取log file的偏移量
            long logfileOffset = header.getLogfileOffset();
            // 获取sql语句执行的时间戳
            long executeTime = header.getExecuteTime();
            // 获取数据库名
            String database = header.getSchemaName();
            // 获取表名
            String tableName = header.getTableName();
            // 获取时间的类型 insert/update/delete
            CanalEntry.EventType eventType = header.getEventType();
//            System.out.println("事件类型：" + eventType.toString().toLowerCase());

            rowDataBuilder.setLogfileName(logfileName);
            rowDataBuilder.setLogfileOffset(logfileOffset);
            rowDataBuilder.setExecuteTime(executeTime);
            rowDataBuilder.setDatabase(database);
            rowDataBuilder.setTableName(tableName);
            rowDataBuilder.setEventType(eventType.toString().toLowerCase());

            CanalEntry.RowChange rowChange = null;
            try{
                // 获取存储数据，并将二进制字节数据解析转换为RowChange对象
                rowChange = CanalEntry.RowChange.parseFrom(entry.getStoreValue());
            }catch (InvalidProtocolBufferException e){
                // canal中传递的数据是Protocol格式的数据
                // 解析时格式不对 便会抛出异常
                e.printStackTrace();
            }

            CanalEntry.EventType eventType1 = rowChange.getEventType();
//            System.out.println("事件类型1：" + eventType1.toString().toLowerCase());

            // 当事件类型是 DDl语句
            if(eventType1 == CanalEntry.EventType.QUERY || rowChange.getIsDdl()){
                System.out.println("查询语句 ===> " + rowChange.getSql());
            }

            Map<String, String> columnDataMap = new HashMap<String, String>();
            // 当事件类型是 DML语句
            for (CanalEntry.RowData rowData1: rowChange.getRowDatasList()) {
                List<CanalEntry.Column> beforeColumnsList = rowData1.getBeforeColumnsList();
                List<CanalEntry.Column> afterColumnsList = rowData1.getAfterColumnsList();
                // 如果是删除事件 输出删除前的数据
                if(eventType1 == CanalEntry.EventType.DELETE){
                    for (CanalEntry.Column column: beforeColumnsList) {
                        columnDataMap.put(column.getName(), column.getValue().toString());
                    }
                }
                // 如果是插入操作 输出插入的数据
                else if(eventType1 == CanalEntry.EventType.INSERT || eventType1 == CanalEntry.EventType.UPDATE){
                    for (CanalEntry.Column column: afterColumnsList) {
                        columnDataMap.put(column.getName(), column.getValue().toString());
                    }
                }
//                // 如果是更新操作 输出更新前后的数据
//                else if(eventType1 == CanalEntry.EventType.UPDATE){
//                    System.out.println("update前 ===> " + rowData.getBeforeColumnsList());
//                    System.out.println("update后 ===> " + rowData.getAfterColumnsList());
//                }
                rowDataBuilder.putAllColumns(columnDataMap);
            }
        }
        return rowDataBuilder.build().toByteArray();
    }

    public static String dealDatas(List<CanalEntry.Entry> entries){

        Map<String, Object> rowDataMap = new HashMap<>();
        for (CanalEntry.Entry entry: entries) {
            // entry 就是mysql的一条记录

            // 首先判断数据类型 当是binlog日志数据的开始和结尾 便不做任何处理
            // binlog日志数据的格式为： 事务头BEGIN｜数据ROWDATA｜事务尾END
            CanalEntry.EntryType entryType = entry.getEntryType();
            if(entryType == CanalEntry.EntryType.TRANSACTIONBEGIN || entryType == CanalEntry.EntryType.TRANSACTIONEND){
                continue;
            }

            // 获取数据的ROWDATA
            // 获取binlog文件名
            CanalEntry.Header header = entry.getHeader();
            String logfileName = header.getLogfileName();
            // 获取log file的偏移量
            long logfileOffset = header.getLogfileOffset();
            // 获取sql语句执行的时间戳
            long executeTime = header.getExecuteTime();
            // 获取数据库名
            String database = header.getSchemaName();
            // 获取表名
            String tableName = header.getTableName();
            // 获取时间的类型 insert/update/delete
            CanalEntry.EventType eventType = header.getEventType();
//            System.out.println("事件类型：" + eventType.toString().toLowerCase());

            rowDataMap.put("logfileName", logfileName);
            rowDataMap.put("logfileOffset", logfileOffset);
            rowDataMap.put("executeTime", executeTime);
            rowDataMap.put("database", database);
            rowDataMap.put("tableName", tableName);
            rowDataMap.put("eventType", eventType);

            CanalEntry.RowChange rowChange = null;
            try{
                // 获取存储数据，并将二进制字节数据解析转换为RowChange对象
                rowChange = CanalEntry.RowChange.parseFrom(entry.getStoreValue());
            }catch (InvalidProtocolBufferException e){
                // canal中传递的数据是Protocol格式的数据
                // 解析时格式不对 便会抛出异常
                e.printStackTrace();
            }

            CanalEntry.EventType eventType1 = rowChange.getEventType();
//            System.out.println("事件类型1：" + eventType1.toString().toLowerCase());

            // 当事件类型是 DDl语句
            if(eventType1 == CanalEntry.EventType.QUERY || rowChange.getIsDdl()){
                System.out.println("查询语句 ===> " + rowChange.getSql());
            }

            Map<String, Object> columnDataMap = new HashMap<String, Object>();
            // 当事件类型是 DML语句
            for (CanalEntry.RowData rowData: rowChange.getRowDatasList()) {
                List<CanalEntry.Column> beforeColumnsList = rowData.getBeforeColumnsList();
                List<CanalEntry.Column> afterColumnsList = rowData.getAfterColumnsList();
                // 如果是删除事件 输出删除前的数据
                if(eventType1 == CanalEntry.EventType.DELETE){
                    for (CanalEntry.Column column: beforeColumnsList) {
                        columnDataMap.put(column.getName(), column.getValue());
                    }
                }
                // 如果是插入操作 输出插入的数据
                else if(eventType1 == CanalEntry.EventType.INSERT || eventType1 == CanalEntry.EventType.UPDATE){
                    for (CanalEntry.Column column: afterColumnsList) {
                        columnDataMap.put(column.getName(), column.getValue());
                    }
                }
//                // 如果是更新操作 输出更新前后的数据
//                else if(eventType1 == CanalEntry.EventType.UPDATE){
//                    System.out.println("update前 ===> " + rowData.getBeforeColumnsList());
//                    System.out.println("update后 ===> " + rowData.getAfterColumnsList());
//                }
                rowDataMap.put("columns", columnDataMap);
            }
        }
        return JSON.toJSONString(rowDataMap);
    }
}

4. canal clientHA实现

4.1 上述操作搞完后，只需要启动一个client实例即可（无需做其他配置）

码上中年

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
1
评论
Canal之HA高可用

1. canal高可用1.1 canal的HA分为两个部分：clientHA和serverHA1.2 canal的HA需要借助于zookeeper来实现高可用1.3 serverHA：针对每一个实例instance要做到同一时间只能有一个处于running状态，其他处于standby状态1.4 clientHA：同一时间只能有一个client进行接收instance的数据，否则无法保证有序2. canal serverHA实现2.1 修改canal server的配置文件canal/conf/
复制链接

扫一扫