canal部署和使用

最新推荐文章于 2023-03-10 01:17:03 发布

Childs丶

最新推荐文章于 2023-03-10 01:17:03 发布

阅读量715

点赞数

分类专栏：组件文章标签： canal

本文链接：https://blog.csdn.net/qq_16248977/article/details/100887556

版权

组件专栏收录该内容

5 篇文章 0 订阅

订阅专栏

下载安装包

release压缩包下载
地址
示例:
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz

修改配置

解压并修改配置
conf/canal.properties可不修改,直接使用默认配置;
conf/example/instance.properties(修改中文注释的地方)

#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=0

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=127.0.0.1:3306  #改成自己的数据库地址
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=false  #不使用可关闭
canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
canal.instance.tsdb.dbUsername=canal
canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=root  #改成自己的数据库用户
canal.instance.dbPassword=123456  #改成自己的数据库密码
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*
#################################################

修改mysql的配置my.cnf,并重启mysql服务
[mysqld]中添加一下内容

[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复

启动并观察日志

执行bin目录的startup.sh
./bin/startup.sh
观察日志
logs/canal/canal.log

2019-09-16 15:08:43.964 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2019-09-16 15:08:43.993 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2019-09-16 15:08:44.002 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2019-09-16 15:08:44.029 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.31.203.20(172.31.203.20):11111]
2019-09-16 15:08:44.580 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......

logs/example/examp.log

2019-09-16 15:08:44.296 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2019-09-16 15:08:44.300 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2019-09-16 15:08:44.431 [main] WARN  o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]
2019-09-16 15:08:44.439 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2019-09-16 15:08:44.446 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2019-09-16 15:08:44.446 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
2019-09-16 15:08:44.503 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2019-09-16 15:08:44.547 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2019-09-16 15:08:44.547 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2019-09-16 15:08:44.547 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2019-09-16 15:08:44.563 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
 {"identity":{"slaveId":-1,"sourceAddress":{"address":"localhost","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mysql-bin.000001","position":2772635,"serverId":1,"timestamp":1568614406000}}
2019-09-16 15:08:44.573 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000001,position=2772635,serverId=1,gtid=,timestamp=1568614406000] cost : 60ms , the next step is binlog dump

出现上述内容说明启动成功.

客户端(Java)

客户端代码

import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import java.net.InetSocketAddress;
import java.util.List;

public class CanalClientTest {

    public static void main(String args[]) {
        // 创建链接
        CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostName(),
                11111), "example", "", "");
        int batchSize = 1000;
        int emptyCount = 0;
        try {
            connector.connect();
            connector.subscribe(".*\\..*");
            connector.rollback();
            int totalEmptyCount = 120;
            while (emptyCount < totalEmptyCount) {
                Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据
                long batchId = message.getId();
                int size = message.getEntries().size();
                if (batchId == -1 || size == 0) {
                    emptyCount++;
                    System.out.println("empty count : " + emptyCount);
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                    }
                } else {
                    emptyCount = 0;
                    // System.out.printf("message[batchId=%s,size=%s] \n", batchId, size);
                    printEntry(message.getEntries());
                }

                connector.ack(batchId); // 提交确认
                // connector.rollback(batchId); // 处理失败, 回滚数据
            }

            System.out.println("empty too many times, exit");
        } finally {
            connector.disconnect();
        }
    }

    private static void printEntry(List<CanalEntry.Entry> entrys) {
        for (CanalEntry.Entry entry : entrys) {
            if (entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONBEGIN || entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONEND) {
                continue;
            }

            CanalEntry.RowChange rowChage = null;
            try {
                rowChage = CanalEntry.RowChange.parseFrom(entry.getStoreValue());
            } catch (Exception e) {
                throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
                        e);
            }

            CanalEntry.EventType eventType = rowChage.getEventType();
            System.out.println(String.format("================&gt; binlog[%s:%s] , name[%s,%s] , eventType : %s",
                    entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
                    entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
                    eventType));

            for (CanalEntry.RowData rowData : rowChage.getRowDatasList()) {
                if (eventType == CanalEntry.EventType.DELETE) {
                    printColumn(rowData.getBeforeColumnsList());
                } else if (eventType == CanalEntry.EventType.INSERT) {
                    printColumn(rowData.getAfterColumnsList());
                } else {
                    System.out.println("-------&gt; before");
                    printColumn(rowData.getBeforeColumnsList());
                    System.out.println("-------&gt; after");
                    printColumn(rowData.getAfterColumnsList());
                }
            }
        }
    }

    private static void printColumn(List<CanalEntry.Column> columns) {
        for (CanalEntry.Column column : columns) {
            System.out.println(column.getName() + " : " + column.getValue() + "    update=" + column.getUpdated());
        }
    }
}

客户端日志
mysql无数据变动时

empty count : 0
empty count : 1
empty count : 2

mysql有数据变动时

================&gt; binlog[mysql-bin.000001:3659803] , name[uoamp,uoamp_server_node_index] , eventType : DELETE
node_id : a4bb89a9efd5425b97ffc487ca6137ba    update=false
node_name : node-01    update=false
node_index : 0,70    update=false
create_time : 2019-09-16 15:12:05    update=false
modify_time : 2019-09-16 15:12:05    update=false
remark :     update=false
================&gt; binlog[mysql-bin.000001:3660081] , name[uoamp,uoamp_server_node_index] , eventType : INSERT
node_id : a4bb89a9efd5425b97ffc487ca6137ba    update=true
node_name : node-01    update=true
node_index : 0,70    update=true
create_time : 2019-09-16 15:14:54    update=true
modify_time : 2019-09-16 15:14:54    update=true
remark :     update=true
================&gt; binlog[mysql-bin.000001:3659441] , name[uoamp_db,uoamp_server_node] , eventType : UPDATE
-------&gt; before
node_id : 074f3bc7978b4eec8f8907f04546b943    update=false
node_name : node-01    update=false
node_ip : 172.31.203.20    update=false
node_state : 1    update=false
node_weight : 5    update=false
create_time : 2019-02-13 15:41:20    update=false
modify_time : 2019-09-16 15:14:14    update=false
remark :     update=false
-------&gt; after
node_id : 074f3bc7978b4eec8f8907f04546b943    update=false
node_name : node-01    update=false
node_ip : 172.31.203.20    update=false
node_state : 1    update=false
node_weight : 5    update=false
create_time : 2019-02-13 15:41:20    update=false
modify_time : 2019-09-16 15:14:17    update=true
remark :     update=false

Childs丶

关注

0
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
canal部署和使用

下载安装包release压缩包下载地址: https://github.com/alibaba/canal/releases示例:wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz修改配置解压并修改配置conf/canal.propert...
复制链接

扫一扫

专栏目录