【Canal】数据同步的终极解决方案，直击优秀开源框架灵魂

================> binlog[mysql-bin.000007:6594] , executeTime : 1540286902000(2020-08-05 23:26:22) , gtid : () , delay : 712ms

BEGIN ----> Thread id: 43

----------------> binlog[mysql-bin.000007:6725] , name[canal,canal_table] , eventType : UPDATE , executeTime : 1540286902000(2020-08-05 23:26:22) , gtid : () , delay : 712 ms

id : 21 type=int(10) unsigned

name : aaac type=varchar(255) update=true

---

END ----> transaction id: 252

================> binlog[mysql-bin.000007:6782] , executeTime : 1540286902000(2020-08-05 23:26:22) , gtid : () , delay : 713ms

数据同步实现

---

需求

将数据库数据的变化, 通过canal解析binlog日志, 实时更新到solr的索引库中。

具体实现

创建工程

创建Maven工程mykit-canal-demo，并在pom.xml文件中添加如下配置。

com.alibaba.otter

canal.client

1.0.24

com.alibaba.otter

canal.protocol

1.0.24

commons-lang

commons-lang

2.6

org.codehaus.jackson

jackson-mapper-asl

1.8.9

org.apache.solr

solr-solrj

4.10.3

junit

junit

4.9

test

创建log4j配置文件

在工程的src/main/resources目录下创建log4j.properties文件，内容如下所示。

log4j.rootCategory=debug, CONSOLE

CONSOLE is set to be a ConsoleAppender using a PatternLayout.

log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender

log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout

log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %-6r [%15.15t] %-5p %30.30c %x - %m\n

LOGFILE is set to be a File appender using a PatternLayout.

log4j.appender.LOGFILE=org.apache.log4j.FileAppender

log4j.appender.LOGFILE.File=d:\axis.log

log4j.appender.LOGFILE.Append=true

log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout

log4j.appender.LOGFILE.layout.ConversionPattern=%d{ISO8601} %-6r [%15.15t] %-5p %30.30c %x - %m\n

创建实体类

在io.mykit.canal.demo.bean包下创建一个Book实体类，用于测试Canal的数据传输，如下所示。

package io.mykit.canal.demo.bean;

import org.apache.solr.client.solrj.beans.Field;

import java.util.Date;

public class Book implements Serializable {

private static final long serialVersionUID = -6350345408771427834L;{

@Field(“id”)

private Integer id;

@Field(“book_name”)

private String name;

@Field(“book_author”)

private String author;

@Field(“book_publishtime”)

private Date publishtime;

@Field(“book_price”)

private Double price;

@Field(“book_publishgroup”)

private String publishgroup;

public Integer getId() {

return id;

}

public void setId(Integer id) {

this.id = id;

}

public String getName() {

return name;

}

public void setName(String name) {

this.name = name;

}

public String getAuthor() {

return author;

}

public void setAuthor(String author) {

this.author = author;

}

public Date getPublishtime() {

return publishtime;

}

public void setPublishtime(Date publishtime) {

this.publishtime = publishtime;

}

public Double getPrice() {

return price;

}

public void setPrice(Double price) {

this.price = price;

}

public String getPublishgroup() {

return publishgroup;

}

public void setPublishgroup(String publishgroup) {

this.publishgroup = publishgroup;

}

@Override

public String toString() {

return “Book{” +

“id=” + id +

“, name='” + name + ‘’’ +

“, author='” + author + ‘’’ +

“, publishtime=” + publishtime +

“, price=” + price +

“, publishgroup='” + publishgroup + ‘’’ +

‘}’;

}

}

其中，我们在Book实体类中，使用Solr的注解@Field定义了实体类字段与Solr域之间的关系。

各种工具类的实现

接下来，我们就在io.mykit.canal.demo.utils包下创建各种工具类。

• BinlogValue

用于存储binlog分析的每行每列的value值，代码如下所示。

package io.mykit.c