文章目录
前言
随着系统业务增长提高接口响应准备把系统中elasticsearch解耦出来,
Canal模拟MySQL Slave的交互协议,伪装自己为MySQL Slave,向MySQL Master发送dump协议,MySQL Master收到dump请求,开始推送binary log给Slave(即Canal)
Canal解析binary log对象(原始为byte流)对elasticsearch增删改.
一、Canal的工作原理
MySQL主备复制原理:
- MySQL的Master实例将数据变更写入二进制日志(binary log,其中记录叫做二进制日志事件binary log events,可以通过show binlog events进行查看
- MySQL的Slave实例将master的binary log events拷贝到它的中继日志(relay log)
- MySQL的Slave实例重放relay log中的事件,将数据变更反映它到自身的数据
Canal的工作原理如下:
- Canal模拟MySQL Slave的交互协议,伪装自己为MySQL Slave,向MySQL Master发送dump协议
- MySQL Master收到dump请求,开始推送binary log给Slave(即Canal)
- Canal解析binary log对象(原始为byte流),并且可以通过连接器发送到对应的消息队列等中间件中
二、修改mysq配置开启Binlog
- 对于自建 MySQL , 需要先开启 Binlog 写入功能,配置 binlog-format 为 ROW 模式,my.cnf 中配置如下
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复
-
确认是否开启了binlog
-
授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限
-
最后在MySQL的Shell执行下面的命令,新建一个用户名canal密码为QWqw12!@的新用户,赋予REPLICATION SLAVE和 REPLICATION CLIENT权限
CREATE USER canal IDENTIFIED BY 'QWqw12!@';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
ALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
github: canal
三、安装和使用Canal
- 下载解压缩
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz
tar -zxvf canal.deployer-1.1.4.tar.gz
- 目录结构
- 配置修改(vim conf/example/instance.properties)
## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info,需要改成自己的数据库信息
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#username/password,需要改成自己的数据库信息
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName = test (需要在MySQL中建立一个test数据库)
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*
- 启动
sh bin/startup.sh
- 查看 server 日志(vi logs/canal/canal.log)
2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
- 查看 instance 的日志(vi logs/example/example.log)
2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
四、新建Springboot项目整合Canal和Elasticsearch
1.新建项目
- maven
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.69</version>
</dependency>
<!-- canal -->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.4</version>
</dependency>
2.Canal.client连接Canal Server(通过Navicat update)
- 首先启动Canal Server,update修改
- 启动Canal Client后,可以从控制台从看到类似消息
- Canal Client连接代码
package com.alibaba.otter.canal.sample;
import java.net.InetSocketAddress;
import java.util.List;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
public class SimpleCanalClientExample {
public static void main(String args[]) {
// 创建链接
CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostIp(),
11111), "example", "", "");
int batchSize = 1000;
int emptyCount = 0;
try {
connector.connect();
connector.subscribe(".*\\..*");
connector.rollback();
int totalEmptyCount = 120;
while (emptyCount < totalEmptyCount) {
Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
emptyCount++;
System.out.println("empty count : " + emptyCount);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
} else {
emptyCount = 0;
// System.out.printf("message[batchId=%s,size=%s] \n", batchId, size);
printEntry(message.getEntries());
}
connector.ack(batchId); // 提交确认
// connector.rollback(batchId); // 处理失败, 回滚数据
}
System.out.println("empty too many times, exit");
} finally {
connector.disconnect();
}
}
private static void printEntry(List<Entry> entrys) {
for (Entry entry : entrys) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
continue;
}
RowChange rowChage = null;
try {
rowChage = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
e);
}
EventType eventType = rowChage.getEventType();
System.out.println(String.format("================> binlog[%s:%s] , name[%s,%s] , eventType : %s",
entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
eventType));
for (RowData rowData : rowChage.getRowDatasList()) {
if (eventType == EventType.DELETE) {
printColumn(rowData.getBeforeColumnsList());
} else if (eventType == EventType.INSERT) {
printColumn(rowData.getAfterColumnsList());
} else {
System.out.println("-------> before");
printColumn(rowData.getBeforeColumnsList());
System.out.println("-------> after");
printColumn(rowData.getAfterColumnsList());
}
}
}
}
private static void printColumn(List<Column> columns) {
for (Column column : columns) {
System.out.println(column.getName() + " : " + column.getValue() + " update=" + column.getUpdated());
}
}
}
3.通过Canal对Elasticsearch增删改.
- 如果直接biglog对此原始对象进行解析,那么会出现很多解析模板代码.通过注解加,策略模式让接收到biglog表名称直接转换为对应的DTO实例来更新Elasticsearch
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface CanalMapping {
/**
* 数据库
*/
String database();
/**
* 表
*/
String table();
}
- 数据库,表映射
@Data
@Document(indexName = "product") //elasticsearch注解
@CanalMapping(database = "test", table = "product") //CanalBinLog对应的实体类
public class ProductEntity {
@Id
Long id;
/**
* 名称
*/
@Field(type = FieldType.Text, searchAnalyzer = "ik_max_word", analyzer = "ik_max_word")
String name;
/**
* 分类
*/
@Field(type = FieldType.Keyword)
String category;
/**
* 品牌
*/
@Field(type = FieldType.Integer, searchAnalyzer = "ik_max_word", analyzer = "ik_max_word")
String brand;
@Field(type = FieldType.Keyword)
private String type;
/**
* 价格
*/
@Field(type = FieldType.Double)
Double price;
}
- 省略其他代码…
@Service
public class ProductService extends BaseCanalBinlogEventProcessor<ProductEntity> {
@Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
@Override
protected void processInsertInternal(CanalBinLogResult<ProductEntity> result) {
updateOrSaveProductSearch(result.getAfterData());
}
@Override
protected void processUpdateInternal(CanalBinLogResult<ProductEntity> result) {
System.out.println(result.getAfterData().getId());
updateOrSaveProductSearch(result.getAfterData());
}
@Override
protected void processDeleteInternal(CanalBinLogResult<ProductEntity> result) {
deleteProductSearch(result.getBeforeData());
}
@Override
protected void processDDLInternal(CanalBinLogResult<ProductEntity> result) {
elasticsearchRestTemplate.delete(result.getAfterData());
}
public void updateOrSaveProductSearch(ProductEntity productEntity) {
elasticsearchRestTemplate.save(productEntity);
}
public void deleteProductSearch(ProductEntity productEntity) {
System.out.println(productEntity.getId());
elasticsearchRestTemplate.delete(productEntity);
}
}
- 演示
- 通过Navicat操作mysq查询id=1的数据
- 更新id=1的name,观察日志
- 可以看到mysql更新就开始推送binary log给Slave(即Canal)
- 通过elasticsearchRestTemplate查看是否修改成功
总结
把系统中elasticsearch解耦出来,Canal模拟MySQL Slave的交互协议,通过注解加,反射,策略模式让接收到biglog表名称直接转换为对应的DTO实例来更新Elasticsearch(省略反射,策略模式代码,其实实现不难主要是思路).
完
感谢您的阅读
如果你发现了错误的地方,可以在留言区提出来,我对其加以修改。