mysql 订阅组件_canal（mysql数据库 binlog的增量订阅，消费组件）

环境准备

1、jdk 环境变量配置vim /etc/profile

export JAVA_HOME=/root/data/jdk/jdk

export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile

2、jdk配置校验root@never-PAF4:~# date

2018年 03月 11日 星期日 00:00:30 CST

root@never-PAF4:~# java -version

java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

root@never-PAF4:~#

3、canal阿里巴巴mysql数据库 binlog的增量订阅，消费组件

基于日志增量订阅和消费支持的业务

数据库镜像

数据库实时备份

多级索引

search build

业务cache刷新

价格变化等重要业务消息

定位：基于数据库增量日志解析，提供增量数据订阅 消费

工作原理canal 模拟mysql slave的交互协议，伪装自己为myql slave ，向mysql master 发送 dump协议

mysql master收到 dump请求，开始推送binary log 给slave (canal)

canal 解析 binary log 对象(原始为byte流)

下载

配置修改(使用默认配置需要对参数进行本地化，创建对应的测试库)root@never-PAF4:~/data/canal/conf/example# pwd

/root/data/canal/conf/example

root@never-PAF4:~/data/canal/conf/example# vim instance.properties

root@never-PAF4:~/data/canal/conf/example# cat instance.properties

#################################################

## mysql serverId

canal.instance.mysql.slaveId=10

# position info

canal.instance.master.address=127.0.0.1:3302

canal.instance.master.journal.name=

canal.instance.master.position=

canal.instance.master.timestamp=

# table meta tsdb info

canal.instance.tsdb.enable=true

canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}

canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;

#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3302/canal_tsdb

canal.instance.tsdb.dbUsername=root

canal.instance.tsdb.dbPassword=root

#canal.instance.standby.address =

#canal.instance.standby.journal.name =

#canal.instance.standby.position =

#canal.instance.standby.timestamp =

# username/password

canal.instance.dbUsername=root

canal.instance.dbPassword=root

canal.instance.defaultDatabaseName=test

canal.instance.connectionCharset=UTF-8

# table regex

canal.instance.filter.regex=.*\\..*

# table black regex

canal.instance.filter.black.regex=

#################################################

root@never-PAF4:~/data/canal/conf/example#

docker 启动mysql(将本地的配置文件挂载到容器中..)root@never-PAF4:~# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

mysql latest 5d4d51c57ea8 11 days ago 374MB

root@never-PAF4:~# docker run -p 3306:3306 --name slave -e MYSQL_ROOT_PASSWORD=canal -d mysql

47292b980e11e2e4e6bfdee7722474303adc8b6a236a427713d1088550682be8

root@never-PAF4:~# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

47292b980e11 mysql "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:3306->3306/tcp slave

进入镜像，修改配置文件，发现无法使用宿主机器的命令，选择构建新的镜像？还是将宿主机文件挂在容器中root@never-PAF4:~# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

efb8f779ff41 mysql "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 0.0.0.0:3305->3305/tcp, 3306/tcp master

47292b980e11 mysql "docker-entrypoint.s…" 31 minutes ago Up 31 minutes 0.0.0.0:3306->3306/tcp slave

root@never-PAF4:~# docker exec -it master /bin/bash

root@never-PAF4:~/data/mysql/master# docker run -p 3302:3306 --name ms4 -v ~/data/mysql/master/my.conf:/etc/mysql/my.cnf -e MYSQL_ROOT_PASSWORD=root -d mysql

my.cnf

root@never-PAF4:~/data/mysql/master# cat my.conf

[mysqld]

log-bin=mysql-bin

binlog-format=ROW

server_id=1

bind-address=0.0.0.0

root@never-PAF4:~/data/mysql/master#

监听的mysql服务器创建校验root@never-PAF4:~/data/mysql/master# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

a7252065ea95 mysql "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:3302->3306/tcp ms4

47292b980e11 mysql "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:3306->3306/tcp slave

root@never-PAF4:~/data/mysql/master# docker exec -it ms4 mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 4

Server version: 5.7.21-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create user canal identified by 'canal';

Query OK, 0 rows affected (0.06 sec)

mysql> grant select,replication slave,replication client on *.* to 'canal'@'%';

Query OK, 0 rows affected (0.05 sec)

mysql> flush privileges

->

-> ;

Query OK, 0 rows affected (0.05 sec)

mysql>

mysql>

结果，binlog 发生变化，也就是所监听的数据库发生变化 canal 客户端就会监听到Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.

2018-03-11 12:04:46.340 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler

2018-03-11 12:04:46.384 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations

2018-03-11 12:04:46.385 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.

2018-03-11 12:04:46.423 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1:11111]

2018-03-11 12:04:46.838 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]

2018-03-11 12:04:47.040 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set

2018-03-11 12:04:47.321 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

2018-03-11 12:04:48.222 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status

2018-03-11 12:53:51.976 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - dup apply for sql : CREATE TABLE `a` (

`a` varchar(255) NULL

)