mysql加载数据源代码_MySQL数据源接入DBus

最新推荐文章于 2022-05-02 20:03:33 发布
最新推荐文章于 2022-05-02 20:03:33 发布
阅读量311 收藏
点赞数
文章标签： mysql加载数据源代码
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_36143290/article/details/113719104
版权

MySQL数据源接入DBus

作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.cannal概述

1>.cannal简介

canal [kə'næl]，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费

早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从2010年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。

基于日志增量订阅和消费的业务包括

数据库镜像

数据库实时备份

索引构建和实时维护(拆分异构索引、倒排索引等)

业务 cache 刷新

带业务逻辑的增量数据处理

当前的canal支持源端MySQL版本包括5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

重要版本更新说明

canal1.1.x 版本(release_note),性能与功能层面有较大的突破,重要提升包括:

整体性能测试&优化,提升了150%. 原生支持prometheus监控原生支持kafka消息投递原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析)

原生支持docker镜像canal1.1.4版本，迎来最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal动态管理能力，支持配置、任务、日志等在线白屏运维能力。

GitHub地址:

https://github.com/alibaba/canal

2>.MySQL主备复制原理概述

如下图所示:

MySQL master 将数据变更写入二进制日志( binary log, 其中记录叫做二进制日志事件binary log events，可以通过 show binlog events 进行查看)

MySQL slave 将 master 的 binary log events 拷贝到它的中继日志(relay log)

MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据

687474703a2f2f646c2e69746579652e636f6d2f75706c6f61642f6174746163686d656e742f303038302f333038362f34363863316131342d653761642d333239302d396433642d3434616335303161373232372e6a7067

3>.canal 工作原理

canal模拟MySQL slave的交互协议，伪装自己为MySQL slave，向MySQL master发送dump协议

MySQL master收到dump请求，开始推送binary log给MySQL slave(即canal)

canal解析binary log对象(原始为byte流)

博主推荐阅读:

https://github.com/alibaba/canal/blob/master/README.md

bcd33a367d2520c11518283dba352d07.png

二.数据库源端创建心跳库

数据库源端配置只在第一次配置环境时需要，在以后的加表流程中不需要再配置。

此步骤中需要在mysql数据源的mysql_instance实例上创建一个名字为dbus的库，并在此库下创建表db_heartbeat_monitor和db_full_pull_requests两张表，用于心跳检测和全量拉取。

在数据源端新建的dbus库，可以实现无侵入方式接入多种数据源，业务系统无需任何修改，以无侵入性读取数据库系统的日志获得增量数据实时变化。

1>.自行配置MySQL主从复制

博主推荐阅读:

https://www.cnblogs.com/yinzhengjie/p/11816066.html

2>.创建dbus库和dbus用户及相应权限

[root@hdp103.yinzhengjie.org.cn ~]# mysql -uroot -p -S /yinzhengjie/softwares/mysql/data/mysql.sock

Enter password:

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MySQL connectionid is 26Server version:5.7.25-log MySQL Community Server (GPL)

Copyright (c)2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type'help;' or '\h' for help. Type '\c' to clearthe current input statement.

MySQL [(none)]>MySQL [(none)]>CREATE DATABASE dbus;

Query OK,1 row affected (0.00sec)

MySQL [(none)]>MySQL [(none)]> CREATE USER 'dbus'@'%' IDENTIFIED BY 'yinzhengjie';

Query OK,0 rows affected (0.00sec)

MySQL [(none)]>MySQL [(none)]> GRANT ALL ON dbus.* TO dbus@'%' IDENTIFIED BY 'yinzhengjie';

Query OK,0 rows affected, 1 warning (0.01sec)

MySQL [(none)]>MySQL [(none)]>FLUSH PRIVILEGES;

Query OK,0 rows affected (0.00sec)

MySQL [(none)]>MySQL [(none)]>QUIT

Bye

[root@hdp103.yinzhengjie.org.cn~]#

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp101.yinzhengjie.org.cn ~]# mysql -udbus -pyinzhengjie -h hdp103.yinzhengjie.org.cn

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connectionid is 39Server version:5.7.25-log MySQL Community Server (GPL)

Copyright (c)2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type'help;' or '\h' for help. Type '\c' to clearthe current input statement.

mysql>mysql>SHOW DATABASES;+---------------------+

| Database |

+---------------------+

| information_schema |

| dbus |

| yinzhengjie_bigdata |

+---------------------+

3 rows in set (0.01sec)

mysql>mysql>QUIT

Bye

[root@hdp101.yinzhengjie.org.cn~]#

[root@hdp101.yinzhengjie.org.cn~]#

[root@hdp101.yinzhengjie.org.cn ~]# mysql -udbus -pyinzhengjie -h hdp103.yinzhengjie.org.cn

3>.创建dbus库中需要包含的2张表

usedbus;--------------------------------Table structure for db_full_pull_requests------------------------------

DROP TABLE IF EXISTS`db_full_pull_requests`;CREATE TABLE`db_full_pull_requests` (

`seqno`bigint(19) NOT NULLAUTO_INCREMENT,

`schema_name`varchar(64) DEFAULT NULL,

`table_name`varchar(64) NOT NULL,

`physical_tables`varchar(10240) DEFAULT NULL,

`scn_no`int(11) DEFAULT NULL,

`split_col`varchar(50) DEFAULT NULL,

`split_bounding_query`varchar(512) DEFAULT NULL,

`pull_target_cols`varchar(512) DEFAULT NULL,

`pull_req_create_time`timestamp(6) NOT NULL,

`pull_start_time`timestamp(6) NULL DEFAULT NULL,

`pull_end_time`timestamp(6) NULL DEFAULT NULL,

`pull_status`varchar(16) DEFAULT NULL,

`pull_remark`varchar(1024) DEFAULT NULL,PRIMARY KEY(`seqno`)

) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;SET FOREIGN_KEY_CHECKS=0;--------------------------------Table structure for db_heartbeat_monitor------------------------------

DROP TABLE IF EXISTS`db_heartbeat_monitor`;CREATE TABLE`db_heartbeat_monitor` (

`ID`bigint(19) NOT NULL AUTO_INCREMENT COMMENT '心跳表主键',

`DS_NAME`varchar(64) NOT NULL COMMENT '数据源名称',

`SCHEMA_NAME`varchar(64) NOT NULL COMMENT '用户名',

`TABLE_NAME`varchar(64) NOT NULL COMMENT '表名',

`PACKET`varchar(256) NOT NULL COMMENT '数据包',

`CREATE_TIME`datetime NOT NULL COMMENT '创建时间',

`UPDATE_TIME`datetime NOT NULL COMMENT '修改时间',PRIMARY KEY(`ID`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8;

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

MySQL [dbus]>SHOW TABLES;+-----------------------+

| Tables_in_dbus |

+-----------------------+

| db_full_pull_requests |

| db_heartbeat_monitor |

+-----------------------+

2 rows in set (0.00sec)

MySQL[dbus]>MySQL[dbus]> DESCdb_full_pull_requests;+----------------------+----------------+------+-----+----------------------+--------------------------------+

| Field | Type | Null | Key | Default | Extra |

+----------------------+----------------+------+-----+----------------------+--------------------------------+

| seqno | bigint(19) | NO | PRI | NULL | auto_increment |

| schema_name | varchar(64) | YES | | NULL | |

| table_name | varchar(64) | NO | | NULL | |

| physical_tables | varchar(10240) | YES | | NULL | |

| scn_no | int(11) | YES | | NULL | |

| split_col | varchar(50) | YES | | NULL | |

| split_bounding_query | varchar(512) | YES | | NULL | |

| pull_target_cols | varchar(512) | YES | | NULL | |

| pull_req_create_time | timestamp(6) | NO | | CURRENT_TIMESTAMP(6) | on update CURRENT_TIMESTAMP(6) |

| pull_start_time | timestamp(6) | YES | | NULL | |

| pull_end_time | timestamp(6) | YES | | NULL | |

| pull_status | varchar(16) | YES | | NULL | |

| pull_remark | varchar(1024) | YES | | NULL | |

+----------------------+----------------+------+-----+----------------------+--------------------------------+

13 rows in set (0.00sec)

MySQL[dbus]>

MySQL [dbus]> DESC db_full_pull_requests;

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

MySQL [dbus]> DESCdb_heartbeat_monitor;+-------------+--------------+------+-----+---------+----------------+

| Field | Type | Null | Key | Default | Extra |

+-------------+--------------+------+-----+---------+----------------+

| ID | bigint(19) | NO | PRI | NULL | auto_increment |

| DS_NAME | varchar(64) | NO | | NULL | |

| SCHEMA_NAME | varchar(64) | NO | | NULL | |

| TABLE_NAME | varchar(64) | NO | | NULL | |

| PACKET | varchar(256) | NO | | NULL | |

| CREATE_TIME | datetime | NO | | NULL | |

| UPDATE_TIME | datetime | NO | | NULL | |

+-------------+--------------+------+-----+---------+----------------+

7 rows in set (0.00sec)

MySQL[dbus]>

MySQL [dbus]> DESC db_heartbeat_monitor;

三.准备业务库表

1>.创建业务库和表(这一步你可以自行定义测试的库和表)

MySQL [(none)]> CREATE DATABASEyinzhengjie_bigdata;

Query OK,1 row affected (0.00sec)

MySQL[(none)]>MySQL[(none)]> USEyinzhengjie_bigdataDatabasechanged

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> CREATE TABLE student(id int,name varchar(50));

Query OK,0 rows affected (0.06sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]>SHOW TABLES;+-------------------------------+

| Tables_in_yinzhengjie_bigdata |

+-------------------------------+

| student |

+-------------------------------+

1 row in set (0.00sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> DESCstudent;+-------+-------------+------+-----+---------+-------+

| Field | Type | Null | Key | Default | Extra |

+-------+-------------+------+-----+---------+-------+

| id | int(11) | YES | | NULL | |

| name | varchar(50) | YES | | NULL | |

+-------+-------------+------+-----+---------+-------+

2 rows in set (0.00sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> CREATE USER 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie';

Query OK,0 rows affected (0.01sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> GRANT ALL ON yinzhengjie_bigdata.* TO 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie';

Query OK,0 rows affected, 1 warning (0.00sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> FLUSH PRIVILEGES;

Query OK,0 rows affected (0.01sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]>QUIT

Bye[root@hdp103.yinzhengjie.org.cn ~]#[root@hdp103.yinzhengjie.org.cn ~]#

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn ~]# mysql -ujason -pyinzhengjie -h hdp103.yinzhengjie.org.cn

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MySQL connectionid is 38Server version:5.7.25-log MySQL Community Server (GPL)

Copyright (c)2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type'help;' or '\h' for help. Type '\c' to clearthe current input statement.

MySQL [(none)]>MySQL [(none)]>SHOW DATABASES;+---------------------+

| Database |

+---------------------+

| information_schema |

| yinzhengjie_bigdata |

+---------------------+

2 rows in set (0.00sec)

MySQL [(none)]>MySQL [(none)]>QUIT

Bye

[root@hdp103.yinzhengjie.org.cn~]#

[root@hdp103.yinzhengjie.org.cn~]#

[root@hdp103.yinzhengjie.org.cn ~]# mysql -ujason -pyinzhengjie -h hdp103.yinzhengjie.org.cn　　　　　　　　　　#创建一个可以管理业务库的用户

2>.为dbus用户分配咱们上一步骤创建的业务数据库权限

MySQL [yinzhengjie_bigdata]> GRANT SELECT ON yinzhengjie_bigdata.* TO 'dbus'@'%';

Query OK,0 rows affected (0.00sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> FLUSH PRIVILEGES;

Query OK,0 rows affected (0.00sec)

MySQL[yinzhengjie_bigdata]>

四.部署cannal

1>.在数据库中创建canal用户

MySQL [yinzhengjie_bigdata]> CREATE USER 'canal'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie';

Query OK,0 rows affected (0.00sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> GRANT SELECT,REPLICATION SLAVE,REPLICATION CLIENT ON *.* TO 'canal'@'172.200.1.%';

Query OK,0 rows affected (0.01sec)

MySQL[yinzhengjie_bigdata]>MySQL[yinzhengjie_bigdata]> FLUSH PRIVILEGES;

Query OK,0 rows affected (0.00sec)

MySQL[yinzhengjie_bigdata]>

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn ~]# mysql -ucanal -pyinzhengjie -h hdp103.yinzhengjie.org.cn

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MySQL connectionid is 32Server version:5.7.25-log MySQL Community Server (GPL)

Copyright (c)2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type'help;' or '\h' for help. Type '\c' to clearthe current input statement.

MySQL [(none)]>MySQL [(none)]>SHOW DATABASES;+---------------------+

| Database |

+---------------------+

| information_schema |

| dbus |

| dbusmgr |

| mysql |

| performance_schema |

| sys |

| yinzhengjie_bigdata |

+---------------------+

7 rows in set (0.01sec)

MySQL [(none)]>MySQL [(none)]>QUIT

Bye

[root@hdp103.yinzhengjie.org.cn~]#

[root@hdp103.yinzhengjie.org.cn ~]# mysql -ucanal -pyinzhengjie -h hdp103.yinzhengjie.org.cn

2>.cannel自动化部署

在DBus-keeper页面添加数据线时可以选择自动部署cannal，当然我们也可以一键部署，只需要把dbus官方提供的依赖canal的安装包放到指定目录并解压即可，具体操作如下所示。

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# ll

total417572

-rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz

drwxr-xr-x 5 root root 222 Mar 13 22:06dbuskeeper_web-rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# tar -zxf dbus-canal-auto-0.5.0.tar.gz

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# ll

total417572drwxr-xr-x 5 500 500 95 Feb 20 2019 dbus-canal-auto-0.5.0

-rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz

drwxr-xr-x 5 root root 222 Mar 13 22:06dbuskeeper_web-rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0total16

-rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.shdrwxr-xr-x 6 500 500 69 Feb 20 2019canal

drwxr-xr-x 2 500 500 35 Feb 20 2019conf-rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh

-rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.shdrwxr-xr-x 2 500 500 4096 Feb 20 2019lib

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

098514561f79ce168e462960cf6e3e38.png

3>.如上图所示，咱们需要修改conf目录下的canal-auto.properties文件

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0total16

-rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.shdrwxr-xr-x 6 500 500 69 Feb 20 2019canal

drwxr-xr-x 2 500 500 35 Feb 20 2019conf-rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh

-rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.shdrwxr-xr-x 2 500 500 4096 Feb 20 2019lib

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0/conf/total4

-rw-r--r-- 1 500 500 331 Oct 19 2018 canal-auto.properties

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# vim dbus-canal-auto-0.5.0/conf/canal-auto.properties

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]# cat dbus-canal-auto-0.5.0/conf/canal-auto.properties

#数据源名称，需要与dbus keeper中添加的一致

dsname=yinzhengjie_bigdata

#zk地址,替换成自己的信息

zk.path=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181#canal 用户连接地址。即：要canal去同步的源端库的备库的地址

canal.address=hdp103.yinzhengjie.org.cn:3306#canal用户名

canal.user=canal

#canal密码，替换成自己配置的

canal.pwd=yinzhengjie

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus]#

4>.执行deploy.sh脚本

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll

total16

-rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.shdrwxr-xr-x 6 500 500 69 Feb 20 2019canal

drwxr-xr-x 2 500 500 35 Mar 14 10:02conf-rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh

-rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.shdrwxr-xr-x 2 500 500 4096 Feb 20 2019lib

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# cat deploy.sh#!/bin/sh#获得当前shell所在路径

basepath=$(cd `dirname $0`; pwd)

#echo$basepathecho "****************************************** starting *****************************************"#jvm启动参数

#GC_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/data/dbus-canal-auto/logs/gc/gc.log"LOG_CONF="-Dlogs.base.path=$basepath -Duser.dir=$basepath"OOM_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$basepath/logs/oom"JVM_OPTS="-server -Xmx4096m -Xms100m -XX:NewRatio=1"CLASS_PATH=""MAIN="com.creditease.dbus.canal.auto.AutoDeployStart"

if [ "x$1" = "xcheck"]thenMAIN="com.creditease.dbus.canal.auto.AutoCheckStart"

fi#导入jar和config进入classpathfor i in $basepath/lib/*.jar;

do CLASS_PATH=$i:"$CLASS_PATH";

done

export CLASS_PATH=.:$CLASS_PATH

java $JVM_OPTS $LOG_CONF $OOM_OPTS -classpath $CLASS_PATH $MAIN

sleep 1

cd reports

filename=`ls -ltr |grep canal_| tail -n 1 | awk '{print $9}'`

if [ "x$filename" != "x" ]; then

cat $filename

echo "report文件： $filename"

sleep 1

fi

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# cat deploy.sh

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll

total16

-rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.shdrwxr-xr-x 6 500 500 69 Feb 20 2019canal

drwxr-xr-x 2 500 500 35 Mar 14 10:02conf-rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh

-rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.shdrwxr-xr-x 2 500 500 4096 Feb 20 2019lib

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh

****************************************** starting *****************************************

************************************ CANAL CONFIG CHECK BEGIN!*******************************数据源名称: yinzhengjie_bigdata

zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181备库地址: hdp103.yinzhengjie.org.cn:3306canal 用户名: canal

canal 密码: yinzhengjie*************************** CHECK DATABASE CANAL ACCOUNT BEGIN *****************************canal user: canal

canalpwd: yinzhengjie

slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8

数据库连接成功...

检查blog format: show variables like'%bin%'binlog_format : ROW****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS *************************

********************************** CHECK CANAL ZK NODE BEGIN ********************************zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181

2020-03-14 10:38:14,778 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting2020-03-14 10:38:14,808 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18GMT2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_2012020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log

4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=

2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd642020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_642020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0

2020-03-14 10:38:14,825 INFO - [org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c

urator.ConnectionState@25359ed82020-03-14 10:38:14,840 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp102.yinzhengjie.org.cn/172.200.1.102:2181. Will not attempt to authenticate using SASL (unknown error)2020-03-14 10:38:14,900 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp102.yinzhengjie.org.cn/172.200.1.102:2181, initiating session2020-03-14 10:38:14,911 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp102.yinzhengjie.org.cn/172.200.1.102:2181, sessionid = 0x270d619e962000a, negotiated timeout = 60000

2020-03-14 10:38:14,915 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED

node exit ,skip zk node:/DBus

node exit ,skip zk node:/DBus/Canal2020-03-14 10:38:14,977 INFO - [com.creditease.dbus.commons.ZkService.createNode(ZkService.java:159)] 节点创建成功, Path: /DBus/Canal/canal-yinzhengjie_bigdata

create zk node:/DBus/Canal/canal-yinzhengjie_bigdata

node exit ,skip zk node:/DBus/Canal/canal-yinzhengjie_bigdata******************************** CHECK CANAL ZK NODE SUCCESS ********************************

2020-03-14 10:38:15,007 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x270d619e962000aclosed************************************ CANAL CONFIG CHECK SUCCESS!*****************************

**************************************** CANAL DEPLOY BEGIN!*********************************

**************************************** COPY CANAL BEGIN ***********************************copy canal files:cp -r canal/. canal-yinzhengjie_bigdata2020-03-14 10:38:15,013 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x270d619e962000acopy instance files:cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata**************************************** COPY CANAL SUCCESS *********************************

************************************ EDIT CANAL.PROPERTIES BEGIN ****************************props: canal.port= 10000props: canal.zkServers= hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata

props: canal.destinations=yinzhengjie_bigdata

props: canal.auto.scan= falseprops: canal.instance.filter.query.dcl= trueprops: canal.instance.filter.query.dml= trueprops: canal.instance.binlog.format=ROW

props: canal.instance.binlog.image=FULL

props: #canal.instance.global.spring.xml= classpath:spring/file-instance.xml

props: canal.instance.global.spring.xml= classpath:spring/default-instance.xml********************************** EDIT CANAL.PROPERTIES SUCCESS ****************************

****************************** UPDATE INSTANCE.PROPERTIES BEGIN *****************************instancefile path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties

props: canal.instance.master.address= hdp103.yinzhengjie.org.cn:3306props: canal.instance.dbUsername=canal

props: canal.instance.dbPassword=yinzhengjie

props: canal.instance.connectionCharset= UTF-8

***************************** UPDATE INSTANCE.PROPERTIES SUCCESS ****************************starting canal.....

exec:sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.shexec:sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.shexec:rm -f canal.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log

exec:rm -f yinzhengjie_bigdata.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log

exec:ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}'canal 进程启动成功， pid80463请手动检查当前目录下canal.log，和yinzhengjie_bigdata.log中有无错误信息。********************************* CANAL DEPLOY SCCESS! **************************************

************************************ CANAL CONFIG CHECK BEGIN!*******************************数据源名称: yinzhengjie_bigdata

zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181备库地址: hdp103.yinzhengjie.org.cn:3306canal 用户名: canal

canal 密码: yinzhengjie*************************** CHECK DATABASE CANAL ACCOUNT BEGIN *****************************canal user: canal

canalpwd: yinzhengjie

slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8

数据库连接成功...

检查blog format: show variables like'%bin%'binlog_format : ROW****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS *************************

********************************** CHECK CANAL ZK NODE BEGIN ********************************zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181node exit ,skip zk node:/DBus

node exit ,skip zk node:/DBus/Canal

create zk node:/DBus/Canal/canal-yinzhengjie_bigdata

node exit ,skip zk node:/DBus/Canal/canal-yinzhengjie_bigdata******************************** CHECK CANAL ZK NODE SUCCESS ********************************

************************************ CANAL CONFIG CHECK SUCCESS!*****************************

**************************************** CANAL DEPLOY BEGIN!*********************************

**************************************** COPY CANAL BEGIN ***********************************copy canal files:cp -r canal/. canal-yinzhengjie_bigdata

copy instance files:cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata**************************************** COPY CANAL SUCCESS *********************************

************************************ EDIT CANAL.PROPERTIES BEGIN ****************************props: canal.port= 10000props: canal.zkServers= hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata

props: canal.destinations=yinzhengjie_bigdata

props: canal.auto.scan= falseprops: canal.instance.filter.query.dcl= trueprops: canal.instance.filter.query.dml= trueprops: canal.instance.binlog.format=ROW

props: canal.instance.binlog.image=FULL

props: #canal.instance.global.spring.xml= classpath:spring/file-instance.xml

props: canal.instance.global.spring.xml= classpath:spring/default-instance.xml********************************** EDIT CANAL.PROPERTIES SUCCESS ****************************

****************************** UPDATE INSTANCE.PROPERTIES BEGIN *****************************instancefile path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties

props: canal.instance.master.address= hdp103.yinzhengjie.org.cn:3306props: canal.instance.dbUsername=canal

props: canal.instance.dbPassword=yinzhengjie

props: canal.instance.connectionCharset= UTF-8

***************************** UPDATE INSTANCE.PROPERTIES SUCCESS ****************************starting canal.....

exec:sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.shexec:sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.shexec:rm -f canal.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log

exec:rm -f yinzhengjie_bigdata.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log

exec:ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}'canal 进程启动成功， pid80463请手动检查当前目录下canal.log，和yinzhengjie_bigdata.log中有无错误信息。********************************* CANAL DEPLOY SCCESS! **************************************report文件： canal_deploy_yinzhengjie_bigdata_20200314103813.txt

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh

a4d262cba12d65e33210ef7ea45e41eb.png

5>.检查部署是否正常

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.shcheck****************************************** starting *****************************************

************************************* CANAL CHECK BEGIN! ************************************

*************************** CHECK DATABASE CANAL ACCOUNT BEGIN *****************************canal user: canal

canalpwd: yinzhengjie

slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8

数据库连接成功...

检查blog format: show variables like'%bin%'binlog_format : ROW****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS *************************

********************************** CHECK CANAL ZK NODE BEGIN ********************************zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181

2020-03-14 10:41:30,246 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting2020-03-14 10:41:30,261 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18GMT2020-03-14 10:41:30,263 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_2012020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log

4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=

2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd642020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_642020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0

2020-03-14 10:41:30,270 INFO - [org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c

urator.ConnectionState@25359ed82020-03-14 10:41:30,292 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp101.yinzhengjie.org.cn/172.200.1.101:2181. Will not attempt to authenticate using SASL (unknown error)2020-03-14 10:41:30,373 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp101.yinzhengjie.org.cn/172.200.1.101:2181, initiating session2020-03-14 10:41:30,385 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp101.yinzhengjie.org.cn/172.200.1.101:2181, sessionid = 0x170d61a5413000e, negotiated timeout = 60000

2020-03-14 10:41:30,393 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED

node exit ,skip zk node:/DBus

node exit ,skip zk node:/DBus/Canal

node exit ,skip zk node:/DBus/Canal/canal-yinzhengjie_bigdata******************************** CHECK CANAL ZK NODE SUCCESS ********************************

2020-03-14 10:41:30,452 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x170d61a5413000eclosed2020-03-14 10:41:30,453 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x170d61a5413000eexec:rm -f canal.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log

exec:rm -f yinzhengjie_bigdata.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log**************************************** CANAL CHECK SUCCESS ********************************

************************************* CANAL CHECK BEGIN! ************************************

*************************** CHECK DATABASE CANAL ACCOUNT BEGIN *****************************canal user: canal

canalpwd: yinzhengjie

slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8

数据库连接成功...

检查blog format: show variables like'%bin%'binlog_format : ROW****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS *************************

********************************** CHECK CANAL ZK NODE BEGIN ********************************zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181node exit ,skip zk node:/DBus

node exit ,skip zk node:/DBus/Canal

node exit ,skip zk node:/DBus/Canal/canal-yinzhengjie_bigdata******************************** CHECK CANAL ZK NODE SUCCESS ********************************exec:rm -f canal.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log

exec:rm -f yinzhengjie_bigdata.log

exec:ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log**************************************** CANAL CHECK SUCCESS ********************************report文件： canal_check_yinzhengjie_bigdata_20200314104129.txt

[root@hdp103.yinzhengjie.org.cn/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#

[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh check

f373acfa0f43d2b1de3bef7e4c9d0c3d.png

6>.管理canal服务

8f900a89c6347c561fdf2122f13be562.png

961ddebeb323a10fe0623af514929fc1.png

[root@hdp103.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/total16

-rw-r--r-- 1 root root 5 Mar 14 16:11canal.pid-rwxr-xr-x 1 root root 1145 Mar 14 10:38startup.bat-rwxr-xr-x 1 root root 2956 Mar 14 10:38 startup.sh

-rwxr-xr-x 1 root root 1356 Mar 14 10:38 stop.sh[root@hdp103.yinzhengjie.org.cn~]#

[root@hdp103.yinzhengjie.org.cn~]#

[root@hdp103.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/　　　　　　　　　　#存放管理canal服务的脚本

732736a6145e12a4baf7adb1bd6dff65.png

五.DBus一键加线

1>.数据线的概念

所谓数据线是一种形象的说法，DBus要想采集某个数据库(数据源)的数据，就得拉一根线连接数据库和DBus，加数据线就是DBus-Keeper页面配置一条数据线，即指定从哪个数据源的哪个库采集哪些表的数据。

Dbus对每个DataSource数据源配置一条数据线，当要添加新的datasource时，需要新添加一条数据线。下面对通过dbus keeper页面添加新数据线的步骤进行介绍

2>.删除自动部署canal的配置

我们采用手工方式部署canal，所以把相关配置删掉，避免加数据线时出现问题，具体操作如下图所示。

依次点击"配置中心","zk管理","DBus","Commons","auto-deploy-canal.conf"(需要鼠标右击),"删除节点"

23e3024529831969298af76f1ff54080.png

删除成功后，会有以下的提示信息，建议大家都删除掉哟，否则你尽管下面再不勾选自动部署canal依旧会部署失败，这算是官方的一个bug啦~

b7e79e77ec8ba93a9ec7bcd298451929.png

3>.管理员身份进入dbus keeper页面，并依次点击"数据源管理","新建数据线"

344b084ad3989f010fa39cdfea9fb575.png

4>.填写数据源基本信息 (master和slave jdbc连接串信息)

jdbc:MySQL://hdp103.yinzhengjie.org.cn:3306/dbus?useUnicode=true&characterEncoding=utf8&autoReconnect=true&failOverReadOnly=false&noAccessToProcedureBodies=true&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false

06c2fc19025baea948e56f62eb57392f.png

5>.下一步添加的schema，勾选要添加的表。Keeper支持一次添加多个schema下的多个table

f296afddcc541a9b8792fcaee5915621.png

6>.启动Topology

9bce2dd75b5cbbdd6259b6fa92f21955.png

我们从上往下依次点击启动按钮，点击按钮后当前页面就不可用被选择啦，如下图所示。

be049e715b8bb1e4f74b54b5bd279c95.png

第一个dsName启动成功后，它的状态会变成running状态哟~如下图所示。

c4158d367fc36101bd8fa88a9221ee71.png

启动成功后，我们通过Ambari的storm服务菜单可以找到打开storm WebUI的界面，可以看到的确是有Topology启动。

e9afd3079e7536c51b3b70d959fad648.png

接下来的工作就简单了，按照上述的操作，依次启动三个Topology，启动成功后，状态均变为running状态，如下图所示。

a9b1bd53baf2c80da96d47f9a08179ce.png

再次查看Storm WebUI，可以看到相应的状态信息。你可以点击每一个任务查看详细哟~

d9eb0638c5195c62dabeabc90ca06b24.png

7>.上一步执行成功后，再次回到"数据源管理"，我们看到数据线就加好了，如下图所示。

7959f95119bc000ec6647a061803d16a.png

活萝卜
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
mysql增量采集_010.使用DBus贴源采集MySQL增量bin-log日志
weixin_28864057的博客
02-08 186
1.数据采集流程2. 环境准备2.1 MySQL主从数据库环境准备组件hdp01(主机)hdp02(主机)MySQL数据库(主)√MySQL数据库(从)√Canal Server√说明：DBus-0.6.1使用Canal-v1.1.4，支持MySQL5.6和5.7被同步的MySQL bin-log需要是row模式考虑到Kafka的message大小不宜太大，目前设置的是最大10MB，因此不支持同步...
MySQL缓冲池源码分析之如何加载数据页
授人以鱼不如授之以渔
07-23 599
最近又开始看了mysql源码，顺便在这里总结一下自己的理解。 为了简单理解，我就简单描述了一下流程。 流程如下： 1.是否命中缓冲池---->命中直接访问（为了避免查询数据页时扫描LRU，还为每个buffer pool instance维护了一个page hash，通过space id和page no查找对应的page。一般情况下，当我们需要读入一个Page时，首先根据space id和page no找到所属的buffer pool instance。然后根据page hash查询，如果page ha
数据库源代码
11-19
数据库源代码'<br>看看吧<br>请多指教<br>哈哈哈哈哈哈
学习数据库的好资料
欢迎来到Sylinx_yqg的博客
04-19 727
/*************************************** 作者：颜清国 学习数据库的资料 2007.4.19*****************************************//*****************************************COURSE表：CNO    CNAME                CPNO   C
数据源（Data Source
weixin_30458043的博客
02-02 1032
数据源（Data Source）顾名思义，数据的来源，是提供某种所需要数据的器件或原始媒体。在数据源中存储了所有建立数据库连接的信息。就像通过指定文件名称可以在文件系统中找到文件一样，通过提供正确的数据源名称，你可以找到相应的数据库连接。中文名数据源外文名Data Source目录1分类2属性3名称4数据包? DBCP? C3P0? ODBC1分类编辑 信息系统的数据源必需可靠且具备更新能力，常...
dbus-cxx-0.7.0.zip_DBUS_DBUS 封装_dbus c++
09-19
DBus是Linux系统中的一种消息总线服务，它允许不同的应用程序之间进行进程间通信（IPC）。DBus-C++是DBus的一个C++库，它...通过深入研究源代码、示例和文档，你可以掌握如何在C++环境中高效地利用DBus进行进程间通信。
dbus-server-socket.rar_DBUS_server dbus
09-20
"dbus-server-socket.c"和"dbus-server-socket.h"这两个文件很可能是DBus Server实现的关键源代码文件。 `dbus-server-socket.c`文件很可能是实现代码，它包含了具体的函数定义和实现，如： 1. **初始化和绑定套接...
Qt例程源代码QtRemoteObjects.7z
07-19
Qt例程源代码QtRemoteObjects.7z
dbus-test-main.rar_DBUS_run
09-21
在压缩包中的 "dbus-test-main.c" 文件，我们可以推断这是一份用 C 语言编写的源代码，它是测试框架的核心部分，负责初始化 DBus 连接，加载和执行测试用例。可能包含的功能有： 1. **DBus 连接设置**：文件可能会...
dbus-mainloop.rar_DBUS
09-22
- `dbus_connection_add_filter()`：添加一个过滤器函数，用于在消息传递前对其进行处理。 - `dbus_connection_unref()`：释放 DBUS 连接资源。 在实际应用中，开发者通常会在主线程中运行 DBUS 主循环，同时在其他...
bbs 数据库源代码
04-15
use master go if exists(select*from sysdatabases where name='bbsDB') drop database bbsDB create database bbsDB on primary ( name='bbsDB_data', filename='D:\project\bbsDB_data.mdf', size=10mb ) log on ( name='bbsDB_log', filename='D:\project\bbsDB_log.ldf', size=10mb ) go use bbsDB go create table bank ( customerName char(8) not null,--顾客姓名 cardID char(10) not null,--卡号 currentMoney Money not null,--当前余额 ) go create table transInfo ( cardID char(10) not null,--卡号 transType char(4) not null,--交易类型 transMoney money not null,--交易金额 transDate datetime not null,--交易时间 ) go alter table bank add constraint ck_currentMoney check(currentMoney>=1) alter table transInfo add constraint df_transDate default(getDate())for transDate,constraint ck_transType check(transType in('存入','支取')) go insert into bank(customerName,cardID,currentMoney)values('张三','100010001',2000) --insert into bank(customerName,cardID,currentMoney)values('李四','100010002',1) print '------取款前的余额------' select*from bank go begin transaction declare @errorSum int declare @myMoney Money set @myMoney=1000 --取款金额 set @errorSum=0 --取款 insert into transInfo(cardID,transType,transMoney)values('100010001','支取', @myMoney) set @errorSum=@errorSum+@@error update bank set currentMoney=currentMoney-@myMoney where cardID='100010001' set @errorSum=@errorSum+@@error print'------取款事务过程中余额和交易信息------' select*from bank select*from transInfo if @errorSum<>0 begin print'交易失败，回滚事务' rollback transaction end else begin print '交易成功，提交事务' commit transaction end go print'-------取款事务结束后的余额和交易信息------' select*from bank select*from transInfo go go begin transaction declare @errorSum int declare @myMoney Money set @myMoney=5000 --存入金额 set @errorSum=0 --存入 insert into transInfo(cardID,transType,transMoney)values('100010001','存入',@myMoney) set @errorSum=@errorSum+@@error update bank set currentMoney=currentMoney+@myMoney where cardID='100010001' set @errorSum=@errorSum+@@error print'------存款事务过程中余额和交易信息------' select*from bank select*from transInfo if @errorSum<>0 begin print'交易失败，回滚事务' rollback transaction end else begin print '交易成功，提交事务' commit transaction end go print'-------存款事务结束后的余额和交易信息------' select*from bank select*from transInfo go
深入理解大数据源码
01-17
里面包含许多经典例子的代码，java实现，易读性强，包含MapReduce的K-Means聚类算法、KNN最近邻分类算法、朴素贝叶斯分类算法、决策树分类算法等，还有基于MapReduce的搜索引擎算法、大规模基因序列化比对算法、重复文档检测算法等
电影网站及数据库源代码
12-14
这是一个电影网站源代码，使用的是PHP语言，数据库为mysql，你可以轻松地假设自己的电影网站了
数据库各种操作源码.zip
06-25
数据库各种操作源码很多种都是有源码的。就是注释有点少吧。。当作学习自己补充注释
Redis源码学习（19），数据类型源代码，学习总结
无痕之意的博客
05-02 1582
  经过将近半个月的学习，终于将五种数据类型的源代码都学习了一遍，虽然不是全部阅读，但是大部分的代码都已经学习到了，趁五一假期好好整理和总结一下近期我们学习的内容。 1 数据类型介绍   在Redis中有五种数据类型，分别是字符串、列表、集合、有序集合、哈希，在源代码 redis.h 头文件中，有对应他们的常量定义，每次判断对象类型的时候会使用到这几个常量。 /* Object types */ #define REDIS_STRING 0 #define REDIS_LIST 1 #define REDI
#DBus# 关系型数据库全表扫描分片详解
敏捷大数据的CSDN博客
07-06 818
作者：尹红春 导读：数据总线（DBus）专注于数据的实时采集与实时分发，可以对IT系统在业务流程中产生的数据进行汇聚，经过转换处理后成为统一JSON的数据格式（UMS），提供给不同数据使用方订阅和消费，充当数仓平台、大数据分析平台、实时报表和实时营销等业务的数据源。在本文中，我们从数据分片的角度出发，具体介绍DBus在数据采集的过程中，运用了什么样的分片策略和分片原理，以及过程中遇到的问题及解决...
mysql全量到kafka_DBus全量拉取MySQL到kafka的过程中，出现类型不匹配的错误。
weixin_30089411的博客
02-08 189
MySQL数据类型：mysql> desc actor;+-------------+----------------------+------+-----+-------------------+-----------------------------+| Field | Type | Null | Key | Default ...
EeasyBI报表系统 数据源选择代码开发手册
通达OA二次开发 联系QQ2524837118
06-05 710
数据源——选择代码开发手册 一、数据源代码路径 数 据 源 代 码 以 文 件 夹 的 形 式 放 在 “ MYOA根 目 录 /webroot/inc/db2.0/data_source”下，即插即用，无需做任何数据库方面的修改。 如图所示，first_demo 即为一个数据源代码，路径为 “D:/MYOA/webroot/inc/db2.0/data_source”。 二、数据源代码包含的文件 1、数据源代码主体文件...
java HttpClient 工具类
性感的杏鲍菇的专栏
09-06 497
package cn.creditease.xuanyuan.server.utils;import java.io.IOException; import java.lang.reflect.Field; import java.lang.reflect.Method; import java.util.HashMap; import java.util.Map;import org.ap
g_dbus_is_member_name g_dbus_is_interface_name
最新发布
07-22
`g_dbus_is_member_name` 和 `g_dbus_is_interface_name` 是 GLib 库中的两个函数，用于检查给定的字符串是否符合 D-Bus 成员名称或接口名称的规范。 `g_dbus_is_member_name` 函数用于检查字符串是否符合 D-Bus ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值