Canal部署

安装MySQL

[hadoop@hadoop003 canal-admin]$ sudo yum -y localinstall https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

[hadoop@hadoop003 canal-admin]$ sudo yum -y install mysql-community-server

[hadoop@hadoop003 canal-admin]$ sudo systemctl enable mysqld

[hadoop@hadoop003 canal-admin]$ sudo systemctl start mysqld

[hadoop@hadoop003 canal-admin]$ sudo systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-01-30 16:13:12 CST; 1s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 8940 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 9059 (mysqld)
   Status: "Server is operational"
   CGroup: /system.slice/mysqld.service
           └─9059 /usr/sbin/mysqld

Jan 30 16:13:05 hadoop003 systemd[1]: Starting MySQL Server...
Jan 30 16:13:12 hadoop003 systemd[1]: Started MySQL Server.

[hadoop@hadoop003 canal-admin]$ sudo grep 'temporary password' /var/log/mysqld.log

[hadoop@hadoop003 canal-admin]$ mysql -uroot -p

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'Root_12root';
Query OK, 0 rows affected (0.02 sec)

mysql> SHOW VARIABLES LIKE 'validate_password%';
+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password.check_user_name    | ON     |
| validate_password.dictionary_file    |        |
| validate_password.length             | 8      |
| validate_password.mixed_case_count   | 1      |
| validate_password.number_count       | 1      |
| validate_password.policy             | MEDIUM |
| validate_password.special_char_count | 1      |
+--------------------------------------+--------+
7 rows in set (0.01 sec)

mysql> set global validate_password.policy=0;
Query OK, 0 rows affected (0.00 sec)

mysql> set global validate_password.length=1;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> CREATE USER canal IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
Query OK, 0 rows affected (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER canaladmin IDENTIFIED BY 'canaladmin%123';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT ALL ON canal_manager.* TO 'canaladmin'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

 生产上Canal-Admin自己用的MySQL和你要增量采集binlog的MySQL大概率不是同一个，甚至在不 同的机器上。为了简单，我们这里采用同一个MySQL，生产上自己要注意。

重启数据库

[hadoop@hadoop003 canal-admin]$ sudo vi /etc/my.cnf

server-id=3
log-bin=mysql-bin
binlog-format=ROW
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=performance_schema
binlog-ignore-db=sys

[hadoop@hadoop003 canal-admin]$ sudo systemctl stop mysqld

[hadoop@hadoop003 canal-admin]$ sudo systemctl start mysqld

如上图，binlog启动成功

注意:针对阿里云 RDS for MySQL , 默认打开了 binlog , 并且账号默认具有 binlog dump 权限 , 不需要任何权限或者 binlog 设置,可以直接跳过这一步

禁用explicit_defaults_for_timestamp

SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%';
set persist explicit_defaults_for_timestamp=0;
SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%'；

重启MySQL

安装MySQL的Java驱动

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.18/mysql-connector-java-8.0.18.jar

sudo mkdir -p /usr/share/java

sudo mv ~/mysql-connector-java-8.0.18.jar  /usr/share/java

hadoop003已经有Java驱动了，就不再操作上面三条语句了

安装Canal-Admin 

[hadoop@hadoop003 software]$ wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.admin-1.1.4.tar.gz

[hadoop@hadoop003 software]$ wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz

 

[hadoop@hadoop003 ~]$ cd app/

[hadoop@hadoop003 app]$ mkdir canal-admin

[hadoop@hadoop003 software]$ tar -zxvf canal.admin-1.1.4.tar.gz -C ../app/canal-admin/

[hadoop@hadoop003 software]$ cd ../app/canal-admin

sudo vi /etc/profile

export CANAL_ADMIN_HOME=/home/hadoop/app/canal-admin
export PATH=${CANAL_ADMIN_HOME}/bin:$PATH

source /etc/profile

[hadoop@hadoop003 canal-admin]$ vi $CANAL_ADMIN_HOME/conf/application.yml

server:
  port: 8089
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: hadoop003:3306
  database: canal_manager
  username: canaladmin
  password: canaladmin%123
  driver-class-name: com.mysql.jdbc.Driver
  #url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true

  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

我们安装的MySQL8，但是Canal-Admin默认的MySQL驱动不支持MySQL8，因此需要替换驱动包:

[hadoop@hadoop003 canal-admin]$ ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_ADMIN_HOME/lib/mysql-connector-java-8.0.18.jar

[hadoop@hadoop003 canal-admin]$ mv $CANAL_ADMIN_HOME/lib/mysql-connector-java-5.1.40.jar $CANAL_ADMIN_HOME/lib/mysql-connector-java-5.1.40.jar.bak

初始化元数据库

mysql> source /home/hadoop/app/canal-admin/conf/canal_manager.sql

启动

[hadoop@hadoop003 bin]$ sh $CANAL_ADMIN_HOME/bin/startup.sh

安装Canal-Server

引入了canal-admin之后，canal-server之前面向命令行的运维方式需要有一些变化，主要的变化在于配置体系上，每个server节点上不应该再去维护复杂而且冗长的 canal.properties/instance.properties，应该选择以最小化、无状态的方式去启动，因此在canal 1.1.4 上，对于配置做了一些重构来支持canal-admin，同时也兼容了原先的命令行运维模式。

[hadoop@hadoop003 app]$ mkdir canal-server

[hadoop@hadoop003 app]$ tar -zxvf ../software/canal.deployer-1.1.4.tar.gz  -C canal-server

[hadoop@hadoop003 app]$ sudo vi /etc/profile

export CANAL_SERVER_HOME=/home/hadoop/app/canal-server
export PATH=${CANAL_SERVER_HOME}/bin:$PATH

[hadoop@hadoop003 app]$ source /etc/profile

采用canal-admin来管理canal-server之后，不再需要按照以前的方式修改大量配置了。

[hadoop@hadoop003 app]$ cd $CANAL_SERVER_HOME/conf/

[hadoop@hadoop003 conf]$ mv canal.properties canal.properties.bak

[hadoop@hadoop003 conf]$ mv canal_local.properties canal.properties

[hadoop@hadoop003 conf]$ vi $CANAL_SERVER_HOME/conf/canal.properties

# register ip
canal.register.ip =

# canal admin config
canal.admin.manager = hadoop003:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =

替换MySQL驱动包

我们安装的MySQL8，但是Canal-Server默认的MySQL驱动不支持MySQL8，因此需要替换驱动包

[hadoop@hadoop003 conf]$ ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_SERVER_HOME/lib/mysql-connector-java-8.0.18.jar

[hadoop@hadoop003 conf]$ mv $CANAL_SERVER_HOME/lib/mysql-connector-java-5.1.47.jar $CANAL_SERVER_HOME/lib/mysql-connector-java-5.1.47.jar.bak

启停canal-server

[hadoop@hadoop003 conf]$ $CANAL_SERVER_HOME/bin/startup.sh

可以看到，Canal-Server部署成功