使用DolphinScheduler调度DataX完成Mysql到Mysql的数据同步
-
DolphinScheduler集群的部署
-
基础环境:
-
Mysql(5.7.25)
-
JDK(1.8.0_144)
-
Zookeeper(3.4.10)
-
-
-
节点部署安排
- 三台主机hadoop01,hadoop02,hadoop03
- hadoop01 : master、alert、api machine
- hadoop02:worker machine
- hadoop03:worker machine
- 三台主机hadoop01,hadoop02,hadoop03
-
集群部署步骤
-
将下载的二进制tar.gz包解压到部署目录。
mkdir -p /opt/module/dolphinscheduler; cd /opt/module/dolphinscheduler; tar -zxvf apache-dolphinscheduler-1.3.6-bin.tar.gz -C /opt/module/dolphinscheduler;
-
创建用户:在所有部署调度的机器上创建部署用户,并且一定要配置sudo免密。假如我们计划在hadoop01,hadoop02,hadoop03这3台机器上部署调度,首先需要在每台机器上都创建部署用户。
useradd dolphinscheduler; echo "密码" | passwd --stdin dolphinscheduler; sudo echo 'dolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers; sudo sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers;
-
配置hosts和ssh,为后续的一键部署做准备。
vi /etc/hosts #127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.32.10 hadoop01 192.168.32.11 hadoop02 192.168.32.12 hadoop03
在hadoop01上,切换到部署用户并配置ssh本机免密登录
su dolphinscheduler ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys ssh-copy-id hadoop02 ssh-copy-id hadoop03
-
修改目录权限,使得部署用户对dolphinscheduler目录有操作权限
sudo chown -R dolphinscheduler:dolphinscheduler dolphinscheduler
-
数据库初始化
mysql -h192.168.xx.xx -P3306 -uroot -p; mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY '密码'
-