Datax安装
-
下载Datax安装包并上传到linux上
下载地址:https://datax-opensource.oss-cn-hangzhou.aliyuncs.com/202303/datax.tar.gz
-
解压datax.tar.gz到目录,我选择把Datax解压到/usr/local/datax目录
tar -zxvf datax.tar.gz -C /usr/local/datax
-
自行检测是否安装成功,执行如下命令
python /usr/local/datax/bin/datax.py /usr/local/datax/job/job.json
最后如果出现如下内容,则表明安装成功
2023-08-08 16:20:58.686 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2023-08-08 16:20:58.687 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do post work. 2023-08-08 16:20:58.688 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do post work. 2023-08-08 16:20:58.688 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2023-08-08 16:20:58.689 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook 2023-08-08 16:20:58.693 [job-0] INFO JobContainer - [total cpu info] => averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime Copy | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s MarkSweepCompact | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s 2023-08-08 16:20:58.693 [job-0] INFO JobContainer - PerfTrace not enable! 2023-08-08 16:20:58.694 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.012s | All Task WaitReaderTime 0.130s | Percentage 100.00% 2023-08-08 16:20:58.694 [job-0] INFO JobContainer - 任务启动时刻 : 2023-08-08 16:20:48 任务结束时刻 : 2023-08-08 16:20:58 任务总计耗时 : 10s 任务平均流量 : 253.91KB/s 记录写入速度 : 10000rec/s 读出记录总数 : 100000 读写失败总数 : 0
!!!!!!如果使用WinSCP上传压缩包可能会遇到权限不够的问题,参考https://www.jianshu.com/p/4f1d4a23eaeb即可解决!!!!!!
Datax-web安装
环境准备
- MySQL 5.X版本 (如果会使用docker技术可以直接拉取MySQL镜像)
- JDK 1.8.0_xxx版本 (需要提前安装)
- DataX
- python(如果安装的虚拟机是(Ubuntu-20.xx不需安装自带python)
安装Datax-web
-
下载datax-web-2.1.2压缩包(见资源可自取)
-
将datax-web-2.1.2.tar.gz上传至虚拟机后解压,命令如下(路径自选)
tar -zxvf datax-web-2.1.2.tar.gz -C /usr/local/datax-web
-
cd到压缩后的目录下输入./bin/install.sh
root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# ./bin/install.sh
成功后会看到如下代码(过程中遇到需要输入(y/n)的全输入y,需要输入用户名,密码,数据库名时需要与MySQL中的一致)
2023-08-08 16:25:41.758 [INFO] (8043) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/bin/../modules]. 2023-08-08 16:25:41.765 [INFO] (8043) ####### Start To Uncompress Packages ###### 2023-08-08 16:25:41.767 [INFO] (8043) Uncompressing.... Do you want to decompress this package: [datax-admin_2.1.2_1.tar.gz]? (Y/N)y 2023-08-08 16:25:46.746 [INFO] (8043) Uncompress package: [datax-admin_2.1.2_1.tar.gz] to modules directory Do you want to decompress this package: [datax-executor_2.1.2_1.tar.gz]? (Y/N)y 2023-08-08 16:25:48.978 [INFO] (8043) Uncompress package: [datax-executor_2.1.2_1.tar.gz] to modules directory 2023-08-08 16:25:49.227 [INFO] (8043) ####### Finish To Umcompress Packages ###### Scan modules directory: [/usr/local/datax-web/datax-web-2.1.2/bin/../modules] to find server under dataxweb 2023-08-08 16:25:49.230 [INFO] (8043) ####### Start To Install Modules ###### 2023-08-08 16:25:49.231 [INFO] (8043) Module servers could be installed: [datax-admin] [datax-executor] Do you want to confiugre and install [datax-admin]? (Y/N)y 2023-08-08 16:25:50.178 [INFO] (8043) Install module server: [datax-admin] Start to make directory 2023-08-08 16:25:50.188 [INFO] (8082) Start to build directory 2023-08-08 16:25:50.192 [INFO] (8082) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../logs]. 2023-08-08 16:25:50.231 [INFO] (8082) Directory or file: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../conf] has been exist 2023-08-08 16:25:50.232 [INFO] (8082) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../data]. end to make directory Start to initalize database 2023-08-08 16:25:50.306 [INFO] (8082) Scan out mysql command, so begin to initalize the database Do you want to initalize database with sql: [/usr/local/datax-web/datax-web-2.1.2/bin/db/datax_web.sql]? (Y/N)y Please input the db host(default: 127.0.0.1): # mysql所在服务器的ip地址,默认是127.0.0.1,这里localhost填与不填没有任何意义,只作为举例 Please input the db port(default: 3306): # mysql端口默认3306,如果是3306可以直接点击enter键,如果不是输入对应端口号 Please input the db username(default: root):# 默认msyql用户是root,根据实际情况填写 Please input the db password(default: ): # 输入mysql用户对应的密码 Please input the db name(default: dataxweb) # 输入作为datax-web的系统数据库,默认为dataxweb mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user 'datax_web'@'localhost' (using password: YES) Do you want to confiugre and install [datax-executor]? (Y/N)y 2023-08-08 16:28:37.537 [INFO] (8043) Install module server: [datax-executor] 2023-08-08 16:28:37.554 [INFO] (8356) Start to build directory 2023-08-08 16:28:37.556 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../logs]. 2023-08-08 16:28:37.583 [INFO] (8356) Directory or file: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../conf] has been exist 2023-08-08 16:28:37.585 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../data]. 2023-08-08 16:28:37.615 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../json]. 2023-08-08 16:28:37.643 [INFO] (8043) ####### Finish To Install Modules ######
-
在安装完成后,可以通过配置文件自由配置所要连接的Mysql数据库,修改datax-web-2.1.2/modules/datax-admin/conf目录下的bootstrap.properties文件
root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# vi ./modules/datax-admin/conf/bootstrap.properties
修改内容依据自己数据库信息进行修改
# 配置文件中内容如下所示: #Database #DB_HOST= #DB_PORT= #DB_USERNAME= #DB_PASSWORD= #DB_DATABASE= # 根据实际情况进行配置即可
-
安装完成后,还需要配置datax-web-2.1.2/modules/datax-admin/bin/env.properties 指定JAVA_HOME的路径
root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# vi ./modules/datax-admin/bin/env.properties
需要修改如下内容
# environment variables JAVA_HOME="/usr/local/java/jdk1.8.0_131" SERVER_PORT=9527
-
安装完成后,还需要配置datax-web-2.1.2/modules/datax-executor/bin/env.properties 指定PYTHON_PATH的路径
root@cqie-VirtualBox:/usr/local/datax-web# vi datax-web-2.1.2/modules/datax-executor/bin/env.properties
执行完命令过后需要修改下面的信息
# environment variables自己安装的jdk路径 JAVA_HOME="/usr/local/java/jdk1.8.0_131" ## 保持和datax-admin端口一致 DATAX_ADMIN_PORT=9527 ## PYTHON脚本执行位置 #PYTHON_PATH=/home/hadoop/install/datax/bin/datax.py PYTHON_PATH=/usr/local/datax/bin/datax.py
-
完成以上所有操作过后就可以启动datax-web了,操作命令如下
root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# ./bin/start-all.sh
成功会看到如下界面
2023-08-08 17:14:38.648 [INFO] (11815) Try To Start Modules In Order 2023-08-08 17:14:38.665 [INFO] (11822) ####### Begin To Start Module: [datax-admin] ###### 2023-08-08 17:14:38.695 [INFO] (11829) load environment variables 2023-08-08 17:14:39.152 [INFO] (11829) /usr/local/java/jdk1.8.0_131/bin/java 2023-08-08 17:14:39.154 [INFO] (11829) Waiting DATAX-ADMIN to start complete ... 2023-08-08 17:14:39.659 [INFO] (11829) DATAX-ADMIN start success 2023-08-08 17:14:39.736 [INFO] (12015) ####### Begin To Start Module: [datax-executor] ###### 2023-08-08 17:14:39.799 [INFO] (12022) load environment variables 2023-08-08 17:14:41.433 [INFO] (12022) /usr/local/java/jdk1.8.0_131/bin/java 2023-08-08 17:14:41.448 [INFO] (12022) Waiting DATAX-EXEXUTOR to start complete ... 2023-08-08 17:14:42.331 [INFO] (12022) DATAX-EXEXUTOR start success
-
输入jps查看是否启动成功(如看到一下进程则说明启动成功)
root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# jps 12240 DataXExecutorApplication 11985 DataXAdminApplication 12350 Jps
``
9. 输入http://192.168.56.101:9527/index.html即可访问datax-web网页(ip根据自己虚拟机ip确定,默认用户名admin密码123456)
登陆成功后会看到此页面