Datax安装

Datax安装

  1. 下载Datax安装包并上传到linux上

    下载地址:https://datax-opensource.oss-cn-hangzhou.aliyuncs.com/202303/datax.tar.gz

  2. 解压datax.tar.gz到目录,我选择把Datax解压到/usr/local/datax目录

    tar -zxvf datax.tar.gz -C /usr/local/datax
    
  3. 自行检测是否安装成功,执行如下命令

    python /usr/local/datax/bin/datax.py /usr/local/datax/job/job.json
    

    最后如果出现如下内容,则表明安装成功

    2023-08-08 16:20:58.686 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
    2023-08-08 16:20:58.687 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
    2023-08-08 16:20:58.688 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
    2023-08-08 16:20:58.688 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
    2023-08-08 16:20:58.689 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
    2023-08-08 16:20:58.693 [job-0] INFO  JobContainer - 
    	 [total cpu info] => 
    		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
    		-1.00%                         | -1.00%                         | -1.00%
                            
    
    	 [total gc info] => 
    		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
    		 Copy                 | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
    		 MarkSweepCompact     | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
    
    2023-08-08 16:20:58.693 [job-0] INFO  JobContainer - PerfTrace not enable!
    2023-08-08 16:20:58.694 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.012s |  All Task WaitReaderTime 0.130s | Percentage 100.00%
    2023-08-08 16:20:58.694 [job-0] INFO  JobContainer - 
    任务启动时刻                    : 2023-08-08 16:20:48
    任务结束时刻                    : 2023-08-08 16:20:58
    任务总计耗时                    :                 10s
    任务平均流量                    :          253.91KB/s
    记录写入速度                    :          10000rec/s
    读出记录总数                    :              100000
    读写失败总数                    :                   0
    

    !!!!!!如果使用WinSCP上传压缩包可能会遇到权限不够的问题,参考https://www.jianshu.com/p/4f1d4a23eaeb即可解决!!!!!!

Datax-web安装

环境准备
  • MySQL 5.X版本 (如果会使用docker技术可以直接拉取MySQL镜像)
  • JDK 1.8.0_xxx版本 (需要提前安装)
  • DataX
  • python(如果安装的虚拟机是(Ubuntu-20.xx不需安装自带python)
安装Datax-web
  1. 下载datax-web-2.1.2压缩包(见资源可自取)

  2. 将datax-web-2.1.2.tar.gz上传至虚拟机后解压,命令如下(路径自选)

    tar -zxvf datax-web-2.1.2.tar.gz -C /usr/local/datax-web
    
  3. cd到压缩后的目录下输入./bin/install.sh

    root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# ./bin/install.sh
    

    成功后会看到如下代码(过程中遇到需要输入(y/n)的全输入y,需要输入用户名,密码,数据库名时需要与MySQL中的一致)

    2023-08-08 16:25:41.758 [INFO] (8043) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/bin/../modules].
    2023-08-08 16:25:41.765 [INFO] (8043)  ####### Start To Uncompress Packages ######
    2023-08-08 16:25:41.767 [INFO] (8043) Uncompressing....
    Do you want to decompress this package: [datax-admin_2.1.2_1.tar.gz]? (Y/N)y
    2023-08-08 16:25:46.746 [INFO] (8043)  Uncompress package: [datax-admin_2.1.2_1.tar.gz] to modules directory
    Do you want to decompress this package: [datax-executor_2.1.2_1.tar.gz]? (Y/N)y
    2023-08-08 16:25:48.978 [INFO] (8043)  Uncompress package: [datax-executor_2.1.2_1.tar.gz] to modules directory
    2023-08-08 16:25:49.227 [INFO] (8043)  ####### Finish To Umcompress Packages ######
    Scan modules directory: [/usr/local/datax-web/datax-web-2.1.2/bin/../modules] to find server under dataxweb
    2023-08-08 16:25:49.230 [INFO] (8043)  ####### Start To Install Modules ######
    2023-08-08 16:25:49.231 [INFO] (8043) Module servers could be installed:
     [datax-admin]  [datax-executor] 
    Do you want to confiugre and install [datax-admin]? (Y/N)y
    2023-08-08 16:25:50.178 [INFO] (8043)  Install module server: [datax-admin]
    Start to make directory
    2023-08-08 16:25:50.188 [INFO] (8082)  Start to build directory
    2023-08-08 16:25:50.192 [INFO] (8082) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../logs].
    2023-08-08 16:25:50.231 [INFO] (8082) Directory or file: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../conf] has been exist
    2023-08-08 16:25:50.232 [INFO] (8082) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-admin/bin/../data].
    end to make directory
    Start to initalize database
    2023-08-08 16:25:50.306 [INFO] (8082)  Scan out mysql command, so begin to initalize the database
    Do you want to initalize database with sql: [/usr/local/datax-web/datax-web-2.1.2/bin/db/datax_web.sql]? (Y/N)y
    Please input the db host(default: 127.0.0.1): # mysql所在服务器的ip地址,默认是127.0.0.1,这里localhost填与不填没有任何意义,只作为举例
    Please input the db port(default: 3306): # mysql端口默认3306,如果是3306可以直接点击enter键,如果不是输入对应端口号
    Please input the db username(default: root):# 默认msyql用户是root,根据实际情况填写
    Please input the db password(default: ): # 输入mysql用户对应的密码
    Please input the db name(default: dataxweb) # 输入作为datax-web的系统数据库,默认为dataxweb
    mysql: [Warning] Using a password on the command line interface can be insecure.
    ERROR 1045 (28000): Access denied for user 'datax_web'@'localhost' (using password: YES)
    Do you want to confiugre and install [datax-executor]? (Y/N)y
    2023-08-08 16:28:37.537 [INFO] (8043)  Install module server: [datax-executor]
    2023-08-08 16:28:37.554 [INFO] (8356)  Start to build directory
    2023-08-08 16:28:37.556 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../logs].
    2023-08-08 16:28:37.583 [INFO] (8356) Directory or file: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../conf] has been exist
    2023-08-08 16:28:37.585 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../data].
    2023-08-08 16:28:37.615 [INFO] (8356) Creating directory: [/usr/local/datax-web/datax-web-2.1.2/modules/datax-executor/bin/../json].
    2023-08-08 16:28:37.643 [INFO] (8043)  ####### Finish To Install Modules ######
    
  4. 在安装完成后,可以通过配置文件自由配置所要连接的Mysql数据库,修改datax-web-2.1.2/modules/datax-admin/conf目录下的bootstrap.properties文件

    root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# vi ./modules/datax-admin/conf/bootstrap.properties
    

    修改内容依据自己数据库信息进行修改

    # 配置文件中内容如下所示:
    #Database
    #DB_HOST=
    #DB_PORT=
    #DB_USERNAME=
    #DB_PASSWORD=
    #DB_DATABASE=
    # 根据实际情况进行配置即可
    
  5. 安装完成后,还需要配置datax-web-2.1.2/modules/datax-admin/bin/env.properties 指定JAVA_HOME的路径

    root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# vi ./modules/datax-admin/bin/env.properties
    

    需要修改如下内容

    # environment variables
    JAVA_HOME="/usr/local/java/jdk1.8.0_131"
    
    SERVER_PORT=9527
    
  6. 安装完成后,还需要配置datax-web-2.1.2/modules/datax-executor/bin/env.properties 指定PYTHON_PATH的路径

    root@cqie-VirtualBox:/usr/local/datax-web# vi datax-web-2.1.2/modules/datax-executor/bin/env.properties
    

    执行完命令过后需要修改下面的信息

    # environment variables自己安装的jdk路径
    JAVA_HOME="/usr/local/java/jdk1.8.0_131"
    
    ## 保持和datax-admin端口一致
    DATAX_ADMIN_PORT=9527
    
    ## PYTHON脚本执行位置
    #PYTHON_PATH=/home/hadoop/install/datax/bin/datax.py
    PYTHON_PATH=/usr/local/datax/bin/datax.py
    
  7. 完成以上所有操作过后就可以启动datax-web了,操作命令如下

    root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# ./bin/start-all.sh 
    

    成功会看到如下界面

    2023-08-08 17:14:38.648 [INFO] (11815)  Try To Start Modules In Order 
    2023-08-08 17:14:38.665 [INFO] (11822)  ####### Begin To Start Module: [datax-admin] ######
    2023-08-08 17:14:38.695 [INFO] (11829) load environment variables
    2023-08-08 17:14:39.152 [INFO] (11829) /usr/local/java/jdk1.8.0_131/bin/java
    2023-08-08 17:14:39.154 [INFO] (11829) Waiting DATAX-ADMIN to start complete ...
    2023-08-08 17:14:39.659 [INFO] (11829) DATAX-ADMIN start success
    2023-08-08 17:14:39.736 [INFO] (12015)  ####### Begin To Start Module: [datax-executor] ######
    2023-08-08 17:14:39.799 [INFO] (12022) load environment variables
    2023-08-08 17:14:41.433 [INFO] (12022) /usr/local/java/jdk1.8.0_131/bin/java
    2023-08-08 17:14:41.448 [INFO] (12022) Waiting DATAX-EXEXUTOR to start complete ...
    2023-08-08 17:14:42.331 [INFO] (12022) DATAX-EXEXUTOR start success
    
  8. 输入jps查看是否启动成功(如看到一下进程则说明启动成功)

    root@cqie-VirtualBox:/usr/local/datax-web/datax-web-2.1.2# jps
    12240 DataXExecutorApplication
    11985 DataXAdminApplication
    12350 Jps
    

``
9. 输入http://192.168.56.101:9527/index.html即可访问datax-web网页(ip根据自己虚拟机ip确定,默认用户名admin密码123456)
在这里插入图片描述
登陆成功后会看到此页面
在这里插入图片描述

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值