DataX简单上手

概述

DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。

下载地址

https://github.com/alibaba/DataX

服务器配置

1. JDK1.8
2. Python2.x
3. Apache Maven 3.x

[root@localhost datax]# mvn --version
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T23:06:16+08:00)
Maven home: /opt/maven
Java version: 1.8.0_11, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_11/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"

[root@localhost datax]# python --version
Python 2.7.5

安装使用

1. 将安装包下载后用xftp5放到/opt目录下
2. 解压

tar -xvzf datax.tar.gz

3. 使用自检测试脚本测试

python datax/bin/datax.py datax/job/job.json

这样就安装成功了

使用

python datax.py -r mysqlreader -w mysqlwriter

这里会生成一个模板，使用模板填写

[root@localhost opt]# python bin/datax.py -r mysqlreader -w mysqlwriter
python: can't open file 'bin/datax.py': [Errno 2] No such file or directory
[root@localhost opt]# python datax/bin/datax.py -r mysqlreader -w mysqlwriter

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


Please refer to the mysqlreader document:
     https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md 

Please refer to the mysqlwriter document:
     https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md 
 
Please save the following configuration as a json file and  use
     python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json 
to run the job.

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "column": [], 
                        "connection": [
                            {
                                "jdbcUrl": [], 
                                "table": []
                            }
                        ], 
                        "password": "", 
                        "username": "", 
                        "where": ""
                    }
                }, 
                "writer": {
                    "name": "mysqlwriter", 
                    "parameter": {
                        "column": [], 
                        "connection": [
                            {
                                "jdbcUrl": "", 
                                "table": []
                            }
                        ], 
                        "password": "", 
                        "preSql": [], 
                        "session": [], 
                        "username": "", 
                        "writeMode": ""
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": ""
            }
        }
    }
}

之后新建一个json文件，按模板填写,然后执行下面命令就可以了

python bin/datax.py mysql2mysql.json