概述
DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台,实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。
下载地址
https://github.com/alibaba/DataX
服务器配置
- JDK1.8
- Python2.x
- Apache Maven 3.x
[root@localhost datax]# mvn --version
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T23:06:16+08:00)
Maven home: /opt/maven
Java version: 1.8.0_11, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_11/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"
[root@localhost datax]# python --version
Python 2.7.5
安装使用
- 将安装包下载后用xftp5放到/opt目录下
- 解压
tar -xvzf datax.tar.gz
- 使用自检测试脚本测试
python datax/bin/datax.py datax/job/job.json
这样就安装成功了
使用
python datax.py -r mysqlreader -w mysqlwriter
这里会生成一个模板,使用模板填写
[root@localhost opt]# python bin/datax.py -r mysqlreader -w mysqlwriter
python: can't open file 'bin/datax.py': [Errno 2] No such file or directory
[root@localhost opt]# python datax/bin/datax.py -r mysqlreader -w mysqlwriter
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the mysqlreader document:
https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md
Please refer to the mysqlwriter document:
https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": [],
"table": []
}
],
"password": "",
"username": "",
"where": ""
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": "",
"table": []
}
],
"password": "",
"preSql": [],
"session": [],
"username": "",
"writeMode": ""
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}
之后新建一个json文件,按模板填写,然后执行下面命令就可以了
python bin/datax.py mysql2mysql.json