介绍:
DataX 是一个异构数据源离线同步工具,致力于实现包括关系型数据库(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能
-
方法一、直接下载DataX工具包:DataX下载地址
下载后解压至本地某个目录,进入bin目录,即可运行同步作业:
$ cd {YOUR_DATAX_HOME}/bin $ python datax.py {YOUR_JOB.json}
自检脚本: python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json
-
方法二、下载DataX源码,自己编译:DataX源码
(1)、下载DataX源码:
$ git clone git@github.com:alibaba/DataX.git
(2)、通过maven打包:
$ cd {DataX_source_code_home} $ mvn -U clean package assembly:assembly -Dmaven.test.skip=true
打包成功后的DataX包位于 {DataX_source_code_home}/target/datax/datax/,目录结构跟方法一一样
下面有几个实例:
oracle->file:
{
"job": {
"content": [
{
"reader": {
"name": "oraclereader",
"parameter": {
"username": "loge",
"password": "123456",
"where": "",
"connection": [
{
"querySql": [
"select * from table1"
],
"jdbcUrl": [
"jdbc:oracle:thin:@192.168.1.10:1521:orcl"
]
}
]
}
},
"writer": {
"name": "txtfilewriter",
"parameter": {
"path": "/space/datax/tmp/",
"fileName": "oracledata",
"writeMode": "truncate",
"dateFormat": "yyyy-MM-dd HH:mm:ss"
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}
oracle->mysql:
{
"job": {
"setting": {
"speed": {
"channel": 1
}
},
"content": [
{
"reader": {
"name": "oraclereader",
"parameter": {
"username": "loge",
"password": "123456",
"column": ["*"],
"where": "",
"connection": [
{
"querySql": [
"select * from table1"
],
"jdbcUrl": [
"jdbc:oracle:thin:@192.168.1.10:1521:orcl"
]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "loge",
"password": "123456",
"column": [
"id",
"name"
],
"session": [
"set session sql_mode='ANSI'"
],
"preSql": [
"delete from table1"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://192.168.1.22:3306/t?useUnicode=true&characterEncoding=utf8",
"table": [
"table1"
]
}
]
}
}
}
]
}
}