datax简单使用

系统要求

• Linux
• JDK(1.8以上，推荐1.8)
• Python(推荐Python2.6.X)
• Apache Maven 3.x (Compile DataX)

启动

• 工具部署
  • 方法一、直接下载DataX工具包：DataX下载地址

下载后解压至本地某个目录，进入bin目录，即可运行同步作业：

$ cd  {YOUR_DATAX_HOME}/bin

$ python datax.py {YOUR_JOB.json}

自检脚本：    python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

目录结构：

• ${DATAX_HOME}/bin: 可执行程序目录。
• ${DATAX_HOME}/conf: 框架配置目录。
• ${DATAX_HOME}/lib: 框架依赖库目录。
• ${DATAX_HOME}/plugin: 插件目录。

插件目录分为reader和writer子目录，读写插件分别存放。插件目录规范如下：

• ${PLUGIN_HOME}/libs: 插件的依赖库。
• ${PLUGIN_HOME}/plugin-name-version.jar: 插件本身的jar。
• ${PLUGIN_HOME}/plugin.json: 插件描述文件。

尽管框架加载插件时，会把${PLUGIN_HOME}下所有的jar放到classpath，但还是推荐依赖库的jar和插件本身的jar分开存放。

注意： 插件的目录名字必须和plugin.json中定义的插件名称一致。

配置文件

DataX使用json作为配置文件的格式。一个典型的DataX任务配置如下：

{

  "job": {

    "content": [

      {

        "reader": {

          "name": "odpsreader",

          "parameter": {

            "accessKey": "",

            "accessId": "",

            "column": [""],

            "isCompress": "",

            "odpsServer": "",

            "partition": [

              ""

            ],

            "project": "",

            "table": "",

            "tunnelServer": ""

          }

        },

        "writer": {

          "name": "oraclewriter",

          "parameter": {

            "username": "",

            "password": "",

            "column": ["*"],

            "connection": [

              {

                "jdbcUrl": "",

                "table": [

                  ""

                ]

              }

            ]

          }

        }

      }

    ]

  }

}

示例oracle到mysql

[root@node2 bin]# cat ora2my.json

{
    "job":{
        "setting":{
            "speed":{
                "channel":3
            },
            "errorLimit":{
                "record":0,
                "percentage":0.02
            }
        },
        "content":[
            {
                "reader":{
                    "name":"oraclereader",
                    "parameter":{
                        "username":"test",
                        "password":"test",
                        "column":[
                            "ID",
                            "NAME",
                            "ALIAS_NAME",
                            "AMOUNT",
                            "SCORE",
                            "TEXT_B",
                            "TEXT_C",
                            "GMT_CREATE",
                            "GMT_MODIFIED"
                        ],
                        "connection":[
                            {
                                "table":[
                                    "test_all_one_pk"
                                ],
                                "jdbcUrl":[
                                    "jdbc:oracle:thin:@192.168.21.10:1521:ora"
                                ]
                            }
                        ]
                    }
                },
                "writer":{
                    "name":"mysqlwriter",
                    "parameter":{
                        "writeMode":"insert",
                        "username":"test",
                        "password":"123456",
                        "column":[
                            "ID",
                            "NAME",
                            "ALIAS_NAME",
                            "AMOUNT",
                            "SCORE",
                            "TEXT_B",
                            "TEXT_C",
                            "GMT_CREATE",
                            "GMT_MODIFIED"
                        ],
                        "session":[
                            "set session sql_mode='ANSI'"
                        ],
                        "preSql":[
                            "delete from test_all_one_pk"
                        ],
                        "connection":[
                            {
                                "jdbcUrl":"jdbc:mysql://192.168.21.30:3306/member?useUnicode=true&characterEncoding=gbk",
                                "table":[
                                    "test_all_one_pk"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}

同步测试：