DataX使用记录

DataX使用记录

DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、SQL Server、Oracle、PostgreSQL、HDFS、Hive、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

00特性

DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。

01下载DataX源码

$ git clone https://github.com/alibaba/DataX.git

02源码编译

$ cd /home/hadoop/source-code/DataX/
$ mvn -U clean package assembly:assembly -Dmaven.test.skip=true

编译成功

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:17 min
[INFO] Finished at: 2019-06-29T10:26:35+08:00
[INFO] Final Memory: 504M/2052M
[INFO] ------------------------------------------------------------------------

编译过程遇到的问题

问题一：

[ERROR] Failed to execute goal on project odpsreader: Could not resolve dependencies for project com.alibaba.datax:odpsreader:jar:0.0.1-SNAPSHOT: Could not find artifact com.alibaba.external:bouncycastle.provider:jar:1.38-jdk15 in nexus-aliyun (http://maven.aliyun.com/nexus/content/repositories/central) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :odpsreader

解决办法：将odpsreader模块下对应依赖下的<version>0.19.3-public</version>改为<version>0.20.7-public</version>

		<dependency>
			<groupId>com.aliyun.odps</groupId>
			<artifactId>odps-sdk-core</artifactId>
			<!--<version>0.19.3-public</version>-->
			<version>0.20.7-public</version>
		</dependency>

问题二：

[ERROR] Failed to execute goal on project otsstreamreader: Could not resolve dependencies for project com.alibaba.datax:otsstreamreader:jar:0.0.1-SNAPSHOT: Could not find artifact com.aliyun.openservices:tablestore-streamclient:jar:1.0.0-SNAPSHOT -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :otsstreamreader

解决办法：将otsstreamreader模块下对应依赖下的<version>1.0.0-SNAPSHOT</version>改为<version>1.0.0</version>

        <dependency>
            <groupId>com.aliyun.openservices</groupId>
            <artifactId>tablestore-streamclient</artifactId>
            <!--<version>1.0.0-SNAPSHOT</version>-->
            <version>1.0.0</version>
        </dependency>

问题三：

ERROR] Failed to execute goal on project odps