java集成datax
1、需求
公司搭建自己的数据平台,datax作为数据采集工具还是非常强有力的,但是官方是通过python调取datax.py文件实现数据同步的,没法子很好的集成在自己的数据平台上。
2、获取方式
1、datax 源代码下载
通过源代码的方式进行打包
由于pom文件内容太多,就只是加载了测试用的reader和writer
1、reader
2、writer
3、pom文件:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.datax</groupId>
<artifactId>datax-all</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-core</artifactId>
<version>1.3</version>
</dependency>
</dependencies>
<name>datax-all</name>
<packaging>pom</packaging>
<properties>
<jdk-version>1.8</jdk-version>
<datax-project-version>0.0.1-SNAPSHOT</datax-project-version>
<commons-lang3-version>3.3.2</commons-lang3-version>
<commons-configuration-version>1.10</commons-configuration-version>
<commons-cli-version>1.2</commons-cli-version>
<fastjson-version>1.2.49</fastjson-version>
<guava-version>16.0.1</guava-version>
<diamond.version>3.7.2.1-SNAPSHOT</diamond.version>
<!--slf4j 1.7.10 和 logback-classic 1.0.13 是好基友 -->
<slf4j-api-version>1.7.10</slf4j-api-version>
<logback-classic-version>1.0.13</logback-classic-version>
<commons-io-version>2.4</commons-io-version>
<junit-version>4.13.1</junit-version>
<tddl.version>5.1.22-1</tddl.version>
<swift-version>1.0.0</swift-version>
<project-sourceEncoding>UTF-8</project-sourceEncoding>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<mysql.driver.version>5.1.47</mysql.driver.version>
</properties>
<modules>
<module>common</module>
<module>core</module>
<module>transformer</module>
<!-- reader -->
<module>mysqlreader</module>
<!-- <module>drdsreader</module>-->
<!-- <module>sqlserverreader</module>-->
<!-- <module>postgresqlreader</module>-->
<!-- <module>kingbaseesreader</module>-->
<!-- <module>oraclereader</module>-->
<!-- <module>cassandrareader</module>-->
<!-- <module>oceanbasev10reader</module>-->
<!-- <module>rdbmsreader</module>-->
<!-- <module>odpsreader</module>-->
<!-- <module>otsreader</module>-->
<!-- <module>otsstreamreader</module>-->
<!-- <module>hbase11xreader</module>-->
<!-- <module>hbase094xreader</module>-->
<!-- <module>hbase11xsqlreader</module>-->
<!-- <module>hbase20xsqlreader</module>-->
<!-- <module>ossreader</module>-->
<!-- <module>hdfsreader</module>-->
<!-- <module>ftpreader</module>-->
<!-- <module>txtfilereader</module>-->
<!-- <module>streamreader</module>-->
<!-- <module>mongodbreader</module>-->
<!-- <module>tdenginereader</module>-->
<!-- <module>gdbreader</module>-->
<!-- <module>tsdbreader</module>-->
<!-- <module>opentsdbreader</module>-->
<!-- writer -->
<module>mysqlwriter</module>
<!-- <module>starrockswriter</module>-->
<!-- <module>drdswriter</module>-->
<!-- <module>oraclewriter</module>-->
<!-- <module>sqlserverwriter</module>-->
<!-- <module>postgresqlwriter</module>-->
<!-- <module>kingbaseeswriter</module>-->
<!-- <module>adswriter</module>-->
<!-- <module>oceanbasev10writer</module>-->
<!-- <module>adbpgwriter</module>-->
<!-- <module>hologresjdbcwriter</module>-->
<!-- <module>rdbmswriter</module>-->
<!-- <module>odpswriter</module>-->
<!-- <module>osswriter</module>-->
<!-- <module>otswriter</module>-->
<!-- <module>hbase11xwriter</module>-->
<!-- <module>hbase094xwriter</module>-->
<!-- <module>hbase11xsqlwriter</module>-->
<!-- <module>hbase20xsqlwriter</module>-->
<!-- <module>kuduwriter</module>-->
<!-- <module>ftpwriter</module>-->
<!-- <module>hdfswriter</module>-->
<!-- <module>txtfilewriter</module>-->
<!-- <module>streamwriter</module>-->
<!-- <module>elasticsearchwriter</module>-->
<!-- <module>mongodbwriter</module>-->
<!-- <module>tdenginewriter</module>-->
<!-- <module>ocswriter</module>-->
<!-- <module>tsdbwriter</module>-->
<!-- <module>gdbwriter</module>-->
<!-- <module>oscarwriter</module>-->
<!-- <module>cassandrawriter</module>-->
<!-- <module>clickhousewriter</module>-->
<!-- common support module -->
<module>plugin-rdbms-util</module>
<module>plugin-unstructured-storage-util</module>
</modules>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons-lang3-version}</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson-version}</version>
</dependency>
<!--<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava-version}</version>
</dependency>-->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io-version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j-api-version}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logback-classic-version}</version>
</dependency>
<dependency>
<groupId>com.taobao.tddl</groupId>
<artifactId>tddl-client</artifactId>
<version>${tddl.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>com.taobao.diamond</groupId>
<artifactId>diamond-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.taobao.diamond</groupId>
<artifactId>diamond-client</artifactId>
<version>${diamond.version}</version>
</dependency>
<dependency>
<groupId>com.alibaba.search.swift</groupId>
<artifactId>swift_client</artifactId>
<version>${swift-version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit-version}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
</dependency>
</dependencies>
</dependencyManagement>
<repositories>
<repository>
<id>central</id>
<name>Nexus aliyun</name>
<url>https://maven.aliyun.com/repository/central</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>spring</id>
<name>spring</name>
<url>https://maven.aliyun.com/repository/spring</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>central</id>
<name>Nexus aliyun</name>
<url>https://maven.aliyun.com/repository/central</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<finalName>datax</finalName>
<descriptors>
<descriptor>package.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk-version}</source>
<target>${jdk-version}</target>
<encoding>${project-sourceEncoding}</encoding>
</configuration>
</plugin>
</plugins>
</build>
</project>
4、打包方式
5、找到打包出来的datax文件
2、datax 压缩包下载
下载下来的压缩包,解压如下所示:
3、在pom文件中,加入相关依赖
<!-- datax-->
<dependency>
<groupId>com.datax</groupId>
<artifactId>datax-core</artifactId>
<version>0.0.1</version>
<systemPath>${project.basedir}/lib/datax-core-0.0.1-SNAPSHOT.jar</systemPath>
</dependency>
<dependency>
<groupId>com.datax</groupId>
<artifactId>datax-common</artifactId>
<version>0.0.1</version>
<systemPath>${project.basedir}/lib/datax-common-0.0.1-SNAPSHOT.jar</systemPath>
</dependency>
4、引入相应的jar包
5、编写相应的job测试json文件
job.json文件内容:
{
"job": {
"setting": {
"speed": {
"channel": 4
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "密码",
"connection": [
{
"jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/test"],
"querySql": ["select t.id,t.name,t.status from users t"]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "root",
"password": "密码",
"writeMode": "insert",
"column": ["id","name","status"],
"connection": [
{
"table": [
"temp_users"
],
"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/test"
}
]
}
}
}
]
}
}
6、编写main方法进行测试
1、获取启动类
通过上面的方法你已经将dataX 集成到你的系统了,但是你如何启动dataX呢,在com.alibaba.datax.core.Engine类中有个main(String[])和entry(String[])方法,调用启动。启动代码如下:
/**
* datax
*
* @author LZH
* @version 1.0
* @date 2022/09/19 16:16:13
*/
public class Datax {
public static void main(String[] args) throws Throwable {
System.setProperty("datax.home", "E:\\datax");
String[] datxArgs = {"-job", getCurrentClasspath()+"/datax/job.json", "-mode", "standalone", "-jobid", "-1"};
try {
Engine.entry(datxArgs);
}catch (Exception e){
e.printStackTrace();
}
}
//获取resource的路径
public static String getCurrentClasspath(){
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
String currentClasspath = classLoader.getResource("").getPath();
// 当前操作系统
String osName = System.getProperty("os.name");
if (osName.startsWith("Win")) {
// 删除path中最前面的/
currentClasspath = currentClasspath.substring(1, currentClasspath.length()-1);
}
return currentClasspath;
}
}
2、参数说明:
datax.home中存放的 是datax工具的路径,
getCurrentClasspath()+“/datax/job.json” 是配置同步数据库json文件的路径,
System.setProperty(“datax.home”, “E:\datax”); 设置datax.home环境变量,在系统启动时会通过datax.home环境变量获取配置文件和read、writer数据量插件。—>获取启动
String[] datxArgs = {“-job”, getCurrentClasspath()+“/datax/job.json”, “-mode”, “standalone”, “-jobid”, “-1”};定义构造参数数组,在系统启动时会用到。
通过上面的方法就可以通过Java的方式启动DataX
3、执行结果
D:\Java\jdk1.8.0_91\bin\java.exe "-javaagent:D:\IntelliJ IDEA 2021.1.1\lib\idea_rt.jar=1889:D:\IntelliJ IDEA 2021.1.1\bin" -Dfile.encoding=UTF-8 -classpath D:\Java\jdk1.8.0_91\jre\lib\charsets.jar;D:\Java\jdk1.8.0_91\jre\lib\deploy.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\cldrdata.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\dnsns.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\jaccess.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\jfxrt.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\localedata.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\nashorn.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunec.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunpkcs11.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\zipfs.jar;D:\Java\jdk1.8.0_91\jre\lib\javaws.jar;D:\Java\jdk1.8.0_91\jre\lib\jce.jar;D:\Java\jdk1.8.0_91\jre\lib\jfr.jar;D:\Java\jdk1.8.0_91\jre\lib\jfxswt.jar;D:\Java\jdk1.8.0_91\jre\lib\jsse.jar;D:\Java\jdk1.8.0_91\jre\lib\management-agent.jar;D:\Java\jdk1.8.0_91\jre\lib\plugin.jar;D:\Java\jdk1.8.0_91\jre\lib\resources.jar;D:\Java\jdk1.8.0_91\jre\lib\rt.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\target\classes;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-common-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-core-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-transformer-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\commons-cli-1.2.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\aspose-words-15.8.0-jdk16.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\ax-aes-api-1.0.0.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\target\classes;D:\MavenRepository\com\belerweb\pinyin4j\2.5.1\pinyin4j-2.5.1.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\lib\kingbase8-8.2.0.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\lib\DmJdbcDriver18.jar;D:\MavenRepository\com\alibaba\easyexcel\2.2.7\easyexcel-2.2.7.jar;D:\MavenRepository\org\apache\poi\poi\3.17\poi-3.17.jar;D:\MavenRepository\org\apache\commons\commons-collections4\4.1\commons-collections4-4.1.jar;D:\MavenRepository\org\apache\poi\poi-ooxml\3.17\poi-ooxml-3.17.jar;D:\MavenRepository\com\github\virtuald\curvesapi\1.04\curvesapi-1.04.jar;D:\MavenRepository\org\apache\poi\poi-ooxml-schemas\3.17\poi-ooxml-schemas-3.17.jar;D:\MavenRepository\cglib\cglib\3.1\cglib-3.1.jar;D:\MavenRepository\org\ow2\asm\asm\4.2\asm-4.2.jar;D:\MavenRepository\org\ehcache\ehcache\3.9.0\ehcache-3.9.0.jar;D:\MavenRepository\org\glassfish\jaxb\jaxb-runtime\2.3.3\jaxb-runtime-2.3.3.jar;D:\MavenRepository\org\glassfish\jaxb\txw2\2.3.3\txw2-2.3.3.jar;D:\MavenRepository\com\sun\istack\istack-commons-runtime\3.0.11\istack-commons-runtime-3.0.11.jar;D:\MavenRepository\com\sun\activation\jakarta.activation\1.2.2\jakarta.activation-1.2.2.jar;D:\MavenRepository\net\lingala\zip4j\zip4j\1.3.2\zip4j-1.3.2.jar;D:\MavenRepository\ma\glasnost\orika\orika-core\1.5.4\orika-core-1.5.4.jar;D:\MavenRepository\com\thoughtworks\paranamer\paranamer\2.8\paranamer-2.8.jar;D:\MavenRepository\com\carrotsearch\java-sizeof\0.0.5\java-sizeof-0.0.5.jar;D:\MavenRepository\org\codehaus\janino\janino\3.1.2\janino-3.1.2.jar;D:\MavenRepository\org\codehaus\janino\commons-compiler\3.1.2\commons-compiler-3.1.2.jar;D:\MavenRepository\joda-time\joda-time\2.3\joda-time-2.3.jar;D:\IdeaProject\ycdsj\lx-parent\lx-plugs\target\classes;D:\MavenRepository\org\springframework\boot\spring-boot-starter-data-elasticsearch\2.4.1\spring-boot-starter-data-elasticsearch-2.4.1.jar;D:\MavenRepository\org\springframework\data\spring-data-elasticsearch\4.1.2\spring-data-elasticsearch-4.1.2.jar;D:\MavenRepository\org\springframework\data\spring-data-commons\2.4.2\spring-data-commons-2.4.2.jar;D:\MavenRepository\org\elasticsearch\plugin\transport-netty4-client\7.9.3\transport-netty4-client-7.9.3.jar;D:\MavenRepository\io\netty\netty-codec-http\4.1.55.Final\netty-codec-http-4.1.55.Final.jar;D:\MavenRepository\org\elasticsearch\client\elasticsearch-rest-high-level-client\7.9.3\elasticsearch-rest-high-level-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch\7.9.3\elasticsearch-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-core\7.9.3\elasticsearch-core-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-secure-sm\7.9.3\elasticsearch-secure-sm-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-x-content\7.9.3\elasticsearch-x-content-7.9.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-smile\2.11.3\jackson-dataformat-smile-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-yaml\2.11.3\jackson-dataformat-yaml-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-cbor\2.11.3\jackson-dataformat-cbor-2.11.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-geo\7.9.3\elasticsearch-geo-7.9.3.jar;D:\MavenRepository\org\apache\lucene\lucene-core\8.6.2\lucene-core-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-analyzers-common\8.6.2\lucene-analyzers-common-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-backward-codecs\8.6.2\lucene-backward-codecs-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-grouping\8.6.2\lucene-grouping-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-highlighter\8.6.2\lucene-highlighter-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-join\8.6.2\lucene-join-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-memory\8.6.2\lucene-memory-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-misc\8.6.2\lucene-misc-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-queries\8.6.2\lucene-queries-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-queryparser\8.6.2\lucene-queryparser-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-sandbox\8.6.2\lucene-sandbox-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-spatial-extras\8.6.2\lucene-spatial-extras-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-spatial3d\8.6.2\lucene-spatial3d-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-suggest\8.6.2\lucene-suggest-8.6.2.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-cli\7.9.3\elasticsearch-cli-7.9.3.jar;D:\MavenRepository\net\sf\jopt-simple\jopt-simple\5.0.2\jopt-simple-5.0.2.jar;D:\MavenRepository\com\carrotsearch\hppc\0.8.1\hppc-0.8.1.jar;D:\MavenRepository\com\tdunning\t-digest\3.2\t-digest-3.2.jar;D:\MavenRepository\org\hdrhistogram\HdrHistogram\2.1.9\HdrHistogram-2.1.9.jar;D:\MavenRepository\org\elasticsearch\jna\5.5.0\jna-5.5.0.jar;D:\MavenRepository\org\elasticsearch\client\elasticsearch-rest-client\7.9.3\elasticsearch-rest-client-7.9.3.jar;D:\MavenRepository\org\apache\httpcomponents\httpasyncclient\4.1.4\httpasyncclient-4.1.4.jar;D:\MavenRepository\org\apache\httpcomponents\httpcore-nio\4.4.14\httpcore-nio-4.4.14.jar;D:\MavenRepository\org\elasticsearch\plugin\mapper-extras-client\7.9.3\mapper-extras-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\parent-join-client\7.9.3\parent-join-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\aggs-matrix-stats-client\7.9.3\aggs-matrix-stats-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\rank-eval-client\7.9.3\rank-eval-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\lang-mustache-client\7.9.3\lang-mustache-client-7.9.3.jar;D:\MavenRepository\com\github\spullara\mustache\java\compiler\0.9.6\compiler-0.9.6.jar;D:\MavenRepository\com\baomidou\dynamic-datasource-spring-boot-starter\3.2.1\dynamic-datasource-spring-boot-starter-3.2.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-jdbc\2.4.1\spring-boot-starter-jdbc-2.4.1.jar;D:\MavenRepository\com\zaxxer\HikariCP\3.4.5\HikariCP-3.4.5.jar;D:\MavenRepository\org\springframework\spring-jdbc\5.3.2\spring-jdbc-5.3.2.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-aop\2.4.1\spring-boot-starter-aop-2.4.1.jar;D:\MavenRepository\org\springframework\spring-aop\5.3.2\spring-aop-5.3.2.jar;D:\MavenRepository\org\aspectj\aspectjweaver\1.9.6\aspectjweaver-1.9.6.jar;D:\MavenRepository\org\json\json\20180813\json-20180813.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-web\2.4.1\spring-boot-starter-web-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-json\2.4.1\spring-boot-starter-json-2.4.1.jar;D:\MavenRepository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.11.3\jackson-datatype-jdk8-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.11.3\jackson-datatype-jsr310-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.11.3\jackson-module-parameter-names-2.11.3.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-tomcat\2.4.1\spring-boot-starter-tomcat-2.4.1.jar;D:\MavenRepository\org\apache\tomcat\embed\tomcat-embed-core\9.0.41\tomcat-embed-core-9.0.41.jar;D:\MavenRepository\org\glassfish\jakarta.el\3.0.3\jakarta.el-3.0.3.jar;D:\MavenRepository\org\apache\tomcat\embed\tomcat-embed-websocket\9.0.41\tomcat-embed-websocket-9.0.41.jar;D:\MavenRepository\org\springframework\spring-web\5.3.2\spring-web-5.3.2.jar;D:\MavenRepository\org\springframework\spring-beans\5.3.2\spring-beans-5.3.2.jar;D:\MavenRepository\org\springframework\spring-webmvc\5.3.2\spring-webmvc-5.3.2.jar;D:\MavenRepository\org\springframework\spring-context\5.3.2\spring-context-5.3.2.jar;D:\MavenRepository\org\springframework\spring-expression\5.3.2\spring-expression-5.3.2.jar;D:\MavenRepository\javax\servlet\jstl\1.2\jstl-1.2.jar;D:\MavenRepository\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;D:\MavenRepository\com\itextpdf\itextpdf\5.4.3\itextpdf-5.4.3.jar;D:\MavenRepository\org\apache\pdfbox\pdfbox\3.0.0-RC1\pdfbox-3.0.0-RC1.jar;D:\MavenRepository\org\apache\pdfbox\fontbox\3.0.0-RC1\fontbox-3.0.0-RC1.jar;D:\MavenRepository\com\github\jai-imageio\jai-imageio-jpeg2000\1.3.0\jai-imageio-jpeg2000-1.3.0.jar;D:\MavenRepository\com\github\jai-imageio\jai-imageio-core\1.3.0\jai-imageio-core-1.3.0.jar;D:\MavenRepository\commons-io\commons-io\2.6\commons-io-2.6.jar;D:\MavenRepository\commons-fileupload\commons-fileupload\1.2.1\commons-fileupload-1.2.1.jar;D:\MavenRepository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-freemarker\2.4.1\spring-boot-starter-freemarker-2.4.1.jar;D:\MavenRepository\org\freemarker\freemarker\2.3.30\freemarker-2.3.30.jar;D:\MavenRepository\org\springframework\spring-context-support\5.3.2\spring-context-support-5.3.2.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter\2.4.1\spring-boot-starter-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot\2.4.1\spring-boot-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-autoconfigure\2.4.1\spring-boot-autoconfigure-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-logging\2.4.1\spring-boot-starter-logging-2.4.1.jar;D:\MavenRepository\ch\qos\logback\logback-classic\1.2.3\logback-classic-1.2.3.jar;D:\MavenRepository\ch\qos\logback\logback-core\1.2.3\logback-core-1.2.3.jar;D:\MavenRepository\org\apache\logging\log4j\log4j-to-slf4j\2.13.3\log4j-to-slf4j-2.13.3.jar;D:\MavenRepository\org\apache\logging\log4j\log4j-api\2.13.3\log4j-api-2.13.3.jar;D:\MavenRepository\org\slf4j\jul-to-slf4j\1.7.30\jul-to-slf4j-1.7.30.jar;D:\MavenRepository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;D:\MavenRepository\org\springframework\spring-core\5.3.2\spring-core-5.3.2.jar;D:\MavenRepository\org\springframework\spring-jcl\5.3.2\spring-jcl-5.3.2.jar;D:\MavenRepository\org\yaml\snakeyaml\1.27\snakeyaml-1.27.jar;D:\MavenRepository\jakarta\xml\bind\jakarta.xml.bind-api\2.3.3\jakarta.xml.bind-api-2.3.3.jar;D:\MavenRepository\jakarta\activation\jakarta.activation-api\1.2.2\jakarta.activation-api-1.2.2.jar;D:\MavenRepository\net\bytebuddy\byte-buddy\1.10.18\byte-buddy-1.10.18.jar;D:\MavenRepository\com\baomidou\mybatis-plus-boot-starter\3.4.0\mybatis-plus-boot-starter-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus\3.4.0\mybatis-plus-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-extension\3.4.0\mybatis-plus-extension-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-core\3.4.0\mybatis-plus-core-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-annotation\3.4.0\mybatis-plus-annotation-3.4.0.jar;D:\MavenRepository\com\github\jsqlparser\jsqlparser\3.2\jsqlparser-3.2.jar;D:\MavenRepository\org\mybatis\mybatis\3.5.5\mybatis-3.5.5.jar;D:\MavenRepository\org\mybatis\mybatis-spring\2.0.5\mybatis-spring-2.0.5.jar;D:\MavenRepository\com\alibaba\druid-spring-boot-starter\1.1.17\druid-spring-boot-starter-1.1.17.jar;D:\MavenRepository\com\alibaba\druid\1.1.17\druid-1.1.17.jar;D:\MavenRepository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;D:\MavenRepository\org\projectlombok\lombok\1.18.16\lombok-1.18.16.jar;D:\MavenRepository\com\alibaba\fastjson\1.2.75\fastjson-1.2.75.jar;D:\MavenRepository\org\reflections\reflections\0.9.12\reflections-0.9.12.jar;D:\MavenRepository\org\javassist\javassist\3.26.0-GA\javassist-3.26.0-GA.jar;D:\MavenRepository\mysql\mysql-connector-java\5.1.47\mysql-connector-java-5.1.47.jar;D:\MavenRepository\org\quartz-scheduler\quartz\2.3.2\quartz-2.3.2.jar;D:\MavenRepository\com\mchange\mchange-commons-java\0.2.15\mchange-commons-java-0.2.15.jar;D:\MavenRepository\org\postgresql\postgresql\42.2.6\postgresql-42.2.6.jar;D:\MavenRepository\com\auth0\java-jwt\3.7.0\java-jwt-3.7.0.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-databind\2.11.3\jackson-databind-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-core\2.11.3\jackson-core-2.11.3.jar;D:\MavenRepository\org\apache\shiro\shiro-spring-boot-starter\1.4.0\shiro-spring-boot-starter-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-spring\1.4.0\shiro-spring-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-core\1.4.0\shiro-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-lang\1.4.0\shiro-lang-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-cache\1.4.0\shiro-cache-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-hash\1.4.0\shiro-crypto-hash-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-core\1.4.0\shiro-crypto-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-cipher\1.4.0\shiro-crypto-cipher-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-config-core\1.4.0\shiro-config-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-config-ogdl\1.4.0\shiro-config-ogdl-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-event\1.4.0\shiro-event-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-web\1.4.0\shiro-web-1.4.0.jar;D:\MavenRepository\org\crazycake\shiro-redis\3.1.0\shiro-redis-3.1.0.jar;D:\MavenRepository\redis\clients\jedis\3.3.0\jedis-3.3.0.jar;D:\MavenRepository\org\apache\commons\commons-pool2\2.9.0\commons-pool2-2.9.0.jar;D:\MavenRepository\com\puppycrawl\tools\checkstyle\8.3\checkstyle-8.3.jar;D:\MavenRepository\antlr\antlr\2.7.7\antlr-2.7.7.jar;D:\MavenRepository\org\antlr\antlr4-runtime\4.7\antlr4-runtime-4.7.jar;D:\MavenRepository\commons-cli\commons-cli\1.4\commons-cli-1.4.jar;D:\MavenRepository\com\google\guava\guava\23.0\guava-23.0.jar;D:\MavenRepository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\MavenRepository\com\google\errorprone\error_prone_annotations\2.0.18\error_prone_annotations-2.0.18.jar;D:\MavenRepository\com\google\j2objc\j2objc-annotations\1.1\j2objc-annotations-1.1.jar;D:\MavenRepository\org\codehaus\mojo\animal-sniffer-annotations\1.14\animal-sniffer-annotations-1.14.jar;D:\MavenRepository\net\sf\saxon\Saxon-HE\9.8.0-4\Saxon-HE-9.8.0-4.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-data-redis\2.4.1\spring-boot-starter-data-redis-2.4.1.jar;D:\MavenRepository\org\springframework\data\spring-data-redis\2.4.2\spring-data-redis-2.4.2.jar;D:\MavenRepository\org\springframework\data\spring-data-keyvalue\2.4.2\spring-data-keyvalue-2.4.2.jar;D:\MavenRepository\org\springframework\spring-tx\5.3.2\spring-tx-5.3.2.jar;D:\MavenRepository\org\springframework\spring-oxm\5.3.2\spring-oxm-5.3.2.jar;D:\MavenRepository\io\lettuce\lettuce-core\6.0.1.RELEASE\lettuce-core-6.0.1.RELEASE.jar;D:\MavenRepository\io\netty\netty-common\4.1.55.Final\netty-common-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-handler\4.1.55.Final\netty-handler-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-resolver\4.1.55.Final\netty-resolver-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-buffer\4.1.55.Final\netty-buffer-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-codec\4.1.55.Final\netty-codec-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-transport\4.1.55.Final\netty-transport-4.1.55.Final.jar;D:\MavenRepository\io\projectreactor\reactor-core\3.4.1\reactor-core-3.4.1.jar;D:\MavenRepository\org\reactivestreams\reactive-streams\1.0.3\reactive-streams-1.0.3.jar;D:\MavenRepository\io\springfox\springfox-swagger2\2.9.2\springfox-swagger2-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-spi\2.9.2\springfox-spi-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-core\2.9.2\springfox-core-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-schema\2.9.2\springfox-schema-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-swagger-common\2.9.2\springfox-swagger-common-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-spring-web\2.9.2\springfox-spring-web-2.9.2.jar;D:\MavenRepository\com\fasterxml\classmate\1.5.1\classmate-1.5.1.jar;D:\MavenRepository\org\springframework\plugin\spring-plugin-core\1.2.0.RELEASE\spring-plugin-core-1.2.0.RELEASE.jar;D:\MavenRepository\org\springframework\plugin\spring-plugin-metadata\1.2.0.RELEASE\spring-plugin-metadata-1.2.0.RELEASE.jar;D:\MavenRepository\org\mapstruct\mapstruct\1.2.0.Final\mapstruct-1.2.0.Final.jar;D:\MavenRepository\io\springfox\springfox-swagger-ui\2.9.2\springfox-swagger-ui-2.9.2.jar;D:\MavenRepository\com\github\xiaoymin\swagger-bootstrap-ui\1.9.3\swagger-bootstrap-ui-1.9.3.jar;D:\MavenRepository\io\springfox\springfox-bean-validators\2.9.2\springfox-bean-validators-2.9.2.jar;D:\MavenRepository\io\swagger\swagger-annotations\1.5.20\swagger-annotations-1.5.20.jar;D:\MavenRepository\io\swagger\swagger-models\1.5.20\swagger-models-1.5.20.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-annotations\2.11.3\jackson-annotations-2.11.3.jar;D:\MavenRepository\com\github\ulisesbocchio\jasypt-spring-boot-starter\3.0.3\jasypt-spring-boot-starter-3.0.3.jar;D:\MavenRepository\com\github\ulisesbocchio\jasypt-spring-boot\3.0.3\jasypt-spring-boot-3.0.3.jar;D:\MavenRepository\org\jasypt\jasypt\1.9.3\jasypt-1.9.3.jar;D:\MavenRepository\org\bouncycastle\bcprov-jdk15on\1.65\bcprov-jdk15on-1.65.jar;D:\MavenRepository\net\sf\json-lib\json-lib\2.4\json-lib-2.4-jdk15.jar;D:\MavenRepository\commons-beanutils\commons-beanutils\1.8.0\commons-beanutils-1.8.0.jar;D:\MavenRepository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;D:\MavenRepository\net\sf\ezmorph\ezmorph\1.0.6\ezmorph-1.0.6.jar;D:\MavenRepository\dom4j\dom4j\1.6.1\dom4j-1.6.1.jar;D:\MavenRepository\xml-apis\xml-apis\1.0.b2\xml-apis-1.0.b2.jar;D:\MavenRepository\com\thoughtworks\xstream\xstream\1.4.3\xstream-1.4.3.jar;D:\MavenRepository\xmlpull\xmlpull\1.1.3.1\xmlpull-1.1.3.1.jar;D:\MavenRepository\xpp3\xpp3_min\1.1.4c\xpp3_min-1.1.4c.jar;D:\MavenRepository\commons-codec\commons-codec\1.10\commons-codec-1.10.jar;D:\MavenRepository\cn\afterturn\easypoi-spring-boot-starter\3.3.0\easypoi-spring-boot-starter-3.3.0.jar;D:\MavenRepository\cn\afterturn\easypoi-web\3.3.0\easypoi-web-3.3.0.jar;D:\MavenRepository\cn\afterturn\easypoi-base\3.3.0\easypoi-base-3.3.0.jar;D:\MavenRepository\org\apache\commons\commons-lang3\3.11\commons-lang3-3.11.jar;D:\MavenRepository\ognl\ognl\3.2.6\ognl-3.2.6.jar;D:\MavenRepository\cn\afterturn\easypoi-annotation\3.3.0\easypoi-annotation-3.3.0.jar;D:\MavenRepository\org\apache\poi\poi-scratchpad\3.15\poi-scratchpad-3.15.jar;D:\MavenRepository\org\apache\poi\ooxml-schemas\1.3\ooxml-schemas-1.3.jar;D:\MavenRepository\org\apache\xmlbeans\xmlbeans\2.3.0\xmlbeans-2.3.0.jar;D:\MavenRepository\stax\stax-api\1.0.1\stax-api-1.0.1.jar;D:\MavenRepository\org\apache\httpcomponents\httpclient\4.5.3\httpclient-4.5.3.jar;D:\MavenRepository\org\apache\httpcomponents\httpcore\4.4.14\httpcore-4.4.14.jar;D:\MavenRepository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;D:\MavenRepository\commons-logging\commons-logging\1.0.4\commons-logging-1.0.4.jar;D:\MavenRepository\org\hibernate\validator\hibernate-validator\6.0.10.Final\hibernate-validator-6.0.10.Final.jar;D:\MavenRepository\javax\validation\validation-api\2.0.1.Final\validation-api-2.0.1.Final.jar;D:\MavenRepository\org\jboss\logging\jboss-logging\3.4.1.Final\jboss-logging-3.4.1.Final.jar;D:\MavenRepository\cn\hutool\hutool-all\5.6.1\hutool-all-5.6.1.jar;D:\MavenRepository\commons-net\commons-net\3.3\commons-net-3.3.jar;D:\MavenRepository\e-iceblue\spire.office\4.8.0\spire.office-4.8.0.jar;D:\MavenRepository\com\google\zxing\core\3.3.0\core-3.3.0.jar;D:\MavenRepository\com\google\zxing\javase\3.3.0\javase-3.3.0.jar;D:\MavenRepository\com\beust\jcommander\1.48\jcommander-1.48.jar com.lingxu.module.Datax
10:07:29.729 [main] INFO com.alibaba.datax.common.statistics.VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
10:07:29.736 [main] INFO com.alibaba.datax.core.Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.91-b15
jvmInfo: Windows 10 amd64 10.0
cpu num: 16
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 1,166.00MB | 56.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 9.00MB | 9.00MB
PS Old Gen | 2,368.00MB | 148.00MB
Metaspace | -0.00MB | 0.00MB
10:07:29.753 [main] INFO com.alibaba.datax.core.Engine -
{
"content":[
{
"reader":{
"parameter":{
"password":"****",
"connection":[
{
"querySql":[
"select t.id,t.name,t.status from users t"
],
"jdbcUrl":[
"jdbc:mysql://127.0.0.1:3306/test"
]
}
],
"username":"root"
},
"name":"mysqlreader"
},
"writer":{
"parameter":{
"password":"****",
"column":[
"id",
"name",
"status"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test",
"table":[
"temp_users"
]
}
],
"writeMode":"insert",
"username":"root"
},
"name":"mysqlwriter"
}
}
],
"setting":{
"speed":{
"channel":4
}
}
}
10:07:29.754 [main] DEBUG com.alibaba.datax.core.Engine - {"core":{"container":{"trace":{"enable":"false"},"job":{"reportInterval":10000,"id":-1},"taskGroup":{"channel":5}},"dataXServer":{"reportDataxLog":false,"address":"http://localhost:7001/api","timeout":10000,"reportPerfLog":false},"transport":{"exchanger":{"class":"com.alibaba.datax.core.plugin.BufferedRecordExchanger","bufferSize":32},"channel":{"byteCapacity":67108864,"flowControlInterval":20,"class":"com.alibaba.datax.core.transport.channel.memory.MemoryChannel","speed":{"byte":-1,"record":-1},"capacity":512}},"statistics":{"collector":{"plugin":{"taskClass":"com.alibaba.datax.core.statistics.plugin.task.StdoutPluginCollector","maxDirtyNumber":10}}}},"entry":{"jvm":"-Xms1G -Xmx1G"},"common":{"column":{"dateFormat":"yyyy-MM-dd","datetimeFormat":"yyyy-MM-dd HH:mm:ss","timeFormat":"HH:mm:ss","extraFormats":["yyyyMMdd"],"timeZone":"GMT+8","encoding":"utf-8"}},"plugin":{"reader":{"mysqlreader":{"path":"E:\\datax\\plugin\\reader\\mysqlreader","name":"mysqlreader","description":"useScene: prod. mechanism: Jdbc connection using the database, execute select sql, retrieve data from the ResultSet. warn: The more you know about the database, the less problems you encounter.","developer":"alibaba","class":"com.alibaba.datax.plugin.reader.mysqlreader.MysqlReader"}},"writer":{"mysqlwriter":{"path":"E:\\datax\\plugin\\writer\\mysqlwriter","name":"mysqlwriter","description":"useScene: prod. mechanism: Jdbc connection using the database, execute insert sql. warn: The more you know about the database, the less problems you encounter.","developer":"alibaba","class":"com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter"}}},"job":{"content":[{"reader":{"parameter":{"password":"密码","connection":[{"querySql":["select t.id,t.name,t.status from users t"],"jdbcUrl":["jdbc:mysql://127.0.0.1:3306/test"]}],"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"connection":[{"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test","table":["temp_users"]}],"writeMode":"insert","username":"root"},"name":"mysqlwriter"}}],"setting":{"speed":{"channel":4}}}}
10:07:29.806 [main] WARN com.alibaba.datax.core.Engine - prioriy set to 0, because NumberFormatException, the value is: null
10:07:29.807 [main] INFO com.alibaba.datax.common.statistics.PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
10:07:29.808 [main] INFO com.alibaba.datax.core.job.JobContainer - DataX jobContainer starts job.
10:07:29.808 [main] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do preHandle ...
10:07:29.808 [main] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do init ...
10:07:29.808 [main] INFO com.alibaba.datax.core.job.JobContainer - Set jobId = 0
10:07:30.106 [job-0] INFO com.alibaba.datax.plugin.rdbms.reader.util.OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
10:07:30.108 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Job - After job init(), job config now is:[
{"password":"密码","isTableMode":false,"fetchSize":-2147483648,"connection":[{"querySql":["select t.id,t.name,t.status from users t"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true"}],"tableNumber":0,"username":"root"}
]
10:07:30.133 [job-0] INFO com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil - table:[temp_users] all columns:[
id,name,status
].
10:07:30.138 [job-0] INFO com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,name,status) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
10:07:30.138 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job - After job init(), originalConfig now is:[
{"password":"密码","column":["id","name","status"],"connection":[{"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","table":["temp_users"]}],"insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"username":"root"}
]
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do prepare ...
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
10:07:30.138 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job - After job prepare(), originalConfig now is:[
{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"}
]
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do split ...
10:07:30.139 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Job set Channel-Number to 4 channels.
10:07:30.140 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
10:07:30.140 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
10:07:30.140 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - transformer configuration: null
10:07:30.151 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - contentConfig configuration: [{"internal":{"reader":{"parameter":{"querySql":"select t.id,t.name,t.status from users t","password":"密码","isTableMode":false,"fetchSize":-2147483648,"loadBalanceResourceMark":"127.0.0.1","jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","tableNumber":0,"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"},"name":"mysqlwriter"},"taskId":0},"keys":["writer.parameter.column[1]","writer.parameter.tableNumber","writer.parameter.column[2]","writer.parameter.column[0]","writer.parameter.jdbcUrl","writer.parameter.writeMode","reader.parameter.fetchSize","reader.parameter.jdbcUrl","writer.parameter.insertOrReplaceTemplate","reader.parameter.password","writer.name","reader.parameter.querySql","reader.parameter.loadBalanceResourceMark","writer.parameter.password","writer.parameter.batchSize","writer.parameter.table","reader.parameter.tableNumber","reader.name","reader.parameter.isTableMode","reader.parameter.username","taskId","writer.parameter.username"],"secretKeyPathSet":[]}]
10:07:30.151 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do schedule ...
10:07:30.153 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Scheduler starts [1] taskGroups.
10:07:30.154 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Running by standalone Mode.
10:07:30.157 [taskGroup-0] DEBUG com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0]'s task configs[[{"internal":{"reader":{"parameter":{"querySql":"select t.id,t.name,t.status from users t","password":"密码","isTableMode":false,"fetchSize":-2147483648,"loadBalanceResourceMark":"127.0.0.1","jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","tableNumber":0,"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"},"name":"mysqlwriter"},"taskId":0},"keys":["writer.parameter.column[1]","writer.parameter.tableNumber","writer.parameter.column[2]","writer.parameter.column[0]","writer.parameter.jdbcUrl","writer.parameter.writeMode","reader.parameter.fetchSize","reader.parameter.jdbcUrl","writer.parameter.insertOrReplaceTemplate","reader.parameter.password","writer.name","reader.parameter.querySql","reader.parameter.loadBalanceResourceMark","writer.parameter.password","writer.parameter.batchSize","writer.parameter.table","reader.parameter.tableNumber","reader.name","reader.parameter.isTableMode","reader.parameter.username","taskId","writer.parameter.username"],"secretKeyPathSet":[]}]]
10:07:30.157 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
10:07:30.159 [taskGroup-0] INFO com.alibaba.datax.core.transport.channel.Channel - Channel set byte_speed_limit to -1, No bps activated.
10:07:30.159 [taskGroup-0] INFO com.alibaba.datax.core.transport.channel.Channel - Channel set record_speed_limit to -1, No tps activated.
10:07:30.165 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
10:07:30.165 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do init ...
10:07:30.165 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do init ...
10:07:30.166 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do prepare ...
10:07:30.167 [job-0] DEBUG com.alibaba.datax.core.job.scheduler.AbstractScheduler - com.alibaba.datax.core.statistics.communication.Communication@27ce24aa[
counter={}
message={}
state=RUNNING
throwable=<null>
timestamp=1663639650157
]
10:07:30.167 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do prepare ...
10:07:30.167 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to read ...
10:07:30.167 [0-0-0-reader] INFO com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Task - Begin to read record by Sql: [select t.id,t.name,t.status from users t
] jdbcUrl:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
10:07:30.170 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to write ...
10:07:30.174 [0-0-0-reader] INFO com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Task - Finished read record by Sql: [select t.id,t.name,t.status from users t
] jdbcUrl:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
10:07:30.177 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do post ...
10:07:30.177 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do destroy ...
10:07:30.395 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do post ...
10:07:30.395 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do destroy ...
10:07:30.493 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[329]ms
10:07:30.493 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] completed it's tasks.
10:07:40.179 [job-0] DEBUG com.alibaba.datax.core.job.scheduler.AbstractScheduler - com.alibaba.datax.core.statistics.communication.Communication@188715b5[
counter={writeSucceedRecords=3, readSucceedRecords=2, totalErrorBytes=0, writeSucceedBytes=8, byteSpeed=0, totalErrorRecords=0, recordSpeed=0, waitReaderTime=0, writeReceivedBytes=8, stage=1, waitWriterTime=37000, percentage=1.0, totalReadRecords=2, writeReceivedRecords=3, readSucceedBytes=8, totalReadBytes=8}
message={}
state=SUCCEEDED
throwable=<null>
timestamp=1663639660179
]
10:07:40.180 [job-0] INFO com.alibaba.datax.core.statistics.container.communicator.job.StandAloneJobContainerCommunicator - Total 2 records, 8 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
10:07:40.180 [job-0] INFO com.alibaba.datax.core.job.scheduler.AbstractScheduler - Scheduler accomplished all tasks.
10:07:40.180 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do post ...
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] do post work.
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] do post work.
10:07:40.181 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do postHandle ...
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX jobId [0] completed successfully.
10:07:40.181 [job-0] INFO com.alibaba.datax.core.container.util.HookInvoker - No hook invoked, because base dir not exists or is a file: E:\datax\hook
10:07:40.182 [job-0] INFO com.alibaba.datax.core.job.JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 1 | 1 | 1 | 0.004s | 0.004s | 0.004s
10:07:40.182 [job-0] INFO com.alibaba.datax.core.job.JobContainer - PerfTrace not enable!
10:07:40.182 [job-0] INFO com.alibaba.datax.core.statistics.container.communicator.job.StandAloneJobContainerCommunicator - Total 2 records, 8 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
10:07:40.183 [job-0] INFO com.alibaba.datax.core.job.JobContainer -
任务启动时刻 : 2022-09-20 10:07:29
任务结束时刻 : 2022-09-20 10:07:40
任务总计耗时 : 10s
任务平均流量 : 0B/s
记录写入速度 : 0rec/s
读出记录总数 : 2
读写失败总数 : 0
Process finished with exit code 0
4、数据库中查看
可以看到数据库中temp_users中已经成功的写入了users中的数据,已经成功入库