java集成datax

最新推荐文章于 2024-06-19 16:34:32 发布
最新推荐文章于 2024-06-19 16:34:32 发布
阅读量3k 收藏 13
点赞数 1
分类专栏： datax java 文章标签： java 开发语言
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43987718/article/details/126947706
版权
java 同时被 2 个专栏收录
56 篇文章 2 订阅
订阅专栏
datax
1 篇文章 0 订阅
订阅专栏

java集成datax

1、需求

公司搭建自己的数据平台，datax作为数据采集工具还是非常强有力的，但是官方是通过python调取datax.py文件实现数据同步的，没法子很好的集成在自己的数据平台上。

2、获取方式

1、datax 源代码下载

datax 源代码下载

通过源代码的方式进行打包

由于pom文件内容太多，就只是加载了测试用的reader和writer

1、reader

在这里插入图片描述

2、writer

在这里插入图片描述

3、pom文件：
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.alibaba.datax</groupId>
    <artifactId>datax-all</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-core</artifactId>
            <version>1.3</version>
        </dependency>
    </dependencies>

    <name>datax-all</name>
    <packaging>pom</packaging>

    <properties>
        <jdk-version>1.8</jdk-version>
        <datax-project-version>0.0.1-SNAPSHOT</datax-project-version>
        <commons-lang3-version>3.3.2</commons-lang3-version>
        <commons-configuration-version>1.10</commons-configuration-version>
        <commons-cli-version>1.2</commons-cli-version>
        <fastjson-version>1.2.49</fastjson-version>
        <guava-version>16.0.1</guava-version>
        <diamond.version>3.7.2.1-SNAPSHOT</diamond.version>

        <!--slf4j 1.7.10 和 logback-classic 1.0.13 是好基友 -->
        <slf4j-api-version>1.7.10</slf4j-api-version>
        <logback-classic-version>1.0.13</logback-classic-version>
        <commons-io-version>2.4</commons-io-version>
        <junit-version>4.13.1</junit-version>
        <tddl.version>5.1.22-1</tddl.version>
        <swift-version>1.0.0</swift-version>

        <project-sourceEncoding>UTF-8</project-sourceEncoding>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.encoding>UTF-8</maven.compiler.encoding>
        <mysql.driver.version>5.1.47</mysql.driver.version>
    </properties>

    <modules>
        <module>common</module>
        <module>core</module>
        <module>transformer</module>

        <!-- reader -->
        <module>mysqlreader</module>
<!--        <module>drdsreader</module>-->
<!--        <module>sqlserverreader</module>-->
<!--        <module>postgresqlreader</module>-->
<!--        <module>kingbaseesreader</module>-->
<!--        <module>oraclereader</module>-->
<!--        <module>cassandrareader</module>-->
<!--        <module>oceanbasev10reader</module>-->
<!--        <module>rdbmsreader</module>-->

<!--        <module>odpsreader</module>-->
<!--        <module>otsreader</module>-->
<!--        <module>otsstreamreader</module>-->
<!--        <module>hbase11xreader</module>-->
<!--        <module>hbase094xreader</module>-->
<!--        <module>hbase11xsqlreader</module>-->
<!--        <module>hbase20xsqlreader</module>-->

<!--        <module>ossreader</module>-->
<!--        <module>hdfsreader</module>-->
<!--        <module>ftpreader</module>-->
<!--        <module>txtfilereader</module>-->
<!--        <module>streamreader</module>-->

<!--        <module>mongodbreader</module>-->
<!--        <module>tdenginereader</module>-->
<!--        <module>gdbreader</module>-->
<!--        <module>tsdbreader</module>-->
<!--        <module>opentsdbreader</module>-->


        <!-- writer -->
        <module>mysqlwriter</module>
<!--        <module>starrockswriter</module>-->
<!--        <module>drdswriter</module>-->
<!--        <module>oraclewriter</module>-->
<!--        <module>sqlserverwriter</module>-->
<!--        <module>postgresqlwriter</module>-->
<!--        <module>kingbaseeswriter</module>-->
<!--        <module>adswriter</module>-->
<!--        <module>oceanbasev10writer</module>-->
<!--        <module>adbpgwriter</module>-->
<!--        <module>hologresjdbcwriter</module>-->
<!--        <module>rdbmswriter</module>-->


<!--        <module>odpswriter</module>-->
<!--        <module>osswriter</module>-->
<!--        <module>otswriter</module>-->
<!--        <module>hbase11xwriter</module>-->
<!--        <module>hbase094xwriter</module>-->
<!--        <module>hbase11xsqlwriter</module>-->
<!--        <module>hbase20xsqlwriter</module>-->
<!--        <module>kuduwriter</module>-->
<!--        <module>ftpwriter</module>-->
<!--        <module>hdfswriter</module>-->
<!--        <module>txtfilewriter</module>-->
<!--        <module>streamwriter</module>-->

<!--        <module>elasticsearchwriter</module>-->
<!--        <module>mongodbwriter</module>-->
<!--        <module>tdenginewriter</module>-->
<!--        <module>ocswriter</module>-->
<!--        <module>tsdbwriter</module>-->
<!--        <module>gdbwriter</module>-->
<!--        <module>oscarwriter</module>-->
<!--        <module>cassandrawriter</module>-->
<!--        <module>clickhousewriter</module>-->

        <!-- common support module -->
        <module>plugin-rdbms-util</module>
        <module>plugin-unstructured-storage-util</module>
    </modules>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.apache.commons</groupId>
                <artifactId>commons-lang3</artifactId>
                <version>${commons-lang3-version}</version>
            </dependency>
            <dependency>
                <groupId>com.alibaba</groupId>
                <artifactId>fastjson</artifactId>
                <version>${fastjson-version}</version>
            </dependency>
            <!--<dependency>
                <groupId>com.google.guava</groupId>
                <artifactId>guava</artifactId>
                <version>${guava-version}</version>
            </dependency>-->
            <dependency>
                <groupId>commons-io</groupId>
                <artifactId>commons-io</artifactId>
                <version>${commons-io-version}</version>
            </dependency>
            <dependency>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
                <version>${slf4j-api-version}</version>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>${logback-classic-version}</version>
            </dependency>

            <dependency>
                <groupId>com.taobao.tddl</groupId>
                <artifactId>tddl-client</artifactId>
                <version>${tddl.version}</version>
                <exclusions>
                    <exclusion>
                        <groupId>com.google.guava</groupId>
                        <artifactId>guava</artifactId>
                    </exclusion>
                    <exclusion>
                        <groupId>com.taobao.diamond</groupId>
                        <artifactId>diamond-client</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>

            <dependency>
                <groupId>com.taobao.diamond</groupId>
                <artifactId>diamond-client</artifactId>
                <version>${diamond.version}</version>
            </dependency>

            <dependency>
                <groupId>com.alibaba.search.swift</groupId>
                <artifactId>swift_client</artifactId>
                <version>${swift-version}</version>
            </dependency>

            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>${junit-version}</version>
            </dependency>

            <dependency>
                <groupId>org.mockito</groupId>
                <artifactId>mockito-all</artifactId>
                <version>1.9.5</version>
                <scope>test</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-api</artifactId>
                <version>2.17.1</version>
            </dependency>

            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-core</artifactId>
                <version>2.17.1</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <repositories>
        <repository>
            <id>central</id>
            <name>Nexus aliyun</name>
            <url>https://maven.aliyun.com/repository/central</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>spring</id>
            <name>spring</name>
            <url>https://maven.aliyun.com/repository/spring</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>central</id>
            <name>Nexus aliyun</name>
            <url>https://maven.aliyun.com/repository/central</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <finalName>datax</finalName>
                    <descriptors>
                        <descriptor>package.xml</descriptor>
                    </descriptors>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>${jdk-version}</source>
                    <target>${jdk-version}</target>
                    <encoding>${project-sourceEncoding}</encoding>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
4、打包方式

在这里插入图片描述

5、找到打包出来的datax文件

在这里插入图片描述

2、datax 压缩包下载

datax压缩包下载

下载下来的压缩包，解压如下所示：

在这里插入图片描述

3、在pom文件中，加入相关依赖

        <!--        datax-->
        <dependency>
            <groupId>com.datax</groupId>
            <artifactId>datax-core</artifactId>
            <version>0.0.1</version>
            <systemPath>${project.basedir}/lib/datax-core-0.0.1-SNAPSHOT.jar</systemPath>
        </dependency>

        <dependency>
        <groupId>com.datax</groupId>
        <artifactId>datax-common</artifactId>
        <version>0.0.1</version>
        <systemPath>${project.basedir}/lib/datax-common-0.0.1-SNAPSHOT.jar</systemPath>
        </dependency>

4、引入相应的jar包

在这里插入图片描述

5、编写相应的job测试json文件

在这里插入图片描述

job.json文件内容：

{
  "job": {
    "setting": {
      "speed": {
        "channel": 4
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "root",
            "password": "密码",
            "connection": [
              {
                "jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/test"],
                "querySql": ["select t.id,t.name,t.status from users t"]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "username": "root",
            "password": "密码",
            "writeMode": "insert",
            "column": ["id","name","status"],
            "connection": [
              {
                "table": [
                  "temp_users"
                ],
                "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/test"
              }
            ]
          }
        }
      }
    ]
  }
}

6、编写main方法进行测试

1、获取启动类

通过上面的方法你已经将dataX 集成到你的系统了，但是你如何启动dataX呢，在com.alibaba.datax.core.Engine类中有个main(String[])和entry(String[])方法，调用启动。启动代码如下：

/**
 * datax
 *
 * @author LZH
 * @version 1.0
 * @date 2022/09/19 16:16:13
 */
public class Datax {
    public static void main(String[] args) throws Throwable {
        System.setProperty("datax.home", "E:\\datax");
        String[] datxArgs = {"-job", getCurrentClasspath()+"/datax/job.json", "-mode", "standalone", "-jobid", "-1"};
        try {
            Engine.entry(datxArgs);
        }catch (Exception e){
            e.printStackTrace();
        }
    }
    //获取resource的路径
    public static String getCurrentClasspath(){
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        String currentClasspath = classLoader.getResource("").getPath();
        // 当前操作系统
        String osName = System.getProperty("os.name");
        if (osName.startsWith("Win")) {
            // 删除path中最前面的/
            currentClasspath = currentClasspath.substring(1, currentClasspath.length()-1);
        }
        return currentClasspath;
    }
 }
2、参数说明：

datax.home中存放的 是datax工具的路径，

getCurrentClasspath()+“/datax/job.json” 是配置同步数据库json文件的路径，

System.setProperty(“datax.home”, “E:\datax”); 设置datax.home环境变量，在系统启动时会通过datax.home环境变量获取配置文件和read、writer数据量插件。—>获取启动

String[] datxArgs = {“-job”, getCurrentClasspath()+“/datax/job.json”, “-mode”, “standalone”, “-jobid”, “-1”};定义构造参数数组，在系统启动时会用到。
通过上面的方法就可以通过Java的方式启动DataX

3、执行结果
D:\Java\jdk1.8.0_91\bin\java.exe "-javaagent:D:\IntelliJ IDEA 2021.1.1\lib\idea_rt.jar=1889:D:\IntelliJ IDEA 2021.1.1\bin" -Dfile.encoding=UTF-8 -classpath D:\Java\jdk1.8.0_91\jre\lib\charsets.jar;D:\Java\jdk1.8.0_91\jre\lib\deploy.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\cldrdata.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\dnsns.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\jaccess.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\jfxrt.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\localedata.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\nashorn.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunec.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\sunpkcs11.jar;D:\Java\jdk1.8.0_91\jre\lib\ext\zipfs.jar;D:\Java\jdk1.8.0_91\jre\lib\javaws.jar;D:\Java\jdk1.8.0_91\jre\lib\jce.jar;D:\Java\jdk1.8.0_91\jre\lib\jfr.jar;D:\Java\jdk1.8.0_91\jre\lib\jfxswt.jar;D:\Java\jdk1.8.0_91\jre\lib\jsse.jar;D:\Java\jdk1.8.0_91\jre\lib\management-agent.jar;D:\Java\jdk1.8.0_91\jre\lib\plugin.jar;D:\Java\jdk1.8.0_91\jre\lib\resources.jar;D:\Java\jdk1.8.0_91\jre\lib\rt.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\target\classes;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-common-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-core-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\datax-transformer-0.0.1-SNAPSHOT.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\commons-cli-1.2.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\aspose-words-15.8.0-jdk16.jar;D:\IdeaProject\ycdsj\lx-parent\lx-module\lib\ax-aes-api-1.0.0.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\target\classes;D:\MavenRepository\com\belerweb\pinyin4j\2.5.1\pinyin4j-2.5.1.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\lib\kingbase8-8.2.0.jar;D:\IdeaProject\ycdsj\lx-parent\lx-base\lib\DmJdbcDriver18.jar;D:\MavenRepository\com\alibaba\easyexcel\2.2.7\easyexcel-2.2.7.jar;D:\MavenRepository\org\apache\poi\poi\3.17\poi-3.17.jar;D:\MavenRepository\org\apache\commons\commons-collections4\4.1\commons-collections4-4.1.jar;D:\MavenRepository\org\apache\poi\poi-ooxml\3.17\poi-ooxml-3.17.jar;D:\MavenRepository\com\github\virtuald\curvesapi\1.04\curvesapi-1.04.jar;D:\MavenRepository\org\apache\poi\poi-ooxml-schemas\3.17\poi-ooxml-schemas-3.17.jar;D:\MavenRepository\cglib\cglib\3.1\cglib-3.1.jar;D:\MavenRepository\org\ow2\asm\asm\4.2\asm-4.2.jar;D:\MavenRepository\org\ehcache\ehcache\3.9.0\ehcache-3.9.0.jar;D:\MavenRepository\org\glassfish\jaxb\jaxb-runtime\2.3.3\jaxb-runtime-2.3.3.jar;D:\MavenRepository\org\glassfish\jaxb\txw2\2.3.3\txw2-2.3.3.jar;D:\MavenRepository\com\sun\istack\istack-commons-runtime\3.0.11\istack-commons-runtime-3.0.11.jar;D:\MavenRepository\com\sun\activation\jakarta.activation\1.2.2\jakarta.activation-1.2.2.jar;D:\MavenRepository\net\lingala\zip4j\zip4j\1.3.2\zip4j-1.3.2.jar;D:\MavenRepository\ma\glasnost\orika\orika-core\1.5.4\orika-core-1.5.4.jar;D:\MavenRepository\com\thoughtworks\paranamer\paranamer\2.8\paranamer-2.8.jar;D:\MavenRepository\com\carrotsearch\java-sizeof\0.0.5\java-sizeof-0.0.5.jar;D:\MavenRepository\org\codehaus\janino\janino\3.1.2\janino-3.1.2.jar;D:\MavenRepository\org\codehaus\janino\commons-compiler\3.1.2\commons-compiler-3.1.2.jar;D:\MavenRepository\joda-time\joda-time\2.3\joda-time-2.3.jar;D:\IdeaProject\ycdsj\lx-parent\lx-plugs\target\classes;D:\MavenRepository\org\springframework\boot\spring-boot-starter-data-elasticsearch\2.4.1\spring-boot-starter-data-elasticsearch-2.4.1.jar;D:\MavenRepository\org\springframework\data\spring-data-elasticsearch\4.1.2\spring-data-elasticsearch-4.1.2.jar;D:\MavenRepository\org\springframework\data\spring-data-commons\2.4.2\spring-data-commons-2.4.2.jar;D:\MavenRepository\org\elasticsearch\plugin\transport-netty4-client\7.9.3\transport-netty4-client-7.9.3.jar;D:\MavenRepository\io\netty\netty-codec-http\4.1.55.Final\netty-codec-http-4.1.55.Final.jar;D:\MavenRepository\org\elasticsearch\client\elasticsearch-rest-high-level-client\7.9.3\elasticsearch-rest-high-level-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch\7.9.3\elasticsearch-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-core\7.9.3\elasticsearch-core-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-secure-sm\7.9.3\elasticsearch-secure-sm-7.9.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-x-content\7.9.3\elasticsearch-x-content-7.9.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-smile\2.11.3\jackson-dataformat-smile-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-yaml\2.11.3\jackson-dataformat-yaml-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\dataformat\jackson-dataformat-cbor\2.11.3\jackson-dataformat-cbor-2.11.3.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-geo\7.9.3\elasticsearch-geo-7.9.3.jar;D:\MavenRepository\org\apache\lucene\lucene-core\8.6.2\lucene-core-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-analyzers-common\8.6.2\lucene-analyzers-common-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-backward-codecs\8.6.2\lucene-backward-codecs-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-grouping\8.6.2\lucene-grouping-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-highlighter\8.6.2\lucene-highlighter-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-join\8.6.2\lucene-join-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-memory\8.6.2\lucene-memory-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-misc\8.6.2\lucene-misc-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-queries\8.6.2\lucene-queries-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-queryparser\8.6.2\lucene-queryparser-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-sandbox\8.6.2\lucene-sandbox-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-spatial-extras\8.6.2\lucene-spatial-extras-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-spatial3d\8.6.2\lucene-spatial3d-8.6.2.jar;D:\MavenRepository\org\apache\lucene\lucene-suggest\8.6.2\lucene-suggest-8.6.2.jar;D:\MavenRepository\org\elasticsearch\elasticsearch-cli\7.9.3\elasticsearch-cli-7.9.3.jar;D:\MavenRepository\net\sf\jopt-simple\jopt-simple\5.0.2\jopt-simple-5.0.2.jar;D:\MavenRepository\com\carrotsearch\hppc\0.8.1\hppc-0.8.1.jar;D:\MavenRepository\com\tdunning\t-digest\3.2\t-digest-3.2.jar;D:\MavenRepository\org\hdrhistogram\HdrHistogram\2.1.9\HdrHistogram-2.1.9.jar;D:\MavenRepository\org\elasticsearch\jna\5.5.0\jna-5.5.0.jar;D:\MavenRepository\org\elasticsearch\client\elasticsearch-rest-client\7.9.3\elasticsearch-rest-client-7.9.3.jar;D:\MavenRepository\org\apache\httpcomponents\httpasyncclient\4.1.4\httpasyncclient-4.1.4.jar;D:\MavenRepository\org\apache\httpcomponents\httpcore-nio\4.4.14\httpcore-nio-4.4.14.jar;D:\MavenRepository\org\elasticsearch\plugin\mapper-extras-client\7.9.3\mapper-extras-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\parent-join-client\7.9.3\parent-join-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\aggs-matrix-stats-client\7.9.3\aggs-matrix-stats-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\rank-eval-client\7.9.3\rank-eval-client-7.9.3.jar;D:\MavenRepository\org\elasticsearch\plugin\lang-mustache-client\7.9.3\lang-mustache-client-7.9.3.jar;D:\MavenRepository\com\github\spullara\mustache\java\compiler\0.9.6\compiler-0.9.6.jar;D:\MavenRepository\com\baomidou\dynamic-datasource-spring-boot-starter\3.2.1\dynamic-datasource-spring-boot-starter-3.2.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-jdbc\2.4.1\spring-boot-starter-jdbc-2.4.1.jar;D:\MavenRepository\com\zaxxer\HikariCP\3.4.5\HikariCP-3.4.5.jar;D:\MavenRepository\org\springframework\spring-jdbc\5.3.2\spring-jdbc-5.3.2.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-aop\2.4.1\spring-boot-starter-aop-2.4.1.jar;D:\MavenRepository\org\springframework\spring-aop\5.3.2\spring-aop-5.3.2.jar;D:\MavenRepository\org\aspectj\aspectjweaver\1.9.6\aspectjweaver-1.9.6.jar;D:\MavenRepository\org\json\json\20180813\json-20180813.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-web\2.4.1\spring-boot-starter-web-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-json\2.4.1\spring-boot-starter-json-2.4.1.jar;D:\MavenRepository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.11.3\jackson-datatype-jdk8-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.11.3\jackson-datatype-jsr310-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.11.3\jackson-module-parameter-names-2.11.3.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-tomcat\2.4.1\spring-boot-starter-tomcat-2.4.1.jar;D:\MavenRepository\org\apache\tomcat\embed\tomcat-embed-core\9.0.41\tomcat-embed-core-9.0.41.jar;D:\MavenRepository\org\glassfish\jakarta.el\3.0.3\jakarta.el-3.0.3.jar;D:\MavenRepository\org\apache\tomcat\embed\tomcat-embed-websocket\9.0.41\tomcat-embed-websocket-9.0.41.jar;D:\MavenRepository\org\springframework\spring-web\5.3.2\spring-web-5.3.2.jar;D:\MavenRepository\org\springframework\spring-beans\5.3.2\spring-beans-5.3.2.jar;D:\MavenRepository\org\springframework\spring-webmvc\5.3.2\spring-webmvc-5.3.2.jar;D:\MavenRepository\org\springframework\spring-context\5.3.2\spring-context-5.3.2.jar;D:\MavenRepository\org\springframework\spring-expression\5.3.2\spring-expression-5.3.2.jar;D:\MavenRepository\javax\servlet\jstl\1.2\jstl-1.2.jar;D:\MavenRepository\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;D:\MavenRepository\com\itextpdf\itextpdf\5.4.3\itextpdf-5.4.3.jar;D:\MavenRepository\org\apache\pdfbox\pdfbox\3.0.0-RC1\pdfbox-3.0.0-RC1.jar;D:\MavenRepository\org\apache\pdfbox\fontbox\3.0.0-RC1\fontbox-3.0.0-RC1.jar;D:\MavenRepository\com\github\jai-imageio\jai-imageio-jpeg2000\1.3.0\jai-imageio-jpeg2000-1.3.0.jar;D:\MavenRepository\com\github\jai-imageio\jai-imageio-core\1.3.0\jai-imageio-core-1.3.0.jar;D:\MavenRepository\commons-io\commons-io\2.6\commons-io-2.6.jar;D:\MavenRepository\commons-fileupload\commons-fileupload\1.2.1\commons-fileupload-1.2.1.jar;D:\MavenRepository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-freemarker\2.4.1\spring-boot-starter-freemarker-2.4.1.jar;D:\MavenRepository\org\freemarker\freemarker\2.3.30\freemarker-2.3.30.jar;D:\MavenRepository\org\springframework\spring-context-support\5.3.2\spring-context-support-5.3.2.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter\2.4.1\spring-boot-starter-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot\2.4.1\spring-boot-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-autoconfigure\2.4.1\spring-boot-autoconfigure-2.4.1.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-logging\2.4.1\spring-boot-starter-logging-2.4.1.jar;D:\MavenRepository\ch\qos\logback\logback-classic\1.2.3\logback-classic-1.2.3.jar;D:\MavenRepository\ch\qos\logback\logback-core\1.2.3\logback-core-1.2.3.jar;D:\MavenRepository\org\apache\logging\log4j\log4j-to-slf4j\2.13.3\log4j-to-slf4j-2.13.3.jar;D:\MavenRepository\org\apache\logging\log4j\log4j-api\2.13.3\log4j-api-2.13.3.jar;D:\MavenRepository\org\slf4j\jul-to-slf4j\1.7.30\jul-to-slf4j-1.7.30.jar;D:\MavenRepository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;D:\MavenRepository\org\springframework\spring-core\5.3.2\spring-core-5.3.2.jar;D:\MavenRepository\org\springframework\spring-jcl\5.3.2\spring-jcl-5.3.2.jar;D:\MavenRepository\org\yaml\snakeyaml\1.27\snakeyaml-1.27.jar;D:\MavenRepository\jakarta\xml\bind\jakarta.xml.bind-api\2.3.3\jakarta.xml.bind-api-2.3.3.jar;D:\MavenRepository\jakarta\activation\jakarta.activation-api\1.2.2\jakarta.activation-api-1.2.2.jar;D:\MavenRepository\net\bytebuddy\byte-buddy\1.10.18\byte-buddy-1.10.18.jar;D:\MavenRepository\com\baomidou\mybatis-plus-boot-starter\3.4.0\mybatis-plus-boot-starter-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus\3.4.0\mybatis-plus-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-extension\3.4.0\mybatis-plus-extension-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-core\3.4.0\mybatis-plus-core-3.4.0.jar;D:\MavenRepository\com\baomidou\mybatis-plus-annotation\3.4.0\mybatis-plus-annotation-3.4.0.jar;D:\MavenRepository\com\github\jsqlparser\jsqlparser\3.2\jsqlparser-3.2.jar;D:\MavenRepository\org\mybatis\mybatis\3.5.5\mybatis-3.5.5.jar;D:\MavenRepository\org\mybatis\mybatis-spring\2.0.5\mybatis-spring-2.0.5.jar;D:\MavenRepository\com\alibaba\druid-spring-boot-starter\1.1.17\druid-spring-boot-starter-1.1.17.jar;D:\MavenRepository\com\alibaba\druid\1.1.17\druid-1.1.17.jar;D:\MavenRepository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;D:\MavenRepository\org\projectlombok\lombok\1.18.16\lombok-1.18.16.jar;D:\MavenRepository\com\alibaba\fastjson\1.2.75\fastjson-1.2.75.jar;D:\MavenRepository\org\reflections\reflections\0.9.12\reflections-0.9.12.jar;D:\MavenRepository\org\javassist\javassist\3.26.0-GA\javassist-3.26.0-GA.jar;D:\MavenRepository\mysql\mysql-connector-java\5.1.47\mysql-connector-java-5.1.47.jar;D:\MavenRepository\org\quartz-scheduler\quartz\2.3.2\quartz-2.3.2.jar;D:\MavenRepository\com\mchange\mchange-commons-java\0.2.15\mchange-commons-java-0.2.15.jar;D:\MavenRepository\org\postgresql\postgresql\42.2.6\postgresql-42.2.6.jar;D:\MavenRepository\com\auth0\java-jwt\3.7.0\java-jwt-3.7.0.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-databind\2.11.3\jackson-databind-2.11.3.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-core\2.11.3\jackson-core-2.11.3.jar;D:\MavenRepository\org\apache\shiro\shiro-spring-boot-starter\1.4.0\shiro-spring-boot-starter-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-spring\1.4.0\shiro-spring-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-core\1.4.0\shiro-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-lang\1.4.0\shiro-lang-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-cache\1.4.0\shiro-cache-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-hash\1.4.0\shiro-crypto-hash-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-core\1.4.0\shiro-crypto-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-crypto-cipher\1.4.0\shiro-crypto-cipher-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-config-core\1.4.0\shiro-config-core-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-config-ogdl\1.4.0\shiro-config-ogdl-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-event\1.4.0\shiro-event-1.4.0.jar;D:\MavenRepository\org\apache\shiro\shiro-web\1.4.0\shiro-web-1.4.0.jar;D:\MavenRepository\org\crazycake\shiro-redis\3.1.0\shiro-redis-3.1.0.jar;D:\MavenRepository\redis\clients\jedis\3.3.0\jedis-3.3.0.jar;D:\MavenRepository\org\apache\commons\commons-pool2\2.9.0\commons-pool2-2.9.0.jar;D:\MavenRepository\com\puppycrawl\tools\checkstyle\8.3\checkstyle-8.3.jar;D:\MavenRepository\antlr\antlr\2.7.7\antlr-2.7.7.jar;D:\MavenRepository\org\antlr\antlr4-runtime\4.7\antlr4-runtime-4.7.jar;D:\MavenRepository\commons-cli\commons-cli\1.4\commons-cli-1.4.jar;D:\MavenRepository\com\google\guava\guava\23.0\guava-23.0.jar;D:\MavenRepository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\MavenRepository\com\google\errorprone\error_prone_annotations\2.0.18\error_prone_annotations-2.0.18.jar;D:\MavenRepository\com\google\j2objc\j2objc-annotations\1.1\j2objc-annotations-1.1.jar;D:\MavenRepository\org\codehaus\mojo\animal-sniffer-annotations\1.14\animal-sniffer-annotations-1.14.jar;D:\MavenRepository\net\sf\saxon\Saxon-HE\9.8.0-4\Saxon-HE-9.8.0-4.jar;D:\MavenRepository\org\springframework\boot\spring-boot-starter-data-redis\2.4.1\spring-boot-starter-data-redis-2.4.1.jar;D:\MavenRepository\org\springframework\data\spring-data-redis\2.4.2\spring-data-redis-2.4.2.jar;D:\MavenRepository\org\springframework\data\spring-data-keyvalue\2.4.2\spring-data-keyvalue-2.4.2.jar;D:\MavenRepository\org\springframework\spring-tx\5.3.2\spring-tx-5.3.2.jar;D:\MavenRepository\org\springframework\spring-oxm\5.3.2\spring-oxm-5.3.2.jar;D:\MavenRepository\io\lettuce\lettuce-core\6.0.1.RELEASE\lettuce-core-6.0.1.RELEASE.jar;D:\MavenRepository\io\netty\netty-common\4.1.55.Final\netty-common-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-handler\4.1.55.Final\netty-handler-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-resolver\4.1.55.Final\netty-resolver-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-buffer\4.1.55.Final\netty-buffer-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-codec\4.1.55.Final\netty-codec-4.1.55.Final.jar;D:\MavenRepository\io\netty\netty-transport\4.1.55.Final\netty-transport-4.1.55.Final.jar;D:\MavenRepository\io\projectreactor\reactor-core\3.4.1\reactor-core-3.4.1.jar;D:\MavenRepository\org\reactivestreams\reactive-streams\1.0.3\reactive-streams-1.0.3.jar;D:\MavenRepository\io\springfox\springfox-swagger2\2.9.2\springfox-swagger2-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-spi\2.9.2\springfox-spi-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-core\2.9.2\springfox-core-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-schema\2.9.2\springfox-schema-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-swagger-common\2.9.2\springfox-swagger-common-2.9.2.jar;D:\MavenRepository\io\springfox\springfox-spring-web\2.9.2\springfox-spring-web-2.9.2.jar;D:\MavenRepository\com\fasterxml\classmate\1.5.1\classmate-1.5.1.jar;D:\MavenRepository\org\springframework\plugin\spring-plugin-core\1.2.0.RELEASE\spring-plugin-core-1.2.0.RELEASE.jar;D:\MavenRepository\org\springframework\plugin\spring-plugin-metadata\1.2.0.RELEASE\spring-plugin-metadata-1.2.0.RELEASE.jar;D:\MavenRepository\org\mapstruct\mapstruct\1.2.0.Final\mapstruct-1.2.0.Final.jar;D:\MavenRepository\io\springfox\springfox-swagger-ui\2.9.2\springfox-swagger-ui-2.9.2.jar;D:\MavenRepository\com\github\xiaoymin\swagger-bootstrap-ui\1.9.3\swagger-bootstrap-ui-1.9.3.jar;D:\MavenRepository\io\springfox\springfox-bean-validators\2.9.2\springfox-bean-validators-2.9.2.jar;D:\MavenRepository\io\swagger\swagger-annotations\1.5.20\swagger-annotations-1.5.20.jar;D:\MavenRepository\io\swagger\swagger-models\1.5.20\swagger-models-1.5.20.jar;D:\MavenRepository\com\fasterxml\jackson\core\jackson-annotations\2.11.3\jackson-annotations-2.11.3.jar;D:\MavenRepository\com\github\ulisesbocchio\jasypt-spring-boot-starter\3.0.3\jasypt-spring-boot-starter-3.0.3.jar;D:\MavenRepository\com\github\ulisesbocchio\jasypt-spring-boot\3.0.3\jasypt-spring-boot-3.0.3.jar;D:\MavenRepository\org\jasypt\jasypt\1.9.3\jasypt-1.9.3.jar;D:\MavenRepository\org\bouncycastle\bcprov-jdk15on\1.65\bcprov-jdk15on-1.65.jar;D:\MavenRepository\net\sf\json-lib\json-lib\2.4\json-lib-2.4-jdk15.jar;D:\MavenRepository\commons-beanutils\commons-beanutils\1.8.0\commons-beanutils-1.8.0.jar;D:\MavenRepository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;D:\MavenRepository\net\sf\ezmorph\ezmorph\1.0.6\ezmorph-1.0.6.jar;D:\MavenRepository\dom4j\dom4j\1.6.1\dom4j-1.6.1.jar;D:\MavenRepository\xml-apis\xml-apis\1.0.b2\xml-apis-1.0.b2.jar;D:\MavenRepository\com\thoughtworks\xstream\xstream\1.4.3\xstream-1.4.3.jar;D:\MavenRepository\xmlpull\xmlpull\1.1.3.1\xmlpull-1.1.3.1.jar;D:\MavenRepository\xpp3\xpp3_min\1.1.4c\xpp3_min-1.1.4c.jar;D:\MavenRepository\commons-codec\commons-codec\1.10\commons-codec-1.10.jar;D:\MavenRepository\cn\afterturn\easypoi-spring-boot-starter\3.3.0\easypoi-spring-boot-starter-3.3.0.jar;D:\MavenRepository\cn\afterturn\easypoi-web\3.3.0\easypoi-web-3.3.0.jar;D:\MavenRepository\cn\afterturn\easypoi-base\3.3.0\easypoi-base-3.3.0.jar;D:\MavenRepository\org\apache\commons\commons-lang3\3.11\commons-lang3-3.11.jar;D:\MavenRepository\ognl\ognl\3.2.6\ognl-3.2.6.jar;D:\MavenRepository\cn\afterturn\easypoi-annotation\3.3.0\easypoi-annotation-3.3.0.jar;D:\MavenRepository\org\apache\poi\poi-scratchpad\3.15\poi-scratchpad-3.15.jar;D:\MavenRepository\org\apache\poi\ooxml-schemas\1.3\ooxml-schemas-1.3.jar;D:\MavenRepository\org\apache\xmlbeans\xmlbeans\2.3.0\xmlbeans-2.3.0.jar;D:\MavenRepository\stax\stax-api\1.0.1\stax-api-1.0.1.jar;D:\MavenRepository\org\apache\httpcomponents\httpclient\4.5.3\httpclient-4.5.3.jar;D:\MavenRepository\org\apache\httpcomponents\httpcore\4.4.14\httpcore-4.4.14.jar;D:\MavenRepository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;D:\MavenRepository\commons-logging\commons-logging\1.0.4\commons-logging-1.0.4.jar;D:\MavenRepository\org\hibernate\validator\hibernate-validator\6.0.10.Final\hibernate-validator-6.0.10.Final.jar;D:\MavenRepository\javax\validation\validation-api\2.0.1.Final\validation-api-2.0.1.Final.jar;D:\MavenRepository\org\jboss\logging\jboss-logging\3.4.1.Final\jboss-logging-3.4.1.Final.jar;D:\MavenRepository\cn\hutool\hutool-all\5.6.1\hutool-all-5.6.1.jar;D:\MavenRepository\commons-net\commons-net\3.3\commons-net-3.3.jar;D:\MavenRepository\e-iceblue\spire.office\4.8.0\spire.office-4.8.0.jar;D:\MavenRepository\com\google\zxing\core\3.3.0\core-3.3.0.jar;D:\MavenRepository\com\google\zxing\javase\3.3.0\javase-3.3.0.jar;D:\MavenRepository\com\beust\jcommander\1.48\jcommander-1.48.jar com.lingxu.module.Datax
10:07:29.729 [main] INFO com.alibaba.datax.common.statistics.VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
10:07:29.736 [main] INFO com.alibaba.datax.core.Engine - the machine info  => 

	osInfo:	Oracle Corporation 1.8 25.91-b15
	jvmInfo:	Windows 10 amd64 10.0
	cpu num:	16

	totalPhysicalMemory:	-0.00G
	freePhysicalMemory:	-0.00G
	maxFileDescriptorCount:	-1
	currentOpenFileDescriptorCount:	-1

	GC Names	[PS MarkSweep, PS Scavenge]

	MEMORY_NAME                    | allocation_size                | init_size                      
	PS Eden Space                  | 1,166.00MB                     | 56.00MB                        
	Code Cache                     | 240.00MB                       | 2.44MB                         
	Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
	PS Survivor Space              | 9.00MB                         | 9.00MB                         
	PS Old Gen                     | 2,368.00MB                     | 148.00MB                       
	Metaspace                      | -0.00MB                        | 0.00MB                         


10:07:29.753 [main] INFO com.alibaba.datax.core.Engine - 
{
	"content":[
		{
			"reader":{
				"parameter":{
					"password":"****",
					"connection":[
						{
							"querySql":[
								"select t.id,t.name,t.status from users t"
							],
							"jdbcUrl":[
								"jdbc:mysql://127.0.0.1:3306/test"
							]
						}
					],
					"username":"root"
				},
				"name":"mysqlreader"
			},
			"writer":{
				"parameter":{
					"password":"****",
					"column":[
						"id",
						"name",
						"status"
					],
					"connection":[
						{
							"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test",
							"table":[
								"temp_users"
							]
						}
					],
					"writeMode":"insert",
					"username":"root"
				},
				"name":"mysqlwriter"
			}
		}
	],
	"setting":{
		"speed":{
			"channel":4
		}
	}
}

10:07:29.754 [main] DEBUG com.alibaba.datax.core.Engine - {"core":{"container":{"trace":{"enable":"false"},"job":{"reportInterval":10000,"id":-1},"taskGroup":{"channel":5}},"dataXServer":{"reportDataxLog":false,"address":"http://localhost:7001/api","timeout":10000,"reportPerfLog":false},"transport":{"exchanger":{"class":"com.alibaba.datax.core.plugin.BufferedRecordExchanger","bufferSize":32},"channel":{"byteCapacity":67108864,"flowControlInterval":20,"class":"com.alibaba.datax.core.transport.channel.memory.MemoryChannel","speed":{"byte":-1,"record":-1},"capacity":512}},"statistics":{"collector":{"plugin":{"taskClass":"com.alibaba.datax.core.statistics.plugin.task.StdoutPluginCollector","maxDirtyNumber":10}}}},"entry":{"jvm":"-Xms1G -Xmx1G"},"common":{"column":{"dateFormat":"yyyy-MM-dd","datetimeFormat":"yyyy-MM-dd HH:mm:ss","timeFormat":"HH:mm:ss","extraFormats":["yyyyMMdd"],"timeZone":"GMT+8","encoding":"utf-8"}},"plugin":{"reader":{"mysqlreader":{"path":"E:\\datax\\plugin\\reader\\mysqlreader","name":"mysqlreader","description":"useScene: prod. mechanism: Jdbc connection using the database, execute select sql, retrieve data from the ResultSet. warn: The more you know about the database, the less problems you encounter.","developer":"alibaba","class":"com.alibaba.datax.plugin.reader.mysqlreader.MysqlReader"}},"writer":{"mysqlwriter":{"path":"E:\\datax\\plugin\\writer\\mysqlwriter","name":"mysqlwriter","description":"useScene: prod. mechanism: Jdbc connection using the database, execute insert sql. warn: The more you know about the database, the less problems you encounter.","developer":"alibaba","class":"com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter"}}},"job":{"content":[{"reader":{"parameter":{"password":"密码","connection":[{"querySql":["select t.id,t.name,t.status from users t"],"jdbcUrl":["jdbc:mysql://127.0.0.1:3306/test"]}],"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"connection":[{"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test","table":["temp_users"]}],"writeMode":"insert","username":"root"},"name":"mysqlwriter"}}],"setting":{"speed":{"channel":4}}}}
10:07:29.806 [main] WARN com.alibaba.datax.core.Engine - prioriy set to 0, because NumberFormatException, the value is: null
10:07:29.807 [main] INFO com.alibaba.datax.common.statistics.PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
10:07:29.808 [main] INFO com.alibaba.datax.core.job.JobContainer - DataX jobContainer starts job.
10:07:29.808 [main] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do preHandle ...
10:07:29.808 [main] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do init ...
10:07:29.808 [main] INFO com.alibaba.datax.core.job.JobContainer - Set jobId = 0
10:07:30.106 [job-0] INFO com.alibaba.datax.plugin.rdbms.reader.util.OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
10:07:30.108 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Job - After job init(), job config now is:[
{"password":"密码","isTableMode":false,"fetchSize":-2147483648,"connection":[{"querySql":["select t.id,t.name,t.status from users t"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true"}],"tableNumber":0,"username":"root"}
]
10:07:30.133 [job-0] INFO com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil - table:[temp_users] all columns:[
id,name,status
].
10:07:30.138 [job-0] INFO com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,name,status) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
10:07:30.138 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job - After job init(), originalConfig now is:[
{"password":"密码","column":["id","name","status"],"connection":[{"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","table":["temp_users"]}],"insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"username":"root"}
]
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do prepare ...
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
10:07:30.138 [job-0] DEBUG com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job - After job prepare(), originalConfig now is:[
{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"}
]
10:07:30.138 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do split ...
10:07:30.139 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Job set Channel-Number to 4 channels.
10:07:30.140 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
10:07:30.140 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
10:07:30.140 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - transformer configuration: null
10:07:30.151 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - contentConfig configuration: [{"internal":{"reader":{"parameter":{"querySql":"select t.id,t.name,t.status from users t","password":"密码","isTableMode":false,"fetchSize":-2147483648,"loadBalanceResourceMark":"127.0.0.1","jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","tableNumber":0,"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"},"name":"mysqlwriter"},"taskId":0},"keys":["writer.parameter.column[1]","writer.parameter.tableNumber","writer.parameter.column[2]","writer.parameter.column[0]","writer.parameter.jdbcUrl","writer.parameter.writeMode","reader.parameter.fetchSize","reader.parameter.jdbcUrl","writer.parameter.insertOrReplaceTemplate","reader.parameter.password","writer.name","reader.parameter.querySql","reader.parameter.loadBalanceResourceMark","writer.parameter.password","writer.parameter.batchSize","writer.parameter.table","reader.parameter.tableNumber","reader.name","reader.parameter.isTableMode","reader.parameter.username","taskId","writer.parameter.username"],"secretKeyPathSet":[]}]
10:07:30.151 [job-0] INFO com.alibaba.datax.core.job.JobContainer - jobContainer starts to do schedule ...
10:07:30.153 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Scheduler starts [1] taskGroups.
10:07:30.154 [job-0] INFO com.alibaba.datax.core.job.JobContainer - Running by standalone Mode.
10:07:30.157 [taskGroup-0] DEBUG com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0]'s task configs[[{"internal":{"reader":{"parameter":{"querySql":"select t.id,t.name,t.status from users t","password":"密码","isTableMode":false,"fetchSize":-2147483648,"loadBalanceResourceMark":"127.0.0.1","jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","tableNumber":0,"username":"root"},"name":"mysqlreader"},"writer":{"parameter":{"password":"密码","column":["id","name","status"],"jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true","insertOrReplaceTemplate":"insert INTO %s (id,name,status) VALUES(?,?,?)","writeMode":"insert","batchSize":2048,"tableNumber":1,"table":"temp_users","username":"root"},"name":"mysqlwriter"},"taskId":0},"keys":["writer.parameter.column[1]","writer.parameter.tableNumber","writer.parameter.column[2]","writer.parameter.column[0]","writer.parameter.jdbcUrl","writer.parameter.writeMode","reader.parameter.fetchSize","reader.parameter.jdbcUrl","writer.parameter.insertOrReplaceTemplate","reader.parameter.password","writer.name","reader.parameter.querySql","reader.parameter.loadBalanceResourceMark","writer.parameter.password","writer.parameter.batchSize","writer.parameter.table","reader.parameter.tableNumber","reader.name","reader.parameter.isTableMode","reader.parameter.username","taskId","writer.parameter.username"],"secretKeyPathSet":[]}]]
10:07:30.157 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
10:07:30.159 [taskGroup-0] INFO com.alibaba.datax.core.transport.channel.Channel - Channel set byte_speed_limit to -1, No bps activated.
10:07:30.159 [taskGroup-0] INFO com.alibaba.datax.core.transport.channel.Channel - Channel set record_speed_limit to -1, No tps activated.
10:07:30.165 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
10:07:30.165 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do init ...
10:07:30.165 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do init ...
10:07:30.166 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do prepare ...
10:07:30.167 [job-0] DEBUG com.alibaba.datax.core.job.scheduler.AbstractScheduler - com.alibaba.datax.core.statistics.communication.Communication@27ce24aa[
  counter={}
  message={}
  state=RUNNING
  throwable=<null>
  timestamp=1663639650157
]
10:07:30.167 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do prepare ...
10:07:30.167 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to read ...
10:07:30.167 [0-0-0-reader] INFO com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Task - Begin to read record by Sql: [select t.id,t.name,t.status from users t
] jdbcUrl:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
10:07:30.170 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to write ...
10:07:30.174 [0-0-0-reader] INFO com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Task - Finished read record by Sql: [select t.id,t.name,t.status from users t
] jdbcUrl:[jdbc:mysql://127.0.0.1:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
10:07:30.177 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do post ...
10:07:30.177 [0-0-0-reader] DEBUG com.alibaba.datax.core.taskgroup.runner.ReaderRunner - task reader starts to do destroy ...
10:07:30.395 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do post ...
10:07:30.395 [0-0-0-writer] DEBUG com.alibaba.datax.core.taskgroup.runner.WriterRunner - task writer starts to do destroy ...
10:07:30.493 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[329]ms
10:07:30.493 [taskGroup-0] INFO com.alibaba.datax.core.taskgroup.TaskGroupContainer - taskGroup[0] completed it's tasks.
10:07:40.179 [job-0] DEBUG com.alibaba.datax.core.job.scheduler.AbstractScheduler - com.alibaba.datax.core.statistics.communication.Communication@188715b5[
  counter={writeSucceedRecords=3, readSucceedRecords=2, totalErrorBytes=0, writeSucceedBytes=8, byteSpeed=0, totalErrorRecords=0, recordSpeed=0, waitReaderTime=0, writeReceivedBytes=8, stage=1, waitWriterTime=37000, percentage=1.0, totalReadRecords=2, writeReceivedRecords=3, readSucceedBytes=8, totalReadBytes=8}
  message={}
  state=SUCCEEDED
  throwable=<null>
  timestamp=1663639660179
]
10:07:40.180 [job-0] INFO com.alibaba.datax.core.statistics.container.communicator.job.StandAloneJobContainerCommunicator - Total 2 records, 8 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
10:07:40.180 [job-0] INFO com.alibaba.datax.core.job.scheduler.AbstractScheduler - Scheduler accomplished all tasks.
10:07:40.180 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do post ...
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Writer.Job [mysqlwriter] do post work.
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX Reader.Job [mysqlreader] do post work.
10:07:40.181 [job-0] DEBUG com.alibaba.datax.core.job.JobContainer - jobContainer starts to do postHandle ...
10:07:40.181 [job-0] INFO com.alibaba.datax.core.job.JobContainer - DataX jobId [0] completed successfully.
10:07:40.181 [job-0] INFO com.alibaba.datax.core.container.util.HookInvoker - No hook invoked, because base dir not exists or is a file: E:\datax\hook
10:07:40.182 [job-0] INFO com.alibaba.datax.core.job.JobContainer - 
	 [total cpu info] => 
		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
		-1.00%                         | -1.00%                         | -1.00%
                        

	 [total gc info] => 
		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
		 PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
		 PS Scavenge          | 1                  | 1                  | 1                  | 0.004s             | 0.004s             | 0.004s             

10:07:40.182 [job-0] INFO com.alibaba.datax.core.job.JobContainer - PerfTrace not enable!
10:07:40.182 [job-0] INFO com.alibaba.datax.core.statistics.container.communicator.job.StandAloneJobContainerCommunicator - Total 2 records, 8 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
10:07:40.183 [job-0] INFO com.alibaba.datax.core.job.JobContainer - 
任务启动时刻                    : 2022-09-20 10:07:29
任务结束时刻                    : 2022-09-20 10:07:40
任务总计耗时                    :                 10s
任务平均流量                    :                0B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   2
读写失败总数                    :                   0


Process finished with exit code 0
4、数据库中查看

可以看到数据库中temp_users中已经成功的写入了users中的数据，已经成功入库

在这里插入图片描述

蠢蠢欲动的猫
关注
  • 1
    点赞
  • 踩
  • 13
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
java通过Python命令执行datax任务调度
03-23
1、通过java调用Python命令执行datax任务调度 2、自动开始任务和调度结束关闭任务调度释放内存 3、如果我们在cmd使用命令调度，执行完毕后无法释放内存，会造成内存泄露不足，出现报错“Error occurred during initialization of VM”
java集成datax所需pom依赖包
04-09
Java集成DataX涉及到的是数据同步工具的使用，DataX是阿里巴巴开源的一款用于大数据搬移的工具，它能够高效地支持大量数据在不同数据存储之间进行迁移。在这个场景中，"java datax"标签表明我们将讨论如何在Java项目...
Java Maven项目整合阿里云DataX
我的博客
01-13 748
文章大部分内容参照该地址datax安装 及 springboot整合 datax 以下部分有所改动 由于有个阿里云oss的依赖找不到，所以将其删除了，但是也可以完成打包，但是将所有的包导入到项目中后，运行的时候，发现有部分类找不到了。在pom中添加如下依赖后，程序正常运行。 依赖如下： <dependency> <groupId>commons-cli</groupId> <artifactId>commons-cl.
Java集成阿里巴巴的datax
weixin_43164465的博客
06-15 1229
alibaba datax java调用。
Datax集成到Java项目中，外加用xxljob定时任务自动同步
最新发布
w846736896的博客
06-19 456
我这边需要有四种类型的数据格式，分别是MySQL、Oracle、csv和excel，excel其实是转换成csv来进行操作，但我还是写了四个实现类，整个方法用了工厂模式来进行编写。后面一个方法调用上面创建reader的方法然后生成writer再合并再一起，保存下来json文件的路径。后面再需要的地方调用这个方法，把json文件的保存路径给xxljob，让他创建定时任务。创建定时任务 这样定时任务就会自动执行数据采集任务。excel转csv再进行操作。首先创建你自己所需要的变量。
批量数据之DataX数据同步
上善若泪
01-26 1304
DataX是阿里云DataWorks数据集成 的开源版本，主要就是用于实现数据间的离线同步。DataX致力于实现包括关系型数据库（MySQL、Oracle 等）、HDFS、Hive、ODPS、HBase、FTP 等 各种异构数据源（即不同的数据库） 间稳定高效的数据同步功能。为了解决异构数据源同步问题，DataX将复杂的网状同步链路变成了星型数据链路 ，DataX作为中间传输载体负责连接各种数据源；当需要接入一个新的数据源时，只需要将此数据源对接到DataX，便能跟已有的数据源作为无缝数据同步。
java调用datax实现不同数据库的表的同步
Libra815的博客
12-17 3839
最近工作中需要对不同数据库中的两个表的数据进行同步，然而本菜鸡只会先将表里的数据清空，让后通过@DS注解查询另一张表的数据插入到被清空的表中。本来，代码都写完了，开始运行。哦嚯，由于命名规范，两个表的名称不一样，报错了，找不到那张表。但字段都是一样的，为此再写一个实体，加上@TableName注解又不太合适。前辈告诉我说，用datax同步数据。 1.下载与使用 一开始，下载的datax的源码，弄了半天不会用。所以直接用现成的吧,下载完解压就行了。 放上链接:datax下载地址 ...
Datax入门使用
qq_44134480的博客
04-23 9805
DataX入门使用 一、简介 DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。Datax将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件
DataX + xxl-job
allensandy的博客
12-01 4377
1、部署xx-job 官网：https://www.xuxueli.com/xxl-job/里面由详细的讲解 我用的版本： 安装docker mysql docker pull mysql:8.0.19 docker run -itd --name mysql-xxljob -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 mysql mysql -u root -h 127.0.0.1 -P3306 -p 进行授权远程连接(注意mysql 8.0跟.
大数据技术之DataX （一）DataX插件开发
大数据指北
10-12 7547
文章目录一、背景二、基于java的本地测试datax2.1 github上下载datax的源代码2.2 datax代码导入idea三、docker安装南大通用数据库GBase和GBase 8a3.1 docker安装Gbase 8a3.2 docker安装Gbase 8s三、南大通用数据库GBase 8s To GBase 8a3.1 GBase 8s的reader读插件开发（writer同理）四、南大通用gbase可视化工具4.1 南大通用GBase 8a的连接配置4.2 南大通用GBase 8s的
关于通过java调用datax,返回任务执行的方法
08-25
为了更好地管控 DataX 的任务，我们需要使用 Java 集成的方式去调用 DataX，並返回任务执行的详细信息。 在使用 Java 调用 DataX 之前，我们需要了解 DataX 的源码结构。从 GitHub 上下载 DataX 的源码，并跟踪源码...
Java通过python命令执行DataX任务
12-19
Java调用控制台python命令(含传递参数)执行DataX的job任务，mysql向odps同步数据。
java调用shell向DataX传递参数，where条件，包含特殊字符
04-10
java调用shell向DataX传递参数，where条件，包含特殊字符。java调用shell向DataX传递参数，where条件，包含特殊字符
dataxdemo.zip
07-17
java集成datax的demo代码，相关介绍文档详见 https://blog.csdn.net/linlzk/article/details/107400993
java使用datax增量同步代码
04-19
Java 使用 DataX 进行增量同步是大数据领域中常见的数据迁移任务，DataX 是阿里开源的一个强大、高效的数据同步工具，它可以实现不同数据存储之间的数据迁移。本篇将详细讲解如何在 Java 项目中利用 DataX 实现增量...
datax 定时执行多个job_Spring Boot如何整合Quartz 实现定时任务
weixin_39716521的博客
11-23 428
在项目开发中，经常需要定时任务来后台处理业务，比如定时生成数据报表、生成对账单、数据监控等。Spring Boot 中有两种方案可以选择，第一种是 Spring Boot 内置的注解方式实现，这个方式之前介绍过了，可以看我这篇文章《Spring Boot入门系列(八)整合定时任务Task，一秒搞定定时任务》，如果需要更复杂的应用场景那么就需要Quartz 实现， Quartz是目前最完善的定时任务...
数据同步工具—DataX—Web部署使用
张伟的专栏
02-06 5650
这一节主要部署了DataX—Web ，DataX—Web 主要解决了DataX的任务管理、运维的问题，而且有自己的调度，但是在整个数仓中却难以使用，因为我们无法将DataX—Web 和已有的调度系统进行整合。所以比较好的方式是在我们的调度系统中去集成DataX—Web，那这个时候只有任务构建这个模块是我们需要的，其他的监控、数据源管理调度系统已经有了，所以这个东西应用场景有限，只能作为一个参考。
DataX使用入门
mynameisjinxiaokai的博客
01-17 3026
DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。下载对应的MySQL驱动版本：https://static.runoob.com/download/mysql-connector-java-8.0.16.jar。方法二、下载DataX源码，自己编译：DataX源码。
java 集成datax
12-15
以下是通过Java调用DataX实现数据同步的步骤： 1.下载DataX源代码并解压缩。 2.在Java项目中引入DataX的依赖包。 3.编写Java代码，通过调用DataX的API实现数据同步。具体代码如下： ```java import com.alibaba.datax.core.DataX; import com.alibaba.datax.core.util.ExceptionTracker; public class DataXTest { public static void main(String[] args) { try { String[] datxArgs = {"-p", "-jvm", "-job", "/path/to/job.json"}; DataX.main(datxArgs); } catch (Throwable e) { System.out.println(ExceptionTracker.trace(e)); } } } ``` 其中，`/path/to/job.json`是DataX的作业配置文件路径。 4.运行Java代码，即可实现数据同步。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值