本文主要记录一下最基础版本的使用方法
按本文安装完毕后,可以实现命令行执行任务, 而非界面操作
1. 安装运行环境
记得设置环境变量
- Java (1.8)
- Mave (3.x.x)
- Python (3.x.x)
具体怎么安装左转百度吧,不再赘述了
比如我的环境:
C:\Users\DELL>java -version
java version "1.8.0_421"
Java(TM) SE Runtime Environment (build 1.8.0_421-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.421-b09, mixed mode)
C:\Users\DELL>mvn -v
Apache Maven 3.9.4 (dfbb324ad4a7c8fb0bf182e6d91b0ae20e3d2dd9)
Maven home: C:\soft\apache-maven-3.9.4
Java version: 1.8.0_421, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jre1.8.0_421
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 11", version: "10.0", arch: "amd64", family: "windows"
PS C:\Users\DELL> python
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
2.下载Datax 压缩包
官网地址:https://github.com/alibaba/DataX
拉到最下方点击如图所示下载地址,下载到本地。
下载后的包名:datax.tar.gz
解压缩后的目录如图:
3. 执行测试
进入 bin 目录,在 bin 目录下打开cmd (Linux直接在 bin 目录执行命令)
> python datax.py ../job/job.json
DataX在 job 目录,有个默认任务 job.json
执行后看到这个就是任务执行成功
4. 编写任务脚本
参照 job 目录的 job.json,编写自己的任务
这里分Reader和Writer两部分,例如可以
在 Reader 部分定义读取 Mysql 数据
在 Writer 部分定义写入到 Doris 中去
可是脚本怎么写呢,有没有办法直接复制来呢,有的
在源码的目录下面,就有官方文档 比如这里以Mysql 为例
https://github.com/alibaba/DataX/tree/master/mysqlreader
点击链接看到文档:
5. 脚本说明
贴一个自己写的简单job
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": ["user_id", "serial_no", "sn","status","create_time","update_time"],
"connection": [
{
"jdbcUrl": ["jdbc:mysql://172.172.172.172:9030/demo"],
"table": ["vx_info"]
}
],
"username": "admin",
"password": "admin123",
"where": "1=1 and user_id='1554236'"
}
},
"writer": {
"name": "doriswriter",
"parameter": {
"loadUrl": ["172.172.172.172:8030"],
"loadProps": {
},
"column": ["user_id", "serial_no", "sn","status","create_time","update_time"],
"username": "admin",
"password": "admin",
"postSql": ["select count(1) from ve_info_test"],
"preSql": [],
"flushInterval":30000,
"connection": [
{
"jdbcUrl": "jdbc:mysql://172.172.172.172:9030/demo",
"selectedDatabase": "demo",
"table": ["ve_info_test"]
}
],
"loadProps": {
"format": "json",
"strip_outer_array": true
}
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
}