一、下载代码datax
二、编译
本人开发环境Macbook M1,一切顺利~
三、运行datax
1.拷贝plugin,这里先使用streamReader和streamWriter测试,plugin在对应插件target目录下,拷贝过来即可。
2.测试生成job文件
注意datax.py目录在core目录下,这里以stream进行测试
/opt/homebrew/bin/python3 datax.py -r streamreader -w streamwriter
{
"job": {
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [],
"sliceRecordCount": ""
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding": "",
"print": true
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}
3.运行
配置文件core/target/datax/bin/stream.json文件
{
"job": {
"content": [{
"reader": {
"name": "streamreader",
"parameter": {
"sliceRecordCount": 10,
"column": [{
"type": "long",
"value": "10"
},
{
"type": "string",
"value": "hello,你好,世界-DataX"
}
]
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"encoding": "UTF-8",
"print": true
}
}
}],
"setting": {
"speed": {
"channel": 5
}
}
}
}
运行:/opt/homebrew/bin/python3 datax.py stream.json
结果示例
四、本地debug
4.1 打印启动命令
4.2 根据启动命令配置IDEA参数
程序参数
-mode standalone -jobid -1 -job /Users/shaobing/Desktop/learn/openSource/DataX/core/target/datax/bin/stream.json
JVM参数
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/Users/shaobing/Desktop/learn/openSource/DataX/core/target/datax/log
-Ddatax.home=/Users/shaobing/Desktop/learn/openSource/DataX/core/target/datax
-Dlogback.configurationFile=/Users/shaobing/Desktop/learn/openSource/DataX/core/target/datax/conf/logback.xml
接下来可以愉快的进行Debug啦