1、下载datax
GitHub - alibaba/DataX: DataX是阿里云DataWorks数据集成的开源版本。
在服务器中使用wget 下载url下载tar.gz包
执行tar -zxvf datax.tar.gz
2、配置datax及python环境
配置地址在/dolphinscheduler-bin/bin/env/dolphinscheduler_env.sh
修改如下:
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=postgresql
export SPRING_DATASOURCE_URL=jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
export SPRING_DATASOURCE_USERNAME=postgres
export SPRING_DATASOURCE_PASSWORD=postgres
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1}
export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2}
export PYTHON_HOME=/usr
export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_HOME=/temp/datax
export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/soft/seatunnel}
export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}
export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/lib:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH
注意:
Java地址为apt安装后默认地址;
python位系统自带地址,目录为、/usr/lib/pyhton2.7,但是在配置时,不PYTHONE_HOME只到/usr,下面$PYTHON_HOME/bin 修改为 $PYTHON_HOME/lib,在执行datax服务时默认会默认带上/python2.7,所以这里要去掉;
DATAX_HOME为datax解压地址
3、修改core.json,位置在:/datax/conf/core.json
core.transport.channel.speed.byte=-1修改为2000000
4、如果/usr/lib/python2.7文件里面为空的,就需要安装python2.7
sudo apt install python2-minimal
如果找不到python命令,需要在/usr/bin/为python添加快捷方式
sudo ln -s /usr/bin/python3 /usr/bin/python
4、设置项目及工作流
1)配置数据源
2)添加项目
3)添加工作流
4)执行任务:上线任务,执行任务
5、查看日志,执行完成之后可看到任务的日志