dolphinscheduler搭配datax跨库数据库全量更新

1、下载datax

GitHub - alibaba/DataX: DataX是阿里云DataWorks数据集成的开源版本。

在服务器中使用wget 下载url下载tar.gz包

执行tar -zxvf datax.tar.gz

2、配置datax及python环境

配置地址在/dolphinscheduler-bin/bin/env/dolphinscheduler_env.sh

修改如下：

# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=postgresql
export SPRING_DATASOURCE_URL=jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
export SPRING_DATASOURCE_USERNAME=postgres
export SPRING_DATASOURCE_PASSWORD=postgres

# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}

# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}

# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1}
export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2}
export PYTHON_HOME=/usr
export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_HOME=/temp/datax
export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/soft/seatunnel}
export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/lib:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH

注意：

Java地址为apt安装后默认地址；

python位系统自带地址，目录为、/usr/lib/pyhton2.7，但是在配置时，不PYTHONE_HOME只到/usr，下面$PYTHON_HOME/bin 修改为 $PYTHON_HOME/lib，在执行datax服务时默认会默认带上/python2.7，所以这里要去掉；

DATAX_HOME为datax解压地址

3、修改core.json，位置在：/datax/conf/core.json

core.transport.channel.speed.byte=-1修改为2000000

4、如果/usr/lib/python2.7文件里面为空的，就需要安装python2.7

sudo apt install python2-minimal

如果找不到python命令，需要在/usr/bin/为python添加快捷方式

sudo ln -s /usr/bin/python3 /usr/bin/python

4、设置项目及工作流

1）配置数据源

2）添加项目

3）添加工作流

4）执行任务：上线任务，执行任务

5、查看日志，执行完成之后可看到任务的日志