Dolphinscheduler 3.2.2 新版快速部署使用教程

Apache DolphinScheduler 是一个分布式易扩展的可视化DAG工作流任务调度开源系统。适用于企业级场景，提供了一个可视化操作任务、工作流和全生命周期数据处理过程的解决方案。

Apache DolphinScheduler 旨在解决复杂的大数据任务依赖关系，并为应用程序提供数据和各种 OPS 编排中的关系。解决数据研发ETL依赖错综复杂，无法监控任务健康状态的问题。DolphinScheduler 以 DAG（Directed Acyclic Graph，DAG）流式方式组装任务，可以及时监控任务的执行状态，支持重试、指定节点恢复失败、暂停、恢复、终止任务等操作。

快速体验单机部署 Standalone 仅适用于 DolphinScheduler 的快速体验

部署前需要下载部署 zookeeper

docker pull registry.cn-hangzhou.aliyuncs.com/qiluo-images/zookeeper:latest

创建 zoo.cfg

[root@k8s-normal-master conf]# cat zoo.cfg 
# 设置端口
clientPort=2181

# Zookeeper保存数据的目录，默认情况下，Zookeeper将写数据的日志文件也保存在这个目录里
dataDir=/data

# 事物日志存储地点，如果没提供的话使用的则是 dataDir
dataLogDir=/datalog

# 服务器之间或客户端与服务器之间维持心跳的时间间隔，也就是每个 tickTime 时间就会发送一个心跳。tickTime以毫秒为单位
tickTime=2000

# 集群中的follower服务器(F)与leader服务器(L)之间初始连接时能容忍的最多心跳数（tickTime的数量）
initLimit=5

# 集群中的follower服务器与leader服务器之间请求和应答之间能容忍的最多心跳数（tickTime的数量）
syncLimit=2

# 默认值为3，不支持以系统属性方式配置。用于配置Zookeeper在自动清理的时候需要保留的快照数据文件数量和对应的事务日志文件。此参数的最小值为3，如果配置的值小于3会自动调整到3
autopurge.snapRetainCount=3

# 默认值为0，单位为小时，不支持以系统属性方式配置。用于配置Zookeeper进行历史文件自动清理的频率。如果配置为0或负数，表示不需要开启定时清理功能
autopurge.purgeInterval=0

# 默认为60，不支持以系统属性方式配置。从Socket层面限制单个客户端与单台服务器之间的并发连接数，即以ip地址来进行连接数的限制。
# 如果设置为0，表示不做任何限制。仅仅是单台客户端与单个Zookeeper服务器连接数的限制，不能控制所有客户端的连接数总和
maxClientCnxns=60

# 3.5.0中的新功能：当设置为false时，可以在复制模式下启动单个服务器，单个参与者可以使用观察者运行，并且群集可以重新配置为一个节点，并且从一个节点。
# 对于向后兼容性，默认值为true。可以使用QuorumPeerConfig的setStandaloneEnabled方法或通过将“standaloneEnabled = false”或“standaloneEnabled = true”添加到服务器的配置文件来设置它。
standaloneEnabled=false

# 内嵌的管理控制台，停用这个服务
# admin.enableServer=false

# 开启四字命令，将所有命令添加到白名单中
# 4lw.commands.whitelist=*

执行运行命令

docker run -d   --name zookeeper   -p 2181:2181   -v /data/zookeeper/conf/zoo.cfg:/apache-zookeeper-3.9.2-bin/conf/zoo.cfg  registry.cn-hangzhou.aliyuncs.com/qiluo-images/zookeeper:latest

以下是两个海豚调度二进制包下载链接：
https://dolphinscheduler.apache.org/zh-cn/download

https://dlcdn.apache.org/dolphinscheduler/3.2.2/

下载到服务器自定义目录下

yum install wget
wget  https://dlcdn.apache.org/dolphinscheduler/3.2.2/apache-dolphinscheduler-3.2.2-bin.tar.gz
tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz
mv apache-dolphinscheduler-3.2.2-bin/ dolphinscheduler-3.2.2

修改目录访问权限

chmod -R 755 dolphinscheduler-3.2.2/

由于最新版本却失install_env.sh 和install.sh
需要直接去下载，下载链接：

https://github.com/apache/dolphinscheduler/blob/3.1.5-release/script/env/install_env.sh

3.2.1以上都缺失

快速启动

cd  /data/dolphinscheduler-3.2.2/api-server/conf
vi application.yaml

```bash
server:
  port: 12345 改成自定义端口
  servlet:
    session:
      timeout: 120m
    context-path: /dolphinscheduler/
  compression:
    enabled: true
    mime-types: text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml
  jetty:
    max-http-form-post-size: 5000000
    accesslog:
      enabled: true
      custom-format: '%{client}a - %u %t "%r" %s %O %{ms}Tms'

cd /data/dolphinscheduler-3.2.2/bin

./dolphinscheduler-daemon.sh start standalone-server

修改端口：

把12345改成自己的端口

打开浏览器访问链接http://172.17.0.200:32000/dolphinscheduler/ui/login

输入默认登录用户名密码admin/dolphinscheduler123

伪集群部署
伪集群部署目的是在单台机器部署 DolphinScheduler 服务，该模式下 master、worker、api server 都在同一台机器上。

配置用户免密及权限

创建部署用户，并且一定要配置 sudo 免密。以创建 dolphinscheduler 用户为例：

创建用户需使用 root 登录

useradd dolphinscheduler

添加密码

echo "dolphinscheduler" | passwd --stdin dolphinscheduler

配置 sudo 免密

sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: ALL' /etc/sudoers

sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers

修改配置文件 install_env.sh 和 dolphinscheduler_env.sh

cd  /data/dolphinscheduler-3.2.2/bin/env

修改 install_env.sh

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="172.16.0.200"

# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"

# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="172.16.0.200"

# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="172.16.0.200:default"

# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="172.16.0.200"

# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="172.16.0.200"

# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd). Do not add quotes to it if you using related path.
installPath="/tmp/dolphinscheduler"

# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="root"

# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot="/dolphinscheduler"

然后是修改dolphinscheduler_env.sh

# for postgresql
export DATABASE=postgresql
export SPRING_PROFILES_ACTIVE=dolphinscheduler
export SPRING_DATASOURCE_URL="jdbc:postgresql://172.16.0.201:5432/dolphinscheduler"
export SPRING_DATASOURCE_USERNAME=dolphinscheduler
export SPRING_DATASOURCE_PASSWORD=dolphinscheduler

export SPRING_CACHE_TYPE=-NONE
export SPRING_JACKSON_TIME_ZONE=UTC
export MASTER_FETCH_COMMAND_NUM=10

export REGISTRY_TYPE=zookeeper
export REGISTRY_ZOOKEEPER_CONNECT_STRING=172.16.0.201:2181

执行数据库升级脚本安装：

bash tools/bin/upgrade-schema.sh

执行安装

bash ./bin/install.sh

一键停止集群所有服务

bash ./bin/stop-all.sh

一键开启集群所有服务

bash ./bin/start-all.sh

接口文档访问地址：http://172.16.0.200:32000/dolphinscheduler/swagger-ui/index.html

然后打包镜像 创建 Dockerfile

[root@k8s-normal-master dolphinscheduler-docker]# cat Dockerfile 
FROM openjdk:8-jre

# 更换为阿里云的 Debian 镜像源
RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list

# 安装必要依赖  
RUN apt-get update && \
    apt-get install -y procps

# 设置环境变量  
ENV DS_HOME=/opt/dolphinscheduler  
ENV PATH="$DS_HOME/bin:$PATH"  

# 创建 DolphinScheduler 目录  
RUN mkdir -p $DS_HOME  

# 复制编译后的文件到镜像中  
COPY dolphinscheduler-3.2.2 /opt/dolphinscheduler/  

# 设置工作目录  
WORKDIR $DS_HOME  

# 暴露端口（根据需要暴露端口）  
EXPOSE 12345 56789 

# 定义启动命令  
CMD ["sh", "-c", "bin/dolphinscheduler-daemon.sh start standalone-server"]

打包镜像

docker build -t dolphinscheduler:3.3.2 .
运行服务
docker run -d --name dolphinscheduler -p 32000:12345 -p 56789:56789 dolphinscheduler:3.3.2