DolphinScheduler安装及部署

DolphinScheduler安装及部署

一、前期准备

（1）五台节点均需部署JDK（1.8+），并配置相关环境变量。

（2）需部署数据库，支持MySQL（5.7+）或者PostgreSQL（8.2.15+）。如 MySQL 则需要 JDBC Driver 8.0.16。

（3）需部署Zookeeper（3.4.6+）。

（4）如果启用 HDFS 文件系统，则需要 Hadoop（2.6+）环境。

（5）五台节点均需安装进程管理工具包psmisc。

[atguigu@hadoop100 ~]$ sudo yum install -y psmisc

[atguigu@hadoop101 ~]$ sudo yum install -y psmisc

[atguigu@hadoop102 ~]$ sudo yum install -y psmisc

[atguigu@hadoop103 ~]$ sudo yum install -y psmisc

[atguigu@hadoop104 ~]$ sudo yum install -y psmisc

二、安装部署

1、上传解压至software

注意：解压目录并发最终的安装目录

2、配置一键部署脚本

修改解压目录下的conf/config目录下的install_config.conf文件

[atguigu@hadoop102 apache-dolphinscheduler-2.0.3-bin]$

vim conf/config/install_config.conf

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="hadoop100,hadoop101,hadoop102,hadoop103,hadoop104" 
# 将要部署任一 DolphinScheduler 服务的服务器主机名或 ip 列表

# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"

# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="hadoop100,hadoop101" 
# master 所在主机名列表，必须是 ips 的子集

# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="hadoop102:default,hadoop103:default,hadoop104:default" 
# worker主机名及队列，此处的 ip 必须在 ips 列表中

# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="hadoop102"
# 告警服务所在服务器主机名
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="hadoop102"
# api服务所在服务器主机名

# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
# pythonGatewayServers="ds1" 
# 不需要的配置项，可以保留默认值，也可以用 # 注释

# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/opt/module/dolphinscheduler"
# DS 安装路径，如果不存在会创建

# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="atguigu"
# 部署用户，任务执行服务是以 sudo -u {linux-user} 切换不同 Linux 用户的方式来实现多租户运行作业，因此该用户必须有免密的 sudo 权限。

# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="/tmp/dolphinscheduler"
# 前文配置的所有节点的本地数据存储路径，需要确保部署用户拥有该目录的读写权限

# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/module/jdk1.8.0_212"
# JAVA_HOME 路径

# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"

# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# 注意：数据库相关配置的 value 必须加引号，否则配置无法生效

DATABASE_TYPE="mysql"
# 数据库类型

# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
# SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
SPRING_DATASOURCE_URL="jdbc:mysql://hadoop100:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# 数据库 URL

# Spring datasource username
# SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"sa"}
SPRING_DATASOURCE_USERNAME="dolphinscheduler"
# 数据库用户名

# Spring datasource password
# SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-""}
SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
# 数据库密码

# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# 注册中心插件名称，DS 通过注册中心来确保集群配置的一致性

# Registry Server address.
registryServers="hadoop102:2181,hadoop103:2181,hadoop104:2181"
# 注册中心地址，即 Zookeeper 集群的地址

# Registry Namespace
registryNamespace="dolphinscheduler"
# DS 在 Zookeeper 的结点名称

# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"

# resource storage type: HDFS, S3, NONE
resourceStorageType="HDFS"	
# 资源存储类型

# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# 资源上传路径

# if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://hadoop100:8020"
# 默认文件系统

# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# yarn RM http 访问端口

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps=
# Yarn RM 高可用 ip，若未启用 RM 高可用，则将该值置空

# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="hadoop101"
# Yarn RM 主机名，若启用了 HA 或未启用 RM，保留默认值

# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="atguigu"
# 拥有 HDFS 根目录操作权限的用户

# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"

# use sudo or not
sudoEnable="true"

# worker tenant auto create
workerTenantAutoCreate="false"

3、初始化数据库

（1）创建数据库

mysql>

CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

（2）创建用户

mysql>

CREATE USER ‘dolphinscheduler’@‘%’ IDENTIFIED BY ‘dolphinscheduler’;

注：

若出现以下错误信息，表明新建用户的密码过于简单。

ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

可提高密码复杂度或者执行以下命令降低MySQL密码强度级别。

mysql> set global validate_password_length=4;

mysql> set global validate_password_policy=0;

（3）赋予用户相应权限

mysql>

GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ‘dolphinscheduler’@‘%’;

mysql>

flush privileges;

（4）拷贝MySQL驱动到DolphinScheduler的解压目录下的lib中

[atguigu@hadoop102 apache-dolphinscheduler-2.0.3-bin]$

cp /opt/software/mysql-connector-java-8.0.16.jar lib/

（5）执行数据库初始化脚本

数据库初始化脚本位于DolphinScheduler解压目录下的script目录中，即/opt/software/ds/apache-dolphinscheduler-2.0.3-bin/script/。

[atguigu@hadoop102 apache-dolphinscheduler-2.0.3-bin]$

script/create-dolphinscheduler.sh

4、启动DS

（1）启动zookeeper

zk.sh start

（2）启动DS

[atguigu@hadoop102 apache-dolphinscheduler-2.0.3-bin]$

./install.sh

（3）检查进程

MasterServer（2个）

LoggerServer（5个）

ApiApplicationServer（1个 —> 网页端口）

WorkerServer（3个）

AlertServer （1个）

三、操作入门

1、安全中心配置

（1）租户管理

​ 租户对应的是Linux的用户，用于worker提交作业所使用的用户。如果linux没有这个用户，则会导致任务运行失败。

你可以通过修改 worker.properties 配置文件中参数

worker.tenant.auto.create=true（默认值为 false）

实现当 linux 用户不存在时自动创建该用户。

worker.tenant.auto.create=true参数会要求 worker 可以免密运行 sudo 命令。

注：

Ø 租户编码：租户编码是Linux上的用户，唯一，不能重复。

Ø 队列：该租户提交Yarn任务时的默认队列。

（2）用户管理

注：

Ø 用户名：DolphinScheduler登录账户

Ø 租户：该用户关联的租户

Ø 队列：默认为租户所关联的队列。

Ø 邮件、手机号：主要用于告警通知。

（3）告警组管理

告警组是在启动时设置的参数，在流程结束以后会将流程的状态和其他信息以邮件形式发送给告警组。

管理员进入安全中心->告警组管理页面，点击“创建告警组”按钮，创建告警组。

注：

Ø 组名称：用户自定义的告警组名称。

Ø 告警插件实例：用户创建的告警实例。

（4）worker分组管理

修改步骤为"安全中心 -> worker分组管理 -> 点击 ‘新建worker分组’ -> 输入’组名称’ -> 选择已有worker -> 点击’提交’"。

（5）Yarn队列管理

一旦设置任务只会在队列对应的机器上运行。

（6）环境管理

1）创建更新环境

环境配置等价于dolphinscheduler_env.sh文件内配置。

路径、文件名不同需要作调整！！！

export HADOOP_HOME=/opt/module/hadoop-3.3.4
export HADOOP_CONF_DIR=/opt/module/hadoop-3.3.4/etc/hadoop
export SPARK_HOME=/opt/module/spark
export JAVA_HOME=/opt/module/jdk1.8.0_212
export HIVE_HOME=/opt/module/hive
export DATAX_HOME=/opt/module/datax

export PATH=$PATH:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$DATAX_HOME/bin

注：

Ø 环境名称：用户自定义名称。

Ø 环境配置：与 dolphinscheduler_env.sh 配置格式相同。

Ø 详细描述：环境的详细描述，不能为空，否则无法创建

Ø Worker组：环境所属的 Worker 组。如果不选择依然可以创建成功，但在启动工作流时无法选择该环境。

2）使用环境

在工作流定义中创建任务节点选择Worker分组和Worker分组对应的环境，任务执行时Worker会先执行环境再执行任务。

注意：可以在创建任务节点时选择环境，也可以在工作流执行前选择环境，但只有前者选择的环境可以生效。

2、项目管理

见文档

3、工作流基础配置

见文档