dolphinscheduler3.1.8集群部署
环境信息
主机名 IP 系统 角色
bigtop01 192.168.130.171 CentOS-7.9 Slaver
bigtop02 192.168.130.172 CentOS-7.9 Slaver
bigtop03 192.168.130.173 CentOS-7.9 Slaver
bigtop04 192.168.130.174 CentOS-7.9 Slaver
bigtop05 192.168.130.175 CentOS-7.9 Worker
bigtop06 192.168.130.176 CentOS-7.9 Master
基础环境调优
1.主机名映射
2.时间同步
3.免密登录
注意：直接以root用户部署也可以，不用创建dolphinscheduler用户
部署JDK
下载JDK (1.8+)，安装并配置 JAVA_HOME 环境变量，并将其下的 bin 目录追加到 PATH 环境变量中。如果你的环境中已存在，可以跳过这步。
[root@bigtop06 env]# vim dolphinscheduler_env.sh
export JAVA_HOME=${JAVA_HOME:-/etc/alternatives/java_sdk_1.8.0}
部署zookeeper集群
当前环境和bigtop集群共用，已经有6个节点的zk集群，直接使用即可
[root@bigtop06 env]# vim dolphinscheduler_env.sh
export REGISTRY_TYPE=
R
E
G
I
S
T
R
Y
T
Y
P
E
:
−
z
o
o
k
e
e
p
e
r
e
x
p
o
r
t
R
E
G
I
S
T
R
Y
Z
O
O
K
E
E
P
E
R
C
O
N
N
E
C
T
S
T
R
I
N
G
=
{REGISTRY_TYPE:-zookeeper} export REGISTRY_ZOOKEEPER_CONNECT_STRING=
REGISTRYTYPE:−zookeeperexportREGISTRYZOOKEEPERCONNECTSTRING={REGISTRY_ZOOKEEPER_CONNECT_STRING:-192.168.130.171:2181,192.168.130.172:2181,192.168.130.173:2181,192.168.130.174:2181,192.168.130.175:2181,192.168.130.176:2181}
Master节点部署MySQL 5.7
wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
yum -y install mysql57-community-release-el7-11.noarch.rpm
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum -y install mysql mysql-server mysql-devel
[root@localhost ~]# systemctl start mysqld.service
[root@localhost ~]# systemctl status mysqld.service
[root@localhost ~]# grep “password” /var/log/mysqld.log
[root@bigtop06 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-10-19 15:07:00 CST; 28s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Main PID: 6526 (mysqld)
CGroup: /system.slice/mysqld.service
└─6526 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Oct 19 15:06:53 bigtop06 systemd[1]: Starting MySQL Server…
Oct 19 15:07:00 bigtop06 systemd[1]: Started MySQL Server.
[root@bigtop06 ~]# grep “password” /var/log/mysqld.log
2023-10-19T07:06:57.560077Z 1 [Note] A temporary password is generated for root@localhost: dooo_(FeC1h,
[root@bigtop06 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.43
Copyright © 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> ALTER USER ‘root’@‘localhost’ IDENTIFIED BY ‘Sirun@2023’;
mysql> GRANT ALL PRIVILEGES ON . TO ‘root’@‘%’ IDENTIFIED BY ‘Sirun@2023’ WITH GRANT OPTION;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
[root@bigtop06 apache-dolphinscheduler-3.1.8-bin]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.43 MySQL Community Server (GPL)
Copyright © 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE USER ‘dolphinscheduler’@‘%’ IDENTIFIED BY ‘dolphinscheduler’;
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> CREATE USER ‘dolphinscheduler’@‘%’ IDENTIFIED BY ‘Sirun@2023’;
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
部署dolphinscheduler集群
1.调整dolphinscheduler_env.sh和 install_env.sh配置文件
[root@bigtop06 logs]# cd /opt/apache-dolphinscheduler-3.1.8-bin/bin/env/
[root@bigtop06 env]# ll
total 8
-rwxrwxrwx 1 root root 2898 Oct 20 15:59 dolphinscheduler_env.sh
-rwxrwxrwx 1 root root 3697 Oct 20 15:12 install_env.sh
[root@bigtop06 env]# cat dolphinscheduler_env.sh |grep -v ^# |grep -v ^$
export JAVA_HOME=
J
A
V
A
H
O
M
E
:
−
/
e
t
c
/
a
l
t
e
r
n
a
t
i
v
e
s
/
j
a
v
a
s
d
k
1
.
8.0
e
x
p
o
r
t
D
A
T
A
B
A
S
E
=
{JAVA_HOME:-/etc/alternatives/java_sdk_1.8.0} export DATABASE=
JAVAHOME:−/etc/alternatives/javasdk1.8.0exportDATABASE={DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=KaTeX parse error: Expected 'EOF', got '&' at position 108: …useUnicode=true&̲useSSL=false&ha…{SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=
S
P
R
I
N
G
J
A
C
K
S
O
N
T
I
M
E
Z
O
N
E
:
−
A
s
i
a
/
S
h
a
n
g
h
a
i
e
x
p
o
r
t
M
A
S
T
E
R
F
E
T
C
H
C
O
M
M
A
N
D
N
U
M
=
{SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai} export MASTER_FETCH_COMMAND_NUM=
SPRINGJACKSONTIMEZONE:−Asia/ShanghaiexportMASTERFETCHCOMMANDNUM={MASTER_FETCH_COMMAND_NUM:-10}
export REGISTRY_TYPE=
R
E
G
I
S
T
R
Y
T
Y
P
E
:
−
z
o
o
k
e
e
p
e
r
e
x
p
o
r
t
R
E
G
I
S
T
R
Y
Z
O
O
K
E
E
P
E
R
C
O
N
N
E
C
T
S
T
R
I
N
G
=
{REGISTRY_TYPE:-zookeeper} export REGISTRY_ZOOKEEPER_CONNECT_STRING=
REGISTRYTYPE:−zookeeperexportREGISTRYZOOKEEPERCONNECTSTRING={REGISTRY_ZOOKEEPER_CONNECT_STRING:-192.168.130.171:2181,192.168.130.172:2181,192.168.130.173:2181,192.168.130.174:2181,192.168.130.175:2181,192.168.130.176:2181}
export HADOOP_HOME=
H
A
D
O
O
P
H
O
M
E
:
−
/
u
s
r
/
b
i
g
t
o
p
/
c
u
r
r
e
n
t
/
h
a
d
o
o
p
−
c
l
i
e
n
t
e
x
p
o
r
t
H
A
D
O
O
P
C
O
N
F
D
I
R
=
{HADOOP_HOME:-/usr/bigtop/current/hadoop-client} export HADOOP_CONF_DIR=
HADOOPHOME:−/usr/bigtop/current/hadoop−clientexportHADOOPCONFDIR={HADOOP_CONF_DIR:-/usr/bigtop/current/hadoop-client/etc/hadoop}
export SPARK_HOME1=
S
P
A
R
K
H
O
M
E
1
:
−
/
o
p
t
/
s
o
f
t
/
s
p
a
r
k
1
e
x
p
o
r
t
S
P
A
R
K
H
O
M
E
2
=
{SPARK_HOME1:-/opt/soft/spark1} export SPARK_HOME2=
SPARKHOME1:−/opt/soft/spark1exportSPARKHOME2={SPARK_HOME2:-/opt/soft/spark2}
export PYTHON_HOME=
P
Y
T
H
O
N
H
O
M
E
:
−
/
o
p
t
/
s
o
f
t
/
p
y
t
h
o
n
e
x
p
o
r
t
H
I
V
E
H
O
M
E
=
{PYTHON_HOME:-/opt/soft/python} export HIVE_HOME=
PYTHONHOME:−/opt/soft/pythonexportHIVEHOME={HIVE_HOME:-/opt/soft/hive}
export FLINK_HOME=
F
L
I
N
K
H
O
M
E
:
−
/
o
p
t
/
s
o
f
t
/
f
l
i
n
k
e
x
p
o
r
t
D
A
T
A
X
H
O
M
E
=
{FLINK_HOME:-/opt/soft/flink} export DATAX_HOME=
FLINKHOME:−/opt/soft/flinkexportDATAXHOME={DATAX_HOME:-/opt/soft/datax}
export SEATUNNEL_HOME=
S
E
A
T
U
N
N
E
L
H
O
M
E
:
−
/
o
p
t
/
s
o
f
t
/
s
e
a
t
u
n
n
e
l
e
x
p
o
r
t
C
H
U
N
J
U
N
H
O
M
E
=
{SEATUNNEL_HOME:-/opt/soft/seatunnel} export CHUNJUN_HOME=
SEATUNNELHOME:−/opt/soft/seatunnelexportCHUNJUNHOME={CHUNJUN_HOME:-/opt/soft/chunjun}
export PATH=
H
A
D
O
O
P
H
O
M
E
/
b
i
n
:
HADOOP_HOME/bin:
HADOOPHOME/bin:SPARK_HOME1/bin:
S
P
A
R
K
H
O
M
E
2
/
b
i
n
:
SPARK_HOME2/bin:
SPARKHOME2/bin:PYTHON_HOME/bin:
J
A
V
A
H
O
M
E
/
b
i
n
:
JAVA_HOME/bin:
JAVAHOME/bin:HIVE_HOME/bin:
F
L
I
N
K
H
O
M
E
/
b
i
n
:
FLINK_HOME/bin:
FLINKHOME/bin:DATAX_HOME/bin:
S
E
A
T
U
N
N
E
L
H
O
M
E
/
b
i
n
:
SEATUNNEL_HOME/bin:
SEATUNNELHOME/bin:CHUNJUN_HOME/bin:$PATH
[root@bigtop06 env]# cat install_env.sh |grep -v ^# |grep -v ^$
ips=
i
p
s
:
−
"
192.168.130.171
,
192.168.130.172
,
192.168.130.173
,
192.168.130.174
,
192.168.130.175
,
192.168.130.176
"
s
s
h
P
o
r
t
=
{ips:-"192.168.130.171,192.168.130.172,192.168.130.173,192.168.130.174,192.168.130.175,192.168.130.176"} sshPort=
ips:−"192.168.130.171,192.168.130.172,192.168.130.173,192.168.130.174,192.168.130.175,192.168.130.176"sshPort={sshPort:-“22”}
masters=
m
a
s
t
e
r
s
:
−
"
192.168.130.176
"
w
o
r
k
e
r
s
=
{masters:-"192.168.130.176"} workers=
masters:−"192.168.130.176"workers={workers:-“192.168.130.171:default,192.168.130.172:default,192.168.130.173:default,192.168.130.174:default,192.168.130.175:default,192.168.130.176:default”}
alertServer=
a
l
e
r
t
S
e
r
v
e
r
:
−
"
192.168.130.176
"
a
p
i
S
e
r
v
e
r
s
=
{alertServer:-"192.168.130.176"} apiServers=
alertServer:−"192.168.130.176"apiServers={apiServers:-“192.168.130.176”}
installPath=
i
n
s
t
a
l
l
P
a
t
h
:
−
"
/
o
p
t
/
a
p
a
c
h
e
−
d
o
l
p
h
i
n
s
c
h
e
d
u
l
e
r
−
3.1.8
−
b
i
n
"
d
e
p
l
o
y
U
s
e
r
=
{installPath:-"/opt/apache-dolphinscheduler-3.1.8-bin"} deployUser=
installPath:−"/opt/apache−dolphinscheduler−3.1.8−bin"deployUser={deployUser:-“root”}
zkRoot=${zkRoot:-“/dolphinscheduler”}
放置mysql-connector-java-8.0.16.jar到对用服务libs库中
[root@bigtop06 bin]# cd /opt/
[root@bigtop06 opt]# ll
total 662684
drwxrwxrwx 11 root root 190 Feb 18 2022 apache-dolphinscheduler-3.1.8-bin
-rw-r–r-- 1 root root 674806497 Oct 19 16:52 apache-dolphinscheduler-3.1.8-bin.tar.gz
drwxr-xr-x 3 root root 149 Mar 21 2019 mysql-connector-java-8.0.16
-rw-r–r-- 1 root root 3777925 Oct 19 16:59 mysql-connector-java-8.0.16.tar.gz
drwxr-xr-x. 2 root root 6 Oct 31 2018 rh
drwxr-xr-x. 3 root root 182 Sep 13 10:30 vmtools
[root@bigtop06 opt]# cd mysql-connector-java-8.0.16/
[root@bigtop06 mysql-connector-java-8.0.16]# ll
total 2692
-rw-r–r-- 1 root root 87863 Mar 21 2019 build.xml
-rw-r–r-- 1 root root 260906 Mar 21 2019 CHANGES
-rw-r–r-- 1 root root 183 Mar 21 2019 INFO_BIN
-rw-r–r-- 1 root root 135 Mar 21 2019 INFO_SRC
-rw-r–r-- 1 root root 97188 Mar 21 2019 LICENSE
-rw-r–r-- 1 root root 2293144 Mar 21 2019 mysql-connector-java-8.0.16.jar
-rw-r–r-- 1 root root 1264 Mar 21 2019 README
drwxr-xr-x 8 root root 86 Mar 21 2019 src
[root@bigtop06 mysql-connector-java-8.0.16]# find / -name mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/alert-server/libs/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/api-server/libs/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/master-server/libs/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/worker-server/libs/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/standalone-server/libs/standalone-server/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/standalone-server/libs/mysql-connector-java-8.0.16.jar
/opt/apache-dolphinscheduler-3.1.8-bin/tools/libs/mysql-connector-java-8.0.16.jar
初始化数据库
[root@bigtop06 env]# cd /opt/apache-dolphinscheduler-3.1.8-bin/tools/bin/
[root@bigtop06 bin]# ./upgrade-schema.sh
一键安装dolphinscheduler集群
[root@bigtop06 mysql-connector-java-8.0.16]# cd /opt/apache-dolphinscheduler-3.1.8-bin/bin/
[root@bigtop06 bin]# ll
total 32
-rwxrwxrwx 1 root root 4513 Oct 20 15:12 dolphinscheduler-daemon.sh
drwxrwxrwx 2 root root 59 Oct 20 16:34 env
-rwxrwxrwx 1 root root 1937 Oct 20 15:12 install.sh
-rwxrwxrwx 1 root root 1795 Oct 20 15:12 remove-zk-node.sh
-rwxrwxrwx 1 root root 2257 Oct 20 15:12 scp-hosts.sh
-rwxrwxrwx 1 root root 2008 Oct 20 15:12 start-all.sh
-rwxrwxrwx 1 root root 2808 Oct 20 15:12 status-all.sh
-rwxrwxrwx 1 root root 1912 Oct 20 15:12 stop-all.sh
[root@bigtop06 bin]# bash +x install.sh
确认主节点服务状态
[root@master libs]# jps
14145 AlertServer
14179 ApiApplicationServer
13996 QuorumPeerMain
14079 MasterServer
14111 WorkerServer
15295 Jps
[root@master libs]# netstat -ntlp |grep 123
tcp6 0 0 :::1234 ::😗 LISTEN 14111/java
tcp6 0 0 :::1235 ::😗 LISTEN 14111/java
tcp6 0 0 :::12345 ::😗 LISTEN 14179/java
其他节点默认只有WorkerServer服务
访问方式
http://192.168.130.176:12345/dolphinscheduler/ui/home
admin dolphinscheduler123
安装包目录 /opt/apache-dolphinscheduler-3.1.8-bin
192.168.130.171-176 176是master节点，其他是worker节点
部署遇到的问题：
解决方法：
修改数据库连接地址，添加useSSL=false
export SPRING_DATASOURCE_URL=“jdbc:mysql://192.168.130.176:3306/dolphinscheduler?useUnicode=true&useSSL=false&haracterEncoding=UTF-8”
参考文献
https://dolphinscheduler.apache.org/zh-cn/docs/3.1.8/guide/installation/pseudo-cluster
https://blog.51cto.com/feishujun/6149243
https://blog.csdn.net/qq_44936258/article/details/132582516
扫一扫
629
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
