当前版本基于1.2.0实现,DS其他版本也可以使用相同思路完成
概述
DS安装过程中,需要的DS配置非常多,总结归类一共有六大部分。第一部分是零散的配置部分,包括DS安装路径、zk地址、部署用户等信息;第二部分主要是数据源参数配置;第三部分主要是邮件参数配置;第四部分是文件系统和yarn配置;第五部分是DS服务期部署机器配置;第六部分是DS服务环境变量配置。由于DS每个配置部分又有很多,对于部署人员来说很容易配置操作失误,不是很友好,故准备使用shell脚本方式实现向导模式。
思路:
DS的配置文件信息主要在install.sh和conf/env/.dolphinscheduler_env.sh两个脚本当中,简单来说,我们只需要提示用户输入指定参数,然后将部署人员输入的参数替换调这两个文件里面的参数,一键运行的时候使用的就是用户想要的DS配置参数了。
详细设计
流程图
流程描述
- 开始执行向导脚本;
- 进入第一部分配置;
- 首先检查第一个参数是否有值;
- 如果没有值,输入参数;如果有值,是否覆盖,覆盖的话输入参数值;
- 获取参数值,继续处理下一个参数,直至当前部分参数值处理结束;
- 当前部分处理结束,持久化参数值到配置文件,开始处理下一个部分配置,直至第六部分结束;
- 第六部分配置结束,持久化第六部分;
- 开始正式安装DS服务,执行install.sh文件;
- 服务端安装结束。
实现核心
检查参数原值
我们在替换参数值之前需要判断是否需要使用当前使用的参数值,所以我们需要获取以前老的参数值,这里使用了linux的awk命令,根据“"”分割参数名和参数值,然后获取老的参数值。
获取install.sh文件中zk地址,读取到zkQuorumOld参数 (install.sh:zkQuorum=“192.168.xx.xx:2181,192.168.xx.xx:2181,192.168.xx.xx:2181”)
eval $(awk -F"\"" '{if($1~/zkQuorum=/) printf ("zkQuorumOld=%s",$2)}' ./install.sh)
输入新的参数值
我们需要提示用户输入新的参数值 或者使用老的参数值,这个时候可以使用linux read命令。
读取用户zk地址输入demo
read -p "1.1 原Zookeeper地址为'${zkQuorumOld}', 回车使用原地址, 为空时使用默认地址(localhost:2181), 请输入Zookeeper地址 :" zkQuorum
是否替换判断:
如果输入参数为空,并且老的参数也为空,那么使用默认值为“localhost:2181”,如果输入值为空,老的参数不为空,使用老的参数,如果输入值不为空,使用输入值。
if [ -z "${zkQuorum}" -a -z "${zkQuorumOld}" ];then
zkQuorum="localhost:2181"
elif [ -z "${zkQuorum}" -a -n "${zkQuorumOld}" ]; then
zkQuorum="${zkQuorumOld}"
fi
替换文件的原值
获取到最终的的参数值之后,需要替换掉原文件的值,最后用用户配置的参数去实际运行。这个时候可以使用linux的sed命令完成目标。
sed -i ${txt} "79s#zkQuorum=.*#zkQuorum=\"${zkQuorum}\"#g" ./install.sh
结束
我们替换完成参数之后直接执行sh install.sh即可完成DS项目的最后一步安装工作。
使用DS服务一键部署脚本,再也不用担心配置错误导致任务运行失败啦!
参考:
附完整脚本
setup.sh
#!/bin/bash
#To be compatible with MacOS and Linux
txt=""
if [[ "$OSTYPE" == "darwin"* ]]; then
# Mac OSX
txt="''"
elif [[ "$OSTYPE" == "linux-gnu" ]]; then
# linux
txt=""
elif [[ "$OSTYPE" == "cygwin" ]]; then
# POSIX compatibility layer and Linux environment emulation for Windows
echo "DolphinScheduler not support Windows operating system"
exit 1
elif [[ "$OSTYPE" == "msys" ]]; then
# Lightweight shell and GNU utilities compiled for Windows (part of MinGW)
echo "DolphinScheduler not support Windows operating system"
exit 1
elif [[ "$OSTYPE" == "win32" ]]; then
echo "DolphinScheduler not support Windows operating system"
exit 1
elif [[ "$OSTYPE" == "freebsd"* ]]; then
# ...
txt=""
else
# Unknown.
echo "Operating system unknown, please tell us(submit issue) for better service"
exit 1
fi
function part1(){
#Zookeeper地址,单机本机是localhost:2181,记得把2181端口带上
eval $(awk -F"\"" '{if($1~/zkQuorum=/) printf ("zkQuorumOld=%s",$2)}' ./install.sh)
read -p "1.1 原Zookeeper地址为'${zkQuorumOld}', 回车使用原地址, 为空时使用默认地址(localhost:2181), 请输入Zookeeper地址 :" zkQuorum
if [ -z "${zkQuorum}" -a -z "${zkQuorumOld}" ];then
zkQuorum="localhost:2181"
elif [ -z "${zkQuorum}" -a -n "${zkQuorumOld}" ]; then
zkQuorum="${zkQuorumOld}"
fi
#将DS安装到哪个目录,如: /opt/soft/dolphinscheduler,不同于现在的目录
eval $(awk -F"\"" '{if($1~/installPath=/) printf ("installPathOld=%s",$2)}' ./install.sh)
read -p "1.2 原DS安装目录为'${installPathOld}', 回车使用原安装目录, 请输入DS安装目录 :" installPath
while true
do
if [ -z "${installPath}" -a -z "${installPathOld}" ];then
echo -e "\033[31mDS安装目录不能为空,请重新输入!\033[0m"
continue
elif [ -z "${installPath}" ];then
installPath="${installPathOld}"
break
else
break
fi
done
#使用哪个用户部署
eval $(awk -F"\"" '{if($1~/deployUser=/) printf ("deployUserOld=%s",$2)}' ./install.sh)
read -p "1.3 原部署用户为'${deployUserOld}', 回车使用原部署用户, 为空时使用默认用户(dolphinscheduler), 请输入部署用户 :" deployUser
if [ -z "${deployUser}" -a -z "${deployUserOld}" ];then
deployUser="dolphinscheduler"
elif [ -z "${deployUser}" -a -n "${deployUserOld}" ]; then
deployUser="${deployUserOld}"
fi
#excel下载路径
eval $(awk -F"\"" '{if($1~/xlsFilePath=/) printf ("xlsFilePathOld=%s",$2)}' ./install.sh)
read -p "1.4 原excel下载路径为'${xlsFilePathOld}', 回车使用原路径, 为空时使用默认路径(/tmp/xls), 请输入excel下载路径 :" xlsFilePath
if [ -z "${xlsFilePath}" -a -z "${xlsFilePathOld}" ];then
xlsFilePath="/tmp/xls"
elif [ -z "${xlsFilePath}" -a -n "${xlsFilePathOld}" ]; then
xlsFilePath="${xlsFilePathOld}"
fi
echo -e "\033[33m第一部分通用参数配置结束!\033[0m"
echo "程序将自动自动持久化..."
persistPart1
echo -e "\033[33m第一部分持久化成功!\033[0m"
}
function persistPart1() {
sed -i ${txt} "79s#zkQuorum=.*#zkQuorum=\"${zkQuorum}\"#g" ./install.sh
sed -i ${txt} "72s#installPath=.*#installPath=\"${installPath}\"#g" ./install.sh
sed -i ${txt} "76s#deployUser=.*#deployUser=\"${deployUser}\"#g" ./install.sh
sed -i ${txt} "132s#xlsFilePath=.*#xlsFilePath=\"${xlsFilePath}\"#g" ./install.sh
}
function part2(){
eval $(awk -F"\"" '{if($1~/dbtype=/) printf ("dbtypeOld=%s",$2)}' ./install.sh)
read -p "2.1 原数据库类型为'${dbtypeOld}', 回车使用原类型, 为空时使用默认类型(mysql), 支持mysql和postgresql, 请输入数据库类型 :" dbtype
while true
do
if [ -z "${dbtype}" -a -z "${dbtypeOld}" ];then
dbtype="mysql"
break
elif [ -z "${dbtype}" ];then
dbtype="${dbtypeOld}"
break
elif [ "${dbtype}" != "mysql" -a "${dbtype}" != "postgresql" ];then
echo -e "\033[31m数据库类型只支持mysql和postgresql,请重新输入!\033[0m"
continue
else
break
fi
done
eval $(awk -F"\"" '{if($1~/dbhost=/) printf ("dbhostOld=%s",$2)}' ./install.sh)
read -p "2.2 原数据库连接地址为'${dbhostOld}', 回车使用原地址, 为空时使用默认地址(localhost:3306), 请输入地址 :" dbhost
if [ -z "${dbhost}" -a -z "${dbhostOld}" ];then
dbhost="localhost:3306"
elif [ -z "${dbhost}" -a -n "${dbhostOld}" ]; then
dbhost="${dbhostOld}"
fi
eval $(awk -F"\"" '{if($1~/dbname=/) printf ("dbnameOld=%s",$2)}' ./install.sh)
read -p "2.3 原数据库名为'${dbnameOld}', 回车使用原名, 为空时使用默认数据库名(dolphinscheduler), 请输入数据库名 :" dbname
if [ -z "${dbname}" -a -z "${dbnameOld}" ];then
dbname="dolphinscheduler"
elif [ -z "${dbname}" -a -n "${dbnameOld}" ]; then
dbname="${dbnameOld}"
fi
eval $(awk -F"\"" '{if($1~/username=/) printf ("usernameOld=%s",$2)}' ./install.sh)
read -p "2.4 原数据库用户名为'${usernameOld}', 回车使用原用户名, 请输入数据库用户名 :" username
while true
do
if [ -z "${username}" -a -z "${usernameOld}" ];then
echo -e "\033[31mDS数据库用户名不能为空,请重新输入!\033[0m"
continue
elif [ -z "${username}" ];then
username="${usernameOld}"
break
else
break
fi
done
eval $(awk -F"\"" '{if($1~/passowrd=/) printf ("passowrdOld=%s",$2)}' ./install.sh)
read -p "2.5 原数据库密码为'${passowrdOld}', 回车使用原密码, 请输入密码 :" passowrd
if [ -z "${passowrd}" -a -n "${passowrdOld}" ]; then
passowrd="${passowrdOld}"
fi
echo -e "\033[33m第二部分数据源参数配置结束!\033[0m"
echo "程序将自动自动持久化..."
persistPart2
echo -e "\033[33m第二部分持久化成功!\033[0m"
}
function persistPart2() {
sed -i ${txt} "54s#dbtype=.*#dbtype=\"${dbtype}\"#g" ./install.sh
sed -i ${txt} "58s#dbhost=.*#dbhost=\"${dbhost}\"#g" ./install.sh
sed -i ${txt} "61s#dbname=.*#dbname=\"${dbname}\"#g" ./install.sh
sed -i ${txt} "64s#username=.*#username=\"${username}\"#g" ./install.sh
sed -i ${txt} "68s#passowrd=.*#passowrd=\"${passowrd}\"#g" ./install.sh
}
function part3(){
# 邮件配置,以qq邮箱为例
# 邮件协议
eval $(awk -F"\"" '{if($1~/mailProtocol=/) printf ("mailProtocolOld=%s",$2)}' ./install.sh)
read -p "3.1 原邮件协议为'${mailProtocolOld}', 回车使用原协议, 为空时使用默认协议(SMTP), 请输入邮件协议 :" mailProtocol
if [ -z "${mailProtocol}" -a -z "${mailProtocolOld}" ];then
mailProtocol="SMTP"
elif [ -z "${mailProtocol}" -a -n "${mailProtocolOld}" ]; then
mailProtocol="${mailProtocolOld}"
fi
# 邮件服务地址
eval $(awk -F"\"" '{if($1~/mailServerHost=/) printf ("mailServerHostOld=%s",$2)}' ./install.sh)
read -p "3.2 原邮件服务地址为'${mailServerHostOld}', 回车使用原地址, 为空时使用默认地址(smtp.qq.com), 请输入邮件服务地址 :" mailServerHost
if [ -z "${mailServerHost}" -a -z "${mailServerHostOld}" ];then
mailServerHost="smtp.qq.com"
elif [ -z "${mailServerHost}" -a -n "${mailServerHostOld}" ]; then
mailServerHost="${mailServerHostOld}"
fi
sslTrust="${mailServerHost}"
# 邮件邮件服务端口
eval $(awk -F"\"" '{if($1~/mailServerPort=/) printf ("mailServerPortOld=%s",$2)}' ./install.sh)
read -p "3.3 原邮件服务端口为'${mailServerPortOld}', 回车使用原端口, 为空时使用默认端口(25), 请输入邮件服务端口 :" mailServerPort
if [ -z "${mailServerPort}" -a -z "${mailServerPortOld}" ];then
mailServerPort="25"
elif [ -z "${mailServerPort}" -a -n "${mailServerPortOld}" ]; then
mailServerPort="${mailServerPortOld}"
fi
# 发送者
eval $(awk -F"\"" '{if($1~/mailSender=/) printf ("mailSenderOld=%s",$2)}' ./install.sh)
read -p "3.4 原邮件发送者为'${mailSenderOld}', 回车使用原发送者, 请输入邮件发送者 :" mailSender
while true
do
if [ -z "${mailSender}" -a -z "${mailSenderOld}" ];then
echo -e "\033[31m邮件发送者不能为空,请重新输入!\033[0m"
continue
elif [ -z "${mailSender}" ];then
mailSender="${mailSenderOld}"
break
else
break
fi
done
mailUser=${mailSender}
# 邮箱密码
eval $(awk -F"\"" '{if($1~/mailPassword=/) printf ("mailPasswordOld=%s",$2)}' ./install.sh)
read -p "3.5 原邮箱密码为'${mailPasswordOld}', 回车使用原密码, 请输入邮箱密码 :" mailPassword
while true
do
if [ -z "${mailPassword}" -a -z "${mailPasswordOld}" ];then
echo -e "\033[31m邮箱密码不能为空,请重新输入!\033[0m"
continue
elif [ -z "${mailPassword}" ];then
mailPassword="${mailPasswordOld}"
break
else
break
fi
done
# 邮箱TLS协议设置为true,否则设置为false
eval $(awk -F"\"" '{if($1~/starttlsEnable=/) printf ("starttlsEnableOld=%s",$2)}' ./install.sh)
read -p "3.6 原邮箱TLS协议设置为'${starttlsEnableOld}', 回车使用原设置, 请输入邮箱TLS协议设置 :" starttlsEnable
while true
do
if [ -z "${starttlsEnable}" -a -z "${starttlsEnableOld}" ];then
starttlsEnable="true"
break
elif [ -z "${starttlsEnable}" ];then
starttlsEnable="${starttlsEnableOld}"
if [ "${starttlsEnable}" != "true" -a "${starttlsEnable}" != "false" ];then
echo -e "\033[31m邮箱TLS协议设置只支持true和false,请重新输入!\033[0m"
continue
fi
break
else
break
fi
done
#邮箱SSL协议设置
if [ "${starttlsEnable}" == "true" ]; then
sslEnable="false"
else
eval $(awk -F"\"" '{if($1~/sslEnable=/) printf ("sslEnableOld=%s",$2)}' ./install.sh)
read -p "3.6.1 原邮箱SSL协议设置为'${sslEnableOld}', 回车使用原设置, 请输入邮箱SSL协议设置 :" sslEnable
while true
do
if [ -z "${sslEnable}" -a -z "${sslEnableOld}" ];then
sslEnable="true"
break
elif [ -z "${sslEnable}" ];then
sslEnable="${sslEnableOld}"
break
else
if [ "${sslEnable}" != "true" -a "${sslEnable}" != "false" ];then
echo -e "\033[31m邮箱SSL协议设置设置只支持true和false,请重新输入!\033[0m"
continue
fi
break
fi
done
fi
echo -e "\033[33m第三部分邮件参数配置结束!\033[0m"
echo "程序将自动自动持久化..."
persistPart3
echo -e "\033[33m第三部分持久化成功!\033[0m"
}
function persistPart3() {
sed -i ${txt} "104s#mailProtocol=.*#mailProtocol=\"${mailProtocol}\"#g" ./install.sh
sed -i ${txt} "107s#mailServerHost=.*#mailServerHost=\"${mailServerHost}\"#g" ./install.sh
sed -i ${txt} "124s#sslTrust=.*#sslTrust=\"${sslTrust}\"#g" ./install.sh
sed -i ${txt} "110s#mailServerPort=.*#mailServerPort=\"${mailServerPort}\"#g" ./install.sh
sed -i ${txt} "113s#mailSender=.*#mailSender=\"${mailSender}\"#g" ./install.sh
sed -i ${txt} "116s#mailUser=.*#mailUser=\"${mailUser}\"#g" ./install.sh
sed -i ${txt} "119s#mailPassword=.*#mailPassword=\"${mailPassword}\"#g" ./install.sh
sed -i ${txt} "122s#starttlsEnable=.*#starttlsEnable=\"${starttlsEnable}\"#g" ./install.sh
sed -i ${txt} "129s#sslEnable=.*#sslEnable=\"${sslEnable}\"#g" ./install.sh
}
function part4(){
# 业务用到的比如sql等资源文件上传到哪里,可以设置:HDFS,S3,NONE,单机如果想使用本地文件系统,请配置为HDFS,因为HDFS支持本地文件系统;如果不需要资源上传功能请选择NONE。强调一点:使用本地文件系统不需要部署hadoop
#文件系统类型
eval $(awk -F"\"" '{if($1~/resUploadStartupType=/) printf ("resUploadStartupTypeOld=%s",$2)}' ./install.sh)
read -p "4.1 原文件系统类型为'${resUploadStartupTypeOld}', 回车使用原类型, 为空时默认(NONE), 只支持HDFS/S3/NONE, 请输入类型 :" resUploadStartupType
while true
do
if [ -z "${resUploadStartupType}" -a -z "${resUploadStartupTypeOld}" ];then
resUploadStartupType="NONE"
break
elif [ -z "${resUploadStartupType}" ];then
resUploadStartupType="${resUploadStartupTypeOld}"
break
else
if [ "${resUploadStartupType}" != "HDFS" -a "${resUploadStartupType}" != "S3" -a "${resUploadStartupType}" != "NONE" ];then
echo -e "\033[31m文件系统类型只支持HDFS/S3/NONE,请重新输入!\033[0m"
continue
fi
break
fi
done
if [ "${resUploadStartupType}" == "HDFS" ]; then
eval $(awk -F"\"" '{if($1~/defaultFS=/) printf ("defaultFSOld=%s",$2)}' ./install.sh)
read -p "4.1.1 原文件系统地址为'${defaultFSOld}', 回车使用原地址, 上传HDFS的话,NameNode启用了HA,需要将hadoop的配置文件core-site.xml和hdfs-site.xml copy到conf目录,并输入namenode cluster名称(eg hdfs://{具体的ip/主机名}:8020);如果NameNode不是HA,则修改为具体的ip或者主机名即可, 为空时默认(file:///data/dolphinscheduler), 请输入文件系统地址 :" defaultFS
if [ -z "${defaultFS}" -a -z "${defaultFSOld}" ];then
defaultFS="file:///data/dolphinscheduler"
elif [ -z "${defaultFS}" -a -n "${defaultFSOld}" ]; then
defaultFS="${defaultFSOld}"
fi
fi
# 如果没有使用到Ya没有使用到Yarn,保持以下默认值即可;如果ResourceManager是HA,则配置为ResourceManager节点的主备ip或者hostname,比如"192.168.xx.xx,192.168.xx.xx";如果是单ResourceManager请配置yarnHaIps=""即可
# read -p "4.2 请输入Yarn ip, 如果ResourceManager是HA,则配置为ResourceManager节点的主备ip或者hostname,比如'192.168.xx.xx,192.168.xx.xx';如果是单ResourceManager或rn,保持以下默认值即可;如果ResourceManager是HA,则配置为ResourceManager节点的主备ip或者hostname,比如"192.168.xx.xx,192.168.xx.xx";如果是单ResourceManager请配置yarnHaIps=""即可
eval $(awk -F"\"" '{if($1~/yarnHaIps=/) printf ("yarnHaIpsOld=%s",$2)}' ./install.sh)
read -p "4.2 原yarnHaIps为'${yarnHaIpsOld}', 如果ResourceManager是HA,则配置为ResourceManager节点的主备ip或者hostname,比如'192.168.xx.xx,192.168.xx.xx';如果是单ResourceManager或没有使用到Yarn, 回车置空, 请输入yarnHaIps :" yarnHaIps
if [ -z "${yarnHaIps}" ];then
yarnHaIps=""
fi
# 如果ResourceManager是HA或者没有使用到Yarn保持默认值即可;如果是单ResourceManager,请配置真实的ResourceManager主机名或者ip
eval $(awk -F"\"" '{if($1~/singleYarnIp=/) printf ("singleYarnIpOld=%s",$2)}' ./install.sh)
read -p "4.3 原singleYarnIp为'${singleYarnIpOld}', 如果没有使用到Yarn, 回车即可, 请输入singleYarnIp :" singleYarnIp
if [ -z "${yarnHaIps}" ];then
singleYarnIp="yarnIp1"
fi
# 资源上传根路径,支持HDFS和S3,由于hdfs支持本地文件系统,需要确保本地文件夹存在且有读写权限
eval $(awk -F"\"" '{if($1~/hdfsPath=/) printf ("hdfsPathOld=%s",$2)}' ./install.sh)
read -p "4.4 原资源上传根路径为'${hdfsPathOld}', 资源上传支持HDFS和S3,由于hdfs支持本地文件系统,需要确保本地文件夹存在且有读写权限,回车使用原路径, 为空时使用默认地址(/data/dolphinscheduler), 请输入路径 :" hdfsPath
if [ -z "${hdfsPath}" -a -z "${hdfsPathOld}" ];then
hdfsPath="/data/dolphinscheduler"
elif [ -z "${hdfsPath}" -a -n "${hdfsPathOld}" ]; then
hdfsPath="${hdfsPathOld}"
fi
echo -e "\033[33m第四部分文件系统和yarn参数配置结束!\033[0m"
echo "程序将自动自动持久化..."
persistPart4
echo -e "\033[33m第四部分持久化成功!\033[0m"
}
function persistPart4() {
sed -i ${txt} "154s#resUploadStartupType=.*#resUploadStartupType=\"${resUploadStartupType}\"#g" ./install.sh
sed -i ${txt} "159s#defaultFS=.*#defaultFS=\"${defaultFS}\"#g" ./install.sh
sed -i ${txt} "167s#yarnHaIps=.*#yarnHaIps=\"${yarnHaIps}\"#g" ./install.sh
sed -i ${txt} "170s#singleYarnIp=.*#singleYarnIp=\"${singleYarnIp}\"#g" ./install.sh
sed -i ${txt} "174s#hdfsPath=.*#hdfsPath=\"${hdfsPath}\"#g" ./install.sh
}
function part5() {
# 在哪些机器上部署DS服务,本机选localhost
eval $(awk -F"\"" '{if($1~/ips=/) printf ("ipsOld=%s",$2)}' ./install.sh)
read -p "5.1 原部署DS服务的机器ips'${ipsOld}', 回车使用原ips, 为空时默认(localhost), 请输入DS服务的机器ips :" ips
if [ -z "${ips}" -a -z "${ipsOld}" ];then
ips="localhost"
elif [ -z "${ips}" -a -n "${ipsOld}" ]; then
ips="${ipsOld}"
fi
# master服务部署在哪台机器上
eval $(awk -F"\"" '{if($1~/masters=/) printf ("mastersOld=%s",$2)}' ./install.sh)
read -p "5.2 原部署master服务机器ips'${mastersOld}', 回车使用原ips, 为空时默认(localhost), 请输入master服务机器ips :" masters
if [ -z "${masters}" -a -z "${mastersOld}" ];then
masters="localhost"
elif [ -z "${masters}" -a -n "${mastersOld}" ]; then
masters="${mastersOld}"
fi
# worker服务部署在哪台机器上
eval $(awk -F"\"" '{if($1~/workers=/) printf ("workersOld=%s",$2)}' ./install.sh)
read -p "5.3 原部署worker服务机器ips'${workersOld}', 回车使用原ips, 为空时默认(localhost), 请输入worker服务机器ips :" workers
if [ -z "${workers}" -a -z "${workersOld}" ];then
workers="localhost"
elif [ -z "${workers}" -a -n "${workersOld}" ]; then
workers="${workersOld}"
fi
# 报警服务部署在哪台机器上
eval $(awk -F"\"" '{if($1~/alertServer=/) printf ("alertServerOld=%s",$2)}' ./install.sh)
read -p "5.4 原部署报警服务机器ip'${alertServerOld}', 回车使用原ips, 为空时默认(localhost), 请输入报警服务机器ip :" alertServer
if [ -z "${alertServer}" -a -z "${alertServerOld}" ];then
alertServer="localhost"
elif [ -z "${alertServer}" -a -n "${alertServerOld}" ]; then
alertServer="${alertServerOld}"
fi
# 后端api服务部署在哪台机器上
eval $(awk -F"\"" '{if($1~/apiServers=/) printf ("apiServersOld=%s",$2)}' ./install.sh)
read -p "5.5 原部署后端api服务机器ip'${apiServersOld}', 回车使用原ips, 为空时默认(localhost), 请输入后端api机器ip :" apiServers
if [ -z "${apiServers}" -a -z "${apiServersOld}" ];then
apiServers="localhost"
elif [ -z "${apiServers}" -a -n "${apiServersOld}" ]; then
apiServers="${apiServersOld}"
fi
echo -e "\033[33m第五部分DS服务部署机器配置结束!\033[0m"
echo "程序将自动自动持久化..."
persistPart5
echo -e "\033[33m第五部分持久化成功!\033[0m"
}
function persistPart5() {
sed -i ${txt} "83s#ips=.*#ips=\"${ips}\"#g" ./install.sh
sed -i ${txt} "88s#masters=.*#masters=\"${masters}\"#g" ./install.sh
sed -i ${txt} "92s#workers=.*#workers=\"${workers}\"#g" ./install.sh
sed -i ${txt} "96s#alertServer=.*#alertServer=\"${alertServer}\"#g" ./install.sh
sed -i ${txt} "100s#apiServers=.*#apiServers=\"${apiServers}\"#g" ./install.sh
}
function part6() {
# HADOOP_HOME
ENV_PATH=""
read -p "6.1 请输入HADOOP_HOME环境变量,不需要请按回车键进入下一项 :" HADOOP_HOME
if [ -n "${HADOOP_HOME}" ];then
ENV_PATH="\$HADOOP_HOME/bin:"
fi
# HADOOP_CONF_DIR
read -p "6.2 请输入HADOOP_CONF_DIR目录,不需要请按回车键进入下一项 :" HADOOP_CONF_DIR
# SPARK_HOME1
read -p "6.3 请输入SPARK_HOME1环境变量,不需要请按回车键进入下一项 :" SPARK_HOME1
if [ -n "${SPARK_HOME1}" ];then
ENV_PATH=${ENV_PATH}"\$SPARK_HOME1/bin:"
fi
# SPARK_HOME2
read -p "6.4 请输入SPARK_HOME2环境变量,不需要请按回车键进入下一项 :" SPARK_HOME2
if [ -n "${SPARK_HOME2}" ];then
ENV_PATH=${ENV_PATH}"\$SPARK_HOME2/bin:"
fi
# PYTHON_HOME
read -p "6.5 请输入PYTHON_HOME环境变量,不需要请按回车键进入下一项 :" PYTHON_HOME
if [ -n "${PYTHON_HOME}" ];then
ENV_PATH=${ENV_PATH}"\$PYTHON_HOME:"
fi
# JAVA_HOME
while true
do
read -p "6.6 请输入JAVA_HOME环境变量 :" JAVA_HOME
if [ -z "${JAVA_HOME}" ];then
echo -e "\033[31mjava环境变量不能为空,请重新输入!\033[0m"
continue
else
ENV_PATH=${ENV_PATH}"\$JAVA_HOME/bin:"
break
fi
done
# HIVE_HOME
read -p "6.7 请输入HIVE_HOME环境变量,不需要请按回车键进入下一项 :" HIVE_HOME
if [ -n "${HIVE_HOME}" ];then
ENV_PATH=${ENV_PATH}"\$HIVE_HOME/bin:"
fi
# FLINK_HOME
read -p "6.8 请输入FLINK_HOME环境变量,不需要请按回车键结束 :" FLINK_HOME
if [ -n "${FLINK_HOME}" ];then
ENV_PATH=${ENV_PATH}"\$FLINK_HOME/bin:"
fi
ENV_PATH=${ENV_PATH}"\$PATH"
echo "程序将自动自动持久化..."
persistPart6
echo -e "\033[33m第六部分持久化成功!\033[0m"
}
function persistPart6() {
if [ -n "${HADOOP_HOME}" ]; then
sed -i ${txt} "s#.*export HADOOP_HOME=.*#export HADOOP_HOME=${HADOOP_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export HADOOP_HOME=.*#\#export HADOOP_HOME=${HADOOP_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
fi
if [ -n "${HADOOP_CONF_DIR}" ]; then
sed -i ${txt} "s#.*export HADOOP_CONF_DIR=.*#export HADOOP_CONF_DIR=${HADOOP_CONF_DIR}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export HADOOP_CONF_DIR=.*#\#export HADOOP_CONF_DIR=${HADOOP_CONF_DIR}#g" ./conf/env/.dolphinscheduler_env.sh
fi
if [ -n "${SPARK_HOME1}" ]; then
sed -i ${txt} "s#.*export SPARK_HOME1=.*#export SPARK_HOME1=${SPARK_HOME1}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export SPARK_HOME1=.*#\#export SPARK_HOME1=${SPARK_HOME1}#g" ./conf/env/.dolphinscheduler_env.sh
fi
if [ -n "${SPARK_HOME2}" ]; then
sed -i ${txt} "s#.*export SPARK_HOME2=.*#export SPARK_HOME2=${SPARK_HOME2}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export SPARK_HOME2=.*#\#export SPARK_HOME2=${SPARK_HOME2}#g" ./conf/env/.dolphinscheduler_env.sh
fi
if [ -n "${PYTHON_HOME}" ]; then
sed -i ${txt} "s#.*export PYTHON_HOME=.*#export PYTHON_HOME=${PYTHON_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export PYTHON_HOME=.*#\#export PYTHON_HOME=${PYTHON_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
fi
sed -i ${txt} "s#export JAVA_HOME=.*#export JAVA_HOME=${JAVA_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
if [ -n "${HIVE_HOME}" ]; then
sed -i ${txt} "s#.*export HIVE_HOME=.*#export HIVE_HOME=${HIVE_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export HIVE_HOME=.*#\#export HIVE_HOME=${HIVE_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
fi
if [ -n "${FLINK_HOME}" ]; then
sed -i ${txt} "s#.*export FLINK_HOME=.*#export FLINK_HOME=${FLINK_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
else
sed -i ${txt} "s#.*export FLINK_HOME=.*#\#export FLINK_HOME=${FLINK_HOME}#g" ./conf/env/.dolphinscheduler_env.sh
fi
sed -i ${txt} "s#export PATH=.*#export PATH=${ENV_PATH}#g" ./conf/env/.dolphinscheduler_env.sh
}
function main(){
echo "Welcome to dolphinscheduler deployment script!"
echo "第一部分 —— DS通用参数配置:"
part1
# 数据源参数配置:
echo "第二部分 —— 数据源参数配置:"
part2
echo "第三部分 —— 邮件参数配置:"
part3
echo "第四部分 —— 文件系统和yarn配置:"
part4
echo "第五部分 —— DS服务部署机器配置:"
part5
echo "第六部分 —— DS服务环境变量配置,不存在的请直接回车:"
part6
sh install.sh
}
main