需求说明:
项目和生产环境越来越多,项目的测试发布和线上发布任务繁重
本方案使用Gitlab+Jenkins实现测试环境自动构建和生产多环境手动控制发布
实验主机列表和功能:
192.168.77.100 CentOS7 gitlab
192.168.77.130 CentOS7 jenkins+nginx
192.168.77.200 CentOS6 测试环境主机
192.168.77.211 CentOS6 生产环境主机1
192.168.77.212 CentOS6 生产环境主机2
Gitlab环境搭建:
依据《CentOS7实验机模板搭建部署》克隆实验机,IP:192.168.77.100,部署Gitlab并装入测试用项目:
HOSTNAME=gitlab
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts
cd /tmp
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.8.6-ce.0.el7.x86_64.rpm
yum -y localinstall gitlab-ce-10.8.6-ce.0.el7.x86_64.rpm
# 目前生产使用版本为10.8.6,不使用更新版本
gitlab-ctl reconfigure
systemctl is-enabled gitlab-runsvdir.service
sed -i "s|http://gitlab.example.com|http://$(hostname -i)|g" /etc/gitlab/gitlab.rb
sed -i "s/^.*backup_keep_time.*$/gitlab_rails['backup_keep_time'] = 604800/g" /etc/gitlab/gitlab.rb
crontab -l>/tmp/crontab.tmp
echo '# GitLab Backup Job'>>/tmp/crontab.tmp
echo '30 0 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create'>>/tmp/crontab.tmp
cat /tmp/crontab.tmp |crontab
rm -rf /tmp/crontab.tmp
cat >>/etc/gitlab/gitlab.rb<<EOF
# mail alert setup
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'smtp.126.com'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = 'XXXX@126.com'
gitlab_rails['smtp_password'] = 'xxxx'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto']= true
gitlab_rails['gitlab_email_from']= 'XXXX@126.com'
gitlab_rails['gitlab_email_reply_to']= 'noreply@126.com'
EOF
gitlab-ctl reconfigure
gitlab-ctl restart
root账号浏览器登陆,创建普通开发用户vincent
导入测试用的项目:http://vincent:Vincent_123@192.168.1.226/vincent/shareprofit.git
该项目是一个maven项目,根目录下的pom.xml可以构建出三个子项目的war包
该项目的master分支是受保护的,不能直接push到master分支,而需要提交merge请求合并更新到master分支
Jenkins环境搭建:
依据《CentOS7实验机模板搭建部署》克隆实验机,IP:192.168.77.130,使用用户deploy部署Jenkins
HOSTNAME=jenkins
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts
cd /usr/local/
tar -xf /tmp/jdk-8u172-linux-x64.tar.gz
echo 'export JAVA_HOME=/usr/local/jdk1.8.0_172'>>/etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib'>>/etc/profile
echo 'export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH'>>/etc/profile
source /etc/profile
java -version
yum -y install git
cd /usr/local/
unzip /tmp/apache-maven-3.5.2-bin.zip
echo 'export MAVEN_HOME=/usr/local/apache-maven-3.5.2'>>/etc/profile
echo 'export PATH=$PATH:$MAVEN_HOME/bin'>>/etc/profile
source /etc/profile
mvn --version
cd /tmp
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum -y install jenkins
useradd deploy
echo deploy|passwd --stdin deploy
sed -i 's|^\(JENKINS_JAVA_CMD=\).*$|\1"/usr/local/jdk1.8.0_172/bin/java"|g' /etc/sysconfig/jenkins
sed -i 's|^\(JENKINS_PORT=\).*$|\1"18080"|g' /etc/sysconfig/jenkins
sed -i 's|JENKINS_USER="jenkins"|JENKINS_USER="deploy"|g' /etc/sysconfig/jenkins
chown -R deploy: /var/log/jenkins
chown -R deploy: /var/lib/jenkins
chown -R deploy: /var/cache/jenkins
chkconfig jenkins on
/etc/init.d/jenkins start
参见《CentOS7 Jenkins部署 Maven项目构建测试》的网页配置部分继续网页配置
进一步做Jenkins和Gitlab对接,使用deploy用户做ssh key对接:
su - deploy
ssh-keygen -t rsa
ssh-copy-id 127.0.0.1
ssh 127.0.0.1 date
ssh -o StrictHostKeyChecking=no $(hostname) date
cat ~/.ssh/authorized_keys
# 记录公钥认证文件内容,浏览器使用root用户登陆gitlab,添加ssh key
# User Settings——>SSH Keys——>贴入id_rsa.pub内容——>Add key
# 简单测试:
cd /tmp/
mkdir GitLabTest
cd GitLabTest
git clone git@192.168.77.100:root/shareprofit.git
Nginx环境搭建:
将nginx部署在jenkins主机192.168.77.130之上:
cat >/etc/yum.repos.d/nginx.repo<<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/\$basearch/
gpgcheck=0
enabled=1
EOF
yum -y install nginx
systemctl enable nginx
systemctl start nginx
chown -R deploy: /usr/share/nginx/html/
测试环境主机和生产环境主机1以及生产环境主机2搭建:
依据《CentOS6实验机模板搭建部署》克隆部署三台试验机,IP:192.168.77.200、192.168.77.211和192.168.77.212
依据《CentOS6u8 java和tomcat多版本模板部署搭建配置》配置jdk8/tomcat8项目运行环境
创建运行三个tomcat项目:
yum -y install rsync
su - web_pro
cd /web/checkTOMCAT
bash pro_deploy.sh -n shareprofit-backport -j java_1.8 -t tomcat8
bash pro_deploy.sh -n shareprofit-merchantport -j java_1.8 -t tomcat8
bash pro_deploy.sh -n shareprofit-soaport -j java_1.8 -t tomcat8
Jenkins主机的deploy用户需要直接远程管控这三台机器,因此使用192.168.77.130进行部署:
su - deploy
ssh-copy-id web_pro@192.168.77.200
ssh-copy-id web_pro@192.168.77.211
ssh-copy-id web_pro@192.168.77.212
ssh -o StrictHostKeyChecking=no web_pro@192.168.77.200 date
ssh -o StrictHostKeyChecking=no web_pro@192.168.77.211 date
ssh -o StrictHostKeyChecking=no web_pro@192.168.77.212 date
测试环境自动构建功能的实现:
配置部署gitlab的webhook,当有新merge请求确定时,自动触发jenkins的构建
并将构建出的各个子项目war包同步到测试环境之上,重启对应的tomcat
-
Jenkins新增插件:Gitlab Hook
-
创建构建job,构建触发器标签,勾选Build when a change is pushed to GitLab
勾选Accepted Merge Request Events,其他都不勾选
高级选项卡,Secret token,Generate,生成token,保存下来,同时保存触发该job的URL -
继续配置源码、构建保留策略等
-
gitlab,Admin Area,Settings,Outbound requests,Allow requests to the local network from hooks and services 打开
-
gitlab上的项目,settings,Integrations,webhook,Merge request events请求触发
填写保存的URL和token,测试,返回码为200时配置成功 -
手动修改gitlab上的源码,生成一个merge请求并确认,查看是否自动触发jenkins的自动构建
-
修改构建job,添加Post Steps,在Run only if build succeeds执行脚本,将构建出来的war包同步到77.200之上,并重启tomcat
#!/bin/bash
source ~/.bash_profile
set +x
for i in $(seq 50);do echo -n '#';done;echo
cd ${WORKSPACE}/
for war in $(find ./ -type f -name "*.war")
do
echo "${war} will be deployment"
RootDir=$(echo ${war}|awk -F'/' '{print $2}')
echo "${RootDir} project will be update"
/bin/cp -av ${war} /tmp/ROOT.war
echo "/bin/rsync -avz /tmp/ROOT.war web_pro@192.168.77.200:/web/project/*${RootDir}/"
/bin/ssh web_pro@192.168.77.200 rm -rf /web/project/*${RootDir}/*
/bin/rsync -avz /tmp/ROOT.war web_pro@192.168.77.200:/web/project/*${RootDir}/
/bin/rm -rvf /tmp/ROOT.war
echo "From 192.168.77.200 restart ${RootDir} project"
Pid=$(/bin/ssh web_pro@192.168.77.200 ps -ef|grep ${RootDir}/bin|grep -v grep|awk '{print $2}')
/bin/ssh web_pro@192.168.77.200 kill -9 ${Pid}
echo "Restart ${RootDir} project done"
done
for i in $(seq 50);do echo -n '#';done;echo
set -x
生产多环境手动控制发布功能的实现:
生产多环境手动控制发布功能需求细节:
能够将新版本发布到生产主机
能够将上一个版本发布到生产主机
发布的方式为全量和金丝雀
包发布后,tomcat主机重启方式为全部重启和一定时间的间隔重启
-
创建新的jenkins的job作为生产上线用,为每个子项目单独创建一个job
-
在${WORKSPACE}目录下配置环境文件:
cat deployOS.lst
# IP:ProjectName:WarPath:SingleDeployFlag
192.168.77.211:/web/tomcat8_8081_shareprofit-merchantport/bin:/web/project/tomcat8_8081_shareprofit-merchantport:True
192.168.77.212:/web/tomcat8_8081_shareprofit-merchantport/bin:/web/project/tomcat8_8081_shareprofit-merchantport:False
该文件配置了IP指向项目所在的主机,ProjectName用于杀死项目,依据《CentOS6u8 java和tomcat多版本模板部署搭建配置》创建的项目能够自动拉起
WarPath指向war包所在的目录,用户最终的war包分发,SingleDeployFlag用于标识该主机是否为金丝雀发布主机
该配置文件较为重要,建议加锁,该配置文件保存在${WORKSPACE}目录之下,需要创建job之后再配置
-
job相关配置点1:
参数化构建过程 ——> 选项参数 ——> Option ——> Update;Rollback
参数化构建过程 ——> 选项参数 ——> Range ——> All;Single
参数化构建过程 ——> 选项参数 ——> Interval ——> Same time;One by one -
job相关配置点2:
Pre Steps,执行shell
#!/bin/bash
source ~/.bash_profile
set +x
# war包版本号
Release=$(date +%Y%m%d.%H%M%S)
# war包同步服务器
Web=192.168.77.130:80
# 默认发布间隔
Sleep_time=1
# 判断发布的主机列表,是否为单机发布
osCheck(){
if [ ${Range} == 'All' ]
then
OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/) print $1}')
else
OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/&&$4=="True") print $1}')
fi
echo -e "[+] $(date +%F_%T) 发布的环境为:"
echo "${OS_list}"|awk '{print "\t"$1}'
}
# 判断发布的方式,是否为同时发布,设置发布间隔
intervalCheck(){
if [ "${Interval}" != 'Same time' ]
then
Sleep_time=60
fi
echo "[+] $(date +%F_%T) 发布的方式为: ${Interval}"
echo "[+] $(date +%F_%T) Tomcat重启时间间隔: ${Sleep_time} s"
}
# 同步war包到jenkins主机的nginx目录中,并清理历史,保留5个war包
warMoveAndClear(){
WarFile=${JOB_NAME}-${Release}.war
mkdir -p /usr/share/nginx/html/${JOB_NAME}
cd ${WORKSPACE}/${JOB_NAME}/target/
cp -a ${JOB_NAME}*.war /usr/share/nginx/html/${JOB_NAME}/${WarFile}
WarFileNumber=$(ls /usr/share/nginx/html/${JOB_NAME}/*.war|wc -l)
if [ "${WarFileNumber}" -ge 6 ]
then
StandFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -6|tail -1)
find /usr/share/nginx/html/${JOB_NAME}/ -type f -name "*.war" -not -newer ${StandFile} -exec rm -f {} \;
fi
echo "[+] $(date +%F_%T) 本次发布为升级发布"
echo "[+] $(date +%F_%T) 发布war包为: ${WarFile}"
}
# 回滚功能的包名定位
warFileFind(){
WarFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -2|tail -1|awk -F'/' '{print $NF}')
echo "[+] $(date +%F_%T) 本次发布为回滚发布"
echo "[+] $(date +%F_%T) 发布war包为: ${WarFile}"
}
# 发布环境下载war包,该包可能是回滚包,也可能是发布包
warDownload(){
for IP in ${OS_list}
do
ssh web_pro@${IP} "rm -vf /tmp/${JOB_NAME}.war"
echo "[+] $(date +%F_%T) ${IP} 发布包 ${WarFile} 下载开始"
ssh web_pro@${IP} "wget http://${Web}/${JOB_NAME}/${WarFile} -O /tmp/${JOB_NAME}.war -o /dev/null"
done
}
# 发布功能
restartTomcat(){
TomHost=${1}
WarPath=${2}
Project=${3}
ssh web_pro@${TomHost} "rm -rf ${WarPath}/*"
ssh web_pro@${TomHost} "mv /tmp/${JOB_NAME}.war ${WarPath}/ROOT.war"
TomPid=$(ssh web_pro@${TomHost} ps -ef|grep ${Project}|grep -v grep|awk '{print $2}')
ssh web_pro@${TomHost} kill -9 ${TomPid}
echo "[+] $(date +%F_%T) 休眠${Sleep_time}s"
sleep ${Sleep_time}
}
# 最终发布
finalRestart(){
for IP in ${OS_list}
do
TomHost=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $1}')
WarPath=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $3}')
Project=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $2}')
echo "[+] $(date +%F_%T) 发布开始 ${TomHost} ${WarPath} ${Project}"
restartTomcat ${TomHost} ${WarPath} ${Project}
done
}
# 根据Option来确定是升级还是回滚,升级和回滚的区别在于同步的包是否是构建的包或者历史版本包
# 当执行回滚之时,返回码为非0,表示脚本执行失败,那么放到Pre Steps,执行回滚时,就不会继续做构建的操作了
if [ "${Option}" == 'Rollback' ]
then
for i in $(seq 50);do echo -n '#';done;echo
osCheck
intervalCheck
warFileFind
warDownload
finalRestart
for i in $(seq 50);do echo -n '#';done;echo
exit 1
fi
set -x
- job相关配置点3:
Post Steps,Run only if build succeeds,执行shell
#!/bin/bash
source ~/.bash_profile
set +x
# war包版本号
Release=$(date +%Y%m%d.%H%M%S)
# war包同步服务器
Web=192.168.77.130:80
# 默认发布间隔
Sleep_time=1
# 判断发布的主机列表,是否为单机发布
osCheck(){
if [ ${Range} == 'All' ]
then
OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/) print $1}')
else
OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/&&$4=="True") print $1}')
fi
echo -e "[+] $(date +%F_%T) 发布的环境为:"
echo "${OS_list}"|awk '{print "\t"$1}'
}
# 判断发布的方式,是否为同时发布,设置发布间隔
intervalCheck(){
if [ "${Interval}" != 'Same time' ]
then
Sleep_time=60
fi
echo "[+] $(date +%F_%T) 发布的方式为: ${Interval}"
echo "[+] $(date +%F_%T) Tomcat重启时间间隔: ${Sleep_time} s"
}
# 同步war包到jenkins主机的nginx目录中,并清理历史,保留5个war包
warMoveAndClear(){
WarFile=${JOB_NAME}-${Release}.war
mkdir -p /usr/share/nginx/html/${JOB_NAME}
cd ${WORKSPACE}/${JOB_NAME}/target/
cp -a ${JOB_NAME}*.war /usr/share/nginx/html/${JOB_NAME}/${WarFile}
WarFileNumber=$(ls /usr/share/nginx/html/${JOB_NAME}/*.war|wc -l)
if [ "${WarFileNumber}" -ge 6 ]
then
StandFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -6|tail -1)
find /usr/share/nginx/html/${JOB_NAME}/ -type f -name "*.war" -not -newer ${StandFile} -exec rm -f {} \;
fi
echo "[+] $(date +%F_%T) 本次发布为升级发布"
echo "[+] $(date +%F_%T) 发布war包为: ${WarFile}"
}
# 回滚功能的包名定位
warFileFind(){
WarFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -2|tail -1|awk -F'/' '{print $NF}')
echo "[+] $(date +%F_%T) 本次发布为回滚发布"
echo "[+] $(date +%F_%T) 发布war包为: ${WarFile}"
}
# 发布环境下载war包,该包可能是回滚包,也可能是发布包
warDownload(){
for IP in ${OS_list}
do
ssh web_pro@${IP} "rm -vf /tmp/${JOB_NAME}.war"
echo "[+] $(date +%F_%T) ${IP} 发布包 ${WarFile} 下载开始"
ssh web_pro@${IP} "wget http://${Web}/${JOB_NAME}/${WarFile} -O /tmp/${JOB_NAME}.war -o /dev/null"
done
}
# 发布功能
restartTomcat(){
TomHost=${1}
WarPath=${2}
Project=${3}
ssh web_pro@${TomHost} "rm -rf ${WarPath}/*"
ssh web_pro@${TomHost} "mv /tmp/${JOB_NAME}.war ${WarPath}/ROOT.war"
TomPid=$(ssh web_pro@${TomHost} ps -ef|grep ${Project}|grep -v grep|awk '{print $2}')
ssh web_pro@${TomHost} kill -9 ${TomPid}
echo "[+] $(date +%F_%T) 休眠${Sleep_time}s"
sleep ${Sleep_time}
}
# 最终发布
finalRestart(){
for IP in ${OS_list}
do
TomHost=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $1}')
WarPath=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $3}')
Project=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $2}')
echo "[+] $(date +%F_%T) 发布开始 ${TomHost} ${WarPath} ${Project}"
restartTomcat ${TomHost} ${WarPath} ${Project}
done
}
# 根据Option来确定是升级还是回滚,升级和回滚的区别在于同步的包是否是构建的包或者历史版本包
if [ "${Option}" == 'Update' ]
then
for i in $(seq 50);do echo -n '#';done;echo
osCheck
intervalCheck
warMoveAndClear
warDownload
finalRestart
for i in $(seq 50);do echo -n '#';done;echo
fi
set -x
-
需要收集的信息:
gitlab地址,子项目名
对应的生产环境主机IP,金丝雀发布用的IP,每个IP上的war包目录和重启用的项目名
脚本中配置的war包同步服务器,也就是搭建的nginx服务器IP地址需要根据情况修改
jenkins主机的deploy用户能够直接操控测试环境主机和所有的生产环境主机做部署和重启tomcat -
第一次构建使用Rollback,生成${WORKSPACE}目录,配置deployOS.lst文件
-
建议在构建标签的Goals and options中填写:clean package -U -Dmaven.test.skip=true
清除缓存、忽略测试进行构建
后期优化建议:
配置使用ansible来操纵各个环境主机
将jenkins的job改成pipeline类型,更加精准的控制
tomcat重启后检测catalina.out日志来确认重启是否成功
ELK生产多环境日志监控
[TOC]