基于Openshift的jenkins设计方案

最新推荐文章于 2022-09-06 21:59:01 发布

qq_道可道

最新推荐文章于 2022-09-06 21:59:01 发布

阅读量177

点赞数

分类专栏： Openshift K8S与容器文章标签： cicd jenkins

本文链接：https://blog.csdn.net/qq_16240085/article/details/89452723

版权

K8S与容器同时被 2 个专栏收录

28 篇文章 1 订阅

订阅专栏

Openshift

12 篇文章 0 订阅

订阅专栏

基于Openshift的jenkins高可用方案

目标
方案测试
Master基础镜像
Slave基础镜像
存储挂载说明
Jenkins配置
- 全局配置
- Jenkins job配置
Openshift Jenkins yaml 及 Dockerfile等
附：jenkins通用迁移流程及备份脚本
- 1.通用迁移流程
- 2.备份脚本
排错

目标

将现有jenkins直接迁移至 Openshift中运行 Jenkins Master和Slave，其中Master负责调度，Slave负责创建pod并构建。

方案测试

1.尝试将jenkins版本升级发现很多插件不支持，无奈最终还是选择当前版本。
2.尝试将jenkins数据打包进镜像，修改后重启pod还原，结果丧失了灵活性，最佳实践还是挂载数据。
3.尝试启动多master使用rbd外挂数据，结果同时只能启动一个master，rbd不能挂载多个pod，只支持单个pod。
4.尝试启动多master使用cephfs外挂数据，结果虽然可以启动多个master，但是jenkins本身机制不支持多个tomcat运行一份数据。
5.分别尝试将数据挂载至rbd和cephfs做对比，发现rbd的IO性能高于cephfs，典型现象是在保存修改配置时使用cephfs时间高于rbd几倍，最终选择rbd。
6.自制slave镜像必须使用官方指定配置，参考下方Slave镜像Dockerfile。
7.slave除了使用host模式挂载docker配置外，另需挂载cephfs至/data目录
cephfs根目录包含
jacoco-0.7.9（有修改需求）
paas（发布脚本）
.m2（maven编译仓库缓存，这样所有的slave pod均可共享）

Master基础镜像

基于maven3+jdk8，通过额外jenkinsjob集成自定义配置，避免每次构建浪费时间

# Author: zhangzhidao
# base-jdk8
FROM reg.iqianjin.com/test/maven:3.0.5
MAINTAINER zhangzhidao  "zhangzhidao@iqianjin.com"
# 这里用tomcat启动jenkins、安装tomcat日志分割插件、私钥、基础环境配置
ADD tomcat-jenkins-8080 /data/server/tomcat-jenkins-8080 
ADD cronolog-1.6.2 /data/cronolog-1.6.2 
ADD id_rsa /root/.ssh/id_rsa 
RUN yum install git docker-client -y ; cd /data/cronolog-1.6.2 && ./configure && make && make install && cd /data; rm -rf cronolog-1.6.2 ; chmod 600 /root/.ssh/id_rsa
# BUILD: docker build --no-cache --network=host -t reg.iqianjin.com/test/iqj-jenkins:v2 .
# RUN: docker run -it --rm --net=host  reg.iqianjin.com/test/iqj-jenkins:v2

Slave基础镜像

参考官方github，使用centos作为基础镜像，可以直接使用

# Author: zhangzhidao
# https://hub.docker.com/r/jenkins/slave/dockerfile
# https://github.com/jenkinsci/docker-jnlp-slave
# base-jdk8
FROM reg.iqianjin.com/test/maven:3.0.5
MAINTAINER zhangzhidao  "zhangzhidao@iqianjin.com"
ARG VERSION=3.28
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000

ENV HOME /home/${user}
RUN groupadd -g ${gid} ${group}
RUN useradd -d $HOME -u ${uid} -g ${group} ${user}
LABEL Description="This is a base image, which provides the Jenkins agent executable (slave.jar)" Vendor="Jenkins project" Version="${VERSION}"

ARG AGENT_WORKDIR=/home/${user}/agent

RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar \
  && chmod 755 /usr/share/jenkins \
  && chmod 644 /usr/share/jenkins/slave.jar 
USER root
#USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}

VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}

COPY jenkins-slave /usr/local/bin/jenkins-slave

# 以下4行基于官方镜像添加了私钥、oc客户端、发布脚本、基础环境
COPY id_rsa /root/.ssh/id_rsa
COPY oc /bin/
RUN mkdir /data/jenkins/workspace -p && chmod 600 /root/.ssh/id_rsa && yum install docker-client ant git -y

ENTRYPOINT ["jenkins-slave"]

# docker build --no-cache --network=host -t reg.iqianjin.com/test/jenkins/jnlp-slave .

存储挂载说明

1.master挂载rbd，并将jenkins配置目录全部拷贝进去
2.1 slave挂载本机docker client包括：/var/run/docker.sock /usr/bin/docker /etc/sysconfig/docker
2.2 slave挂载cephfs /data包括：paas目录、.m2目录、jacoco-0.7.9目录

Jenkins配置

全局配置

指定开放agent固定50000端口
在这里插入图片描述
安装kubernates插件，按下图配置
主要注意image镜像、docker挂载、pvc共享存储（.m2）挂载、代理存活时间适当调长。

jnlp名称说明参考：https://blog.51cto.com/ylw6006/2159769

Jenkins job配置

关键是定义label（与全局配置中创建POD模板关联，这里名字是jenkins-slave）与配置podTemplate()
示例：

def label = "jenkins-slave"
podTemplate(label: label, cloud: 'kubernetes') {
node(label){
  stage('Pull') {
    try {
        checkout([$class: 'GitSCM', branches: [[name: '${branch_name}']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${git_credentialsId}", refspec: '+refs/heads/${branch_name}:refs/remotes/origin/${branch_name}', url: '${git_addr}']]])
        // sh '/bin/sh /data/paas/ione_var.sh pull ${deploy_id} true ${BUILD_URL}console'
    } catch(err) {
        sh '/bin/sh /data/paas/ione_var.sh pull ${deploy_id} false ${BUILD_URL}console && exit 1'
    }
  }
  stage('Build') {
    try {
      sh '${mvn_args}'
    //   sh '/bin/sh /data/paas/ione_var.sh package ${deploy_id} true ${BUILD_URL}console'
    } catch(err) {
      sh '/bin/sh /data/paas/ione_var.sh package ${deploy_id} false ${BUILD_URL}console && exit 1'
    }
  }
  stage('Deploy') {
      try {
        sh '/bin/sh /data/paas/deploy.sh'
        sh '/bin/sh /data/paas/ione_var.sh develop ${deploy_id} true ${BUILD_URL}console'
      } catch (err) {
        sh '/bin/sh /data/paas/ione_var.sh develop ${deploy_id} false ${BUILD_URL}console && exit 1'
      }
  }
}
}

Openshift Jenkins yaml 及 Dockerfile等

[iqj-jenkins]# ls
Dockerfile  id_rsa  openshift.yaml  run.sh

openshift.yaml

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  labels:
    app: iqj-jenkins
    #iOneDeployId: template_deployid
    #pkgName: template_pkgName
  name: iqj-jenkins
  namespace: template_namespace
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    app: iqj-jenkins
    #deploymentconfig: iqj-jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        openshift.io/generated-by: OpenShiftNewApp
      labels:
        app: iqj-jenkins
        #iOneDeployId: template_deployid
        #pkgName: template_pkgName
        #deploymentconfig: iqj-jenkins
    spec:
      containers:
        - image: reg.iqianjin.com/test/iqj-jenkins:v2
          imagePullPolicy: Always
          name: iqj-jenkins
          ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 50000
              protocol: TCP
          securityContext:
          securityContext:
            runAsUser: 0 # 容器运行用户,会覆盖默认,这里以root启动
          resources:
            requests:
              cpu: "2000m"
              memory: "2000Mi"
            limits:
              cpu: "6000m"
              memory: "6000Mi"
          volumeMounts:
            - mountPath: /data/jenkins
              name: jenkinsdata
            #  subPath: iqj-jenkins/jenkins
            #- mountPath: /data/paas
            #  name: jenkinspaas
            #  subPath: iqj-jenkins/template_namespace/paas
            - mountPath: /var/run/docker.sock
              name: dockersock
            - mountPath: /usr/bin/docker
              name: dockercli
            - mountPath: /etc/sysconfig/docker
              name: dockerconf
      volumes:
        - name: jenkinsdata
          persistentVolumeClaim:
            claimName: ceph-pvc-jenkinsdata
        #- name: jenkinspaas
        #  persistentVolumeClaim:
        #    claimName: cephfs-pvc
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
        - name: dockercli
          hostPath:
            path: /usr/bin/docker
        - name: dockerconf
          hostPath:
            path: /etc/sysconfig/docker
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  labels:
    app: iqj-jenkins
  name: iqj-jenkins
  namespace: template_namespace
spec:
  ports:
    - name: 8080-tcp
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: 50000-slave-tcp
      port: 50000
      protocol: TCP
      targetPort: 50000
  selector:
    app: iqj-jenkins
    #deploymentconfig: iqj-jenkins
  type: ClusterIP

Dockerfile

# dockerfile 基础配置
FROM reg.iqianjin.com/test/iqj-jenkins:v2
MAINTAINER zhangzhidao  "zhangzhidao@iqianjin.com"
COPY hosts run.sh /
COPY id_rsa /root/.ssh/id_rsa
RUN chmod +x /run.sh && chmod 600 /root/.ssh/id_rsa
EXPOSE 8080 50000
#USER 10000
CMD ["/run.sh"]

run.sh

#!/bin/bash
# namespace hosts bind for nginx
source /etc/profile
chmod 666 /etc/hosts
cat /hosts >> /etc/hosts
export JAVA_OPTS="run_args"
/bin/sh /data/server/tomcat-jenkins-8080/bin/catalina.sh run

附：jenkins通用迁移流程及备份脚本

1.通用迁移流程

#jenkins迁移流程
#1.打包tomcat备份
#tar fcz tomcat_jenkins_yz_8081.tar.gz tomcat_jenkins_yz_8081 --exclude=tomcat_jenkins_yz_8081/logs/*
#scp  tomcat_jenkins_yz_8081.tar.gz 192.168.36.53:/data/server/
#2.打包jenkins数据
#tar fcvz jenkinsdata_yz.bak.tar.gz jenkinsdata_yz --exclude=jenkinsdata_yz/jobs/*/workspace*/*
#scp jenkinsdata_yz.bak.tar.gz 192.168.36.53:/data/server/
#3.打包依赖软件
#tar fcz middleware.tar.gz apache-maven-3.0.5/ node-v8.9.4-linux-x64 git-1.9/
#scp middleware.tar.gz 192.168.36.53:/usr/local/
#4.打包脚本
#tar fcz scripts_yz.tar.gz scripts_yz/
#scp  scripts_yz.tar.gz 192.168.36.53:/data/server/
#5.检查/etc/hosts文件和/etc/profile变量
#6.免密登录
#ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no -p60000 user@ip

2.备份脚本

cat backup_jenkins.sh 
#!/bin/bash
set -x
##备份jenkins脚本

del_time=`date -d '-3 days' '+%Y-%m-%d'`
jenkins_pkg=jenkinsdata_yz.bak`date '+%F'`.tar.gz
del_jenkins_pkg=jenkinsdata_yz.bak${del_time}.tar.gz
#2.打包jenkins数据
cd /data/server
tar fcz ${jenkins_pkg} jenkinsdata_yz --exclude=jenkinsdata_yz/jobs/*/workspace*/*
scp ${jenkins_pkg} 192.168.36.53:/data/server/ && rm -rf ${jenkins_pkg}
ssh 192.168.36.53 "rm -rf /data/server/${del_jenkins_pkg}"
#4.打包脚本
tar fcz scripts_yz.tar.gz scripts_yz/
scp  scripts_yz.tar.gz 192.168.36.53:/data/server/

排错

根据jenkins日志排查

qq_道可道

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
基于Openshift的jenkins设计方案

基于Openshift的jenkins设计及搭建Author: zhangzhidaobase-jdk8这里用tomcat启动jenkins、安装tomcat日志分割插件、私钥、基础环境配置BUILD: docker build --no-cache --network=host -t reg.iqianjin.com/test/iqj-jenkins:v2 .RUN: docker run -i...
复制链接

扫一扫