kubesphere来实现的CICD

环境准备

首先需要装好一个k8s集群和kubesphere,安装这些的过程这里不做演示了

[root@k8s-node3 ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
k8s-node2   Ready    control-plane,master   11d   v1.23.17
k8s-node3   Ready    <none>                 11d   v1.23.17
k8s-node5   Ready    <none>                 11d   v1.23.17

image-20230814114633882

DevOps项目

首先需要调试好一个完整的DevOps部署的系统,里面包含了前端、后端、中间件(Redis、Nacos)

部署之前要创建好专门的工作空间和企业空间,工作空间里要提前设置好存储和配置

像Dockerfile、deployment.yaml、Jenkinsfile这种东西,要尽量做成通用版,把一些需要替换的东西都改成获取变量,减少后期重复操作

存储

创建了工作空间以后,创建几个存储类,我创建了两个分别给redis存储数据和日志文件,存储量给小一点,2g一般都够用了

image-20230814162124273

image-20230814162307869

配置

工作空间配置

配置分保密字典和配置字典,保密字典用来放一些账号密码,配置字典用来存放服务相关配置

工作空间的配置哟配置字典就够了,这里我创建了这样几个配置,有nginx、redis、前端的配置文件,kube-root-ca.crt是本身就有的不用管

image-20230814162814091

系统配置

除了工作空间的配置,还要改下系统的配置,这个第一次用流水线的时候要改,改好了以后就不需要改了

如果公司用了私有的编译库,就要改下maven配置,在集群管理选择配置字典,搜索devops,选择ks-devops-agent,里面有个MavenSetting,这里就是maven的配置文件,把私库的地址加进去

image-20230814165651013

image-20230814165659227

用maven下载的依赖如果不用存储来保存的话,每次都要重新下载依赖包,这样很浪费时间,所以要保存每次下载的依赖

搜索jenkins,在jenkins-casc-config里把jenkins_user.yaml配置的maven里的路径重新设置一下,/data/nfs/maven_repo/这个路径是我在nfs里创建的

image-20230814170026688

image-20230814170034107

image-20230814170242158

凭证

创建DevOps项目,先创建凭证,这个和保密字典类似,用来在流水线里连Git和Harbor的用户

我这里创建了GitLab和Harbor的管理员用户
admin-kubeconfig是kubeconfig类型,不是用户密码的类型,创建的时候选好类型就可以了

image-20230814161718358

Dockerfile和deployment.yaml

Dockerfile是生成容器镜像用的编排文件,deployment.yaml是将服务部署到集群的编排文件

两个文件都要添加到Git相关分支内,前端和后端都需要

image-20230814163444592

image-20230814163506704

deployment要注意,如果是不需要映射到外网的服务,就把倒数两行nodePort相关的注释掉

deployment里面#开头的内容都是命名空间、pod名、端口号等参数,这些都会在jenkinsfile里用sed替换掉

deployment要添加副本数就改replicas后面的值,这个只适用后端程序,如果是nacos或者redis这种是不能通过直接改这个值来实现负载均衡的,中间件需要在这个工作空间里部署集群

FROM 192.168.0.192/basic_images/gm_centos79_jdk:latest
LABEL CHENWENCONG="cwc@123.com"
ARG SERVICE_NAME
ARG PORT
ENV JVM_OPT -Dnacos.standalone=true
RUN mkdir -p /data/log/${SERVICE_NAME} && mkdir -p /data/register4/app
COPY lib/${SERVICE_NAME}.jar /data/register4/app/app.jar
CMD ["sh", "-c", "java $JVM_OPT -jar /data/register4/app/app.jar"]
EXPOSE ${PORT}

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: #service_name
  namespace: #namespace
  labels:
    app: #service_name
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubesphere.io/creator: cwc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: #service_name
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: #service_name
      annotations:
        kubesphere.io/creator: cwc
        kubesphere.io/imagepullsecrets: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: #service_name
          image: '#registry/#project_name/#service_name:#version'
          ports:
            - name: #service_name
              containerPort: #service_port
              protocol: TCP
          resources:
            limits:
              memory: 2Gi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

---
apiVersion: v1
kind: Service
metadata:
  namespace: #namespace
  labels:
    app: #service_name
  name: #service_name
spec:
  sessionAffinity: None
  selector:
    app: #service_name
  ports:
    - name: #service_name
      protocol: TCP
      targetPort: #service_port
      port: #service_port
      nodePort: #nat_port
  type: NodePort

工作负载

我需要创建一个Redis的pod给系统使用,因为Redis基本上不会更新,所以不需要流水线

在工作负载里创建Redis的工作负载(pod),在服务里创建Redis对应的服务(service)

image-20230814164119073

image-20230814164131945

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis
  namespace: register4-dev
  labels:
    app: redis
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubesphere.io/creator: chenwencong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
      annotations:
        kubesphere.io/creator: chenwencong
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2023-08-04T03:33:16.040Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
        - name: volume-kqm5pe
          persistentVolumeClaim:
            claimName: redis-data
        - name: volume-ilw6vd
          persistentVolumeClaim:
            claimName: redis-log
        - name: volume-fe8kw6
          configMap:
            name: register4-redis
            items:
              - key: redis.conf
                path: redis.conf
            defaultMode: 420
      containers:
        - name: redis
          image: '192.168.0.192/basic_images/gm_redis:7.0.8'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
          resources:
            limits:
              memory: 1Gi
            requests:
              memory: 512Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: volume-kqm5pe
              mountPath: /data
            - name: volume-ilw6vd
              mountPath: /logs
            - name: volume-fe8kw6
              readOnly: true
              mountPath: /etc/redis/
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
---
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: register4-dev
  labels:
    app: redis
  annotations:
    kubesphere.io/creator: chenwencong
spec:
  ports:
    - name: redis
      protocol: TCP
      port: 6379
      targetPort: 6379
  selector:
    app: redis
  type: NodePort
  sessionAffinity: None

流水线

开始创建流水线,目前这个系统里需要部署3个前端服务,2个后端服务,两个后端服务其中一个是nacos,第一次要一个个调试好

  • 后端
    • 先做jdk的基础镜像打包到harbor
    • maven的相关配置要提前设置好
  • 前端
    • 用nginx的镜像作为前端的基础镜像,配置里把要代理的服务用pod名替换掉
    • 前端如果.git文件过大,可以在git步骤设置浅克隆,超时时间也改长一点,前端研发自行打包压缩文件上传到Git里,流水线跳过编译的步骤

最好调试完成,每个都能使用了,这里贴了部署后端和前端的jenkinsfile作为参考

def createVersion() {
    // 定义一个版本号作为当次构建的版本,输出结果 20191210175842_69
    return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}

pipeline {
  agent {
    kubernetes {
      inheritFrom 'maven'
    }

  }
  stages {
    stage('Checkout scm') {
      agent none
      steps {
        container('maven') {
          git(url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq', changelog: true, poll: false, branch: "${params.BRANCH_NAME}")
        }

      }
    }

    stage('Compile') {
      agent none
      steps {
        container('maven') {
          sh "mvn clean install -pl ${params.SERVICE_NAME} -am  -amd -Dmaven.test.skip=true"
        }

      }
    }

    stage('Archive the artifacts') {
      agent none
      steps {
        container('maven') {
          archiveArtifacts "${params.SERVICE_NAME}/target/*jar"
        }

      }
    }

    stage('Make && Push images') {
      agent none
      steps {
        container('maven') {
          withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
            sh "docker image build --build-arg SERVICE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.SERVICE_NAME}/"
            sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
            sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
            sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
            sh "docker push ${registry}/${project_name}/${service_name1}:latest"
          }

        }

      }
    }

    stage('Deploy to server') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
            sh """cd ${params.SERVICE_NAME}
            sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
            sed -i \'s/#registry/${registry}/g\' deployment.yaml
            sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
            sed -i \'s/#service_port/${port1}/g\' deployment.yaml
            sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
            sed -i \'s/#version/${version}/g\' deployment.yaml
            sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
            cat deployment.yaml
                        
            mkdir ~/.kube
            echo "$ADMIN_KUBECONFIG" > ~/.kube/config
            cat ~/.kube/config
            kubectl apply -f .
            """
          }

        }

      }
    }

  }
  environment {
    service_name1 = 'estate-server' //服务1的名字
    port1 = '8085' // 服务1的端口
    nat_port1 = 'random' // 服务1的外网端口,如果是randon就是随机端口
    registry = '192.168.0.192' // 内网harbor镜像仓库地址
    project_name = 'register4.0_dev' // 服务在harbor的项目名
    namespace = 'register4-dev' // 要把服务部署在哪个命名空间
    version = createVersion() // 获取镜像构建的时间为构建版本
  }
  parameters {
    string(name: 'BRANCH_NAME', defaultValue: 'devops', description: '请输入要构建的分支名称')
    string(name: 'SERVICE_NAME', defaultValue: 'estate-lowcode-plateform-cloud-server', description: '请输入服务名')
    string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
    string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/estate-lowcode-platform.git', description: 'GIT仓库后缀')
  }
}
def createVersion() {
    // 定义一个版本号作为当次构建的版本,输出结果 20191210175842_69
    return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}

pipeline {
  agent {
    kubernetes {
      inheritFrom 'nodejs'
    }

  }
  stages {
    stage('Clone repository') {
      agent none
      steps {
        checkout([$class: 'GitSCM', doGenerateSubmoduleConfigurations: false, submoduleCfg: [], extensions: [[$class: 'CloneOption', noTags: false, timeout: 1000, shallow: true, depth: 1]],
                    branches: [[name: "${params.BRANCH_NAME}"]],userRemoteConfigs: [[url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq']]])
      }
    }



    stage('Archive the artifacts') {
      agent none
      steps {
        container('nodejs') {
          archiveArtifacts "${params.PACKAGE_DIR}/${params.SERVICE_NAME}.zip"
        }

      }
    }

    stage('Make && Push images') {
      agent none
      steps {
        container('nodejs') {
          withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
            sh "docker image build --build-arg SERVICE_NAME=${service_name1} --build-arg PACKAGE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.PACKAGE_DIR}/"
            sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
            sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
            sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
            sh "docker push ${registry}/${project_name}/${service_name1}:latest"
          }

        }

      }
    }

    stage('Deploy to server') {
      agent none
      steps {
        container('nodejs') {
          withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
            sh """cd ${params.PACKAGE_DIR}
            sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
            sed -i \'s/#registry/${registry}/g\' deployment.yaml
            sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
            sed -i \'s/#service_port/${port1}/g\' deployment.yaml
            sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
            sed -i \'s/#version/${version}/g\' deployment.yaml
            sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
            cat deployment.yaml
                        
            mkdir ~/.kube
            echo "$ADMIN_KUBECONFIG" > ~/.kube/config
            cat ~/.kube/config
            kubectl apply -f .
            """
          }

        }

      }
    }

  }
  environment {
    service_name1 = 'low-code-ui' //服务1的名字
    port1 = '81' // 服务1的端口
    nat_port1 = '30022' // 服务1的外网端口,如果是randon就是随机端口
    registry = '192.168.0.192' // 内网harbor镜像仓库地址
    project_name = 'register4.0_sx' // 服务在harbor的项目名
    namespace = 'register4-sx' // 要把服务部署在哪个命名空间
    version = createVersion() // 获取镜像构建的时间为构建版本
  }
  parameters {
    string(name: 'BRANCH_NAME', defaultValue: 'shanxi', description: '请输入要构建的分支名称')
    string(name: 'PACKAGE_DIR', defaultValue: 'package', description: '请输入打包文件夹')
    string(name: 'SERVICE_NAME', defaultValue: 'low-code-ui', description: '请输入服务名')
    string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
    string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/portal/low-code/low-code-ui.git', description: 'GIT仓库后缀')
  }
}

image-20230814170734585
image-20230815093459198

第N次创建DevOps项目

保存好第一次配置的所有东西,它们基本上都能保存成yaml文件,它们就是你这个系统的DevOps项目的模板了

image-20230814170855597

如果这个系统还需要部署多套环境,就可以用这些模板进行快速的部署

创建好新的工作空间→导入存储、配置字典→创建DevOps凭证→添加Dockerfile和deployment→导入jenkinsfile

其中配置字典里的前端配置肯定需要修改,因为端口号变了

jenkinsfile里要改工作空间名、harbor组织名、外网端口号等,都在最下面的变量里进行修改

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值