Gitlab-cicd实战

一、 Gitlab部署

参考本博客gitlab docker部署url

二、测试代码编写

2.1 Java测试代码

新建一个只有一个RestControll的项目，示例如下
配置web项目的端口为：8000

代码测试页面如下：

2.2 Dockerfile文件内容如下

#该dockerfile用于build上一步代码的镜像，并把该文件放置于项目根目录下

FROM xldevops/jdk17-lts:latest
# author mail: 15234299@qq.com
MAINTAINER yuhaozhi
WORKDIR /opt
COPY ./target/*.jar app.jar
CMD java -jar app.jar

2.3 k8s 部署的yaml模板

##文件中用{}引起来的为变量，在gilab-ci文件中会进行替换
kind: Deployment
apiVersion: apps/v1
metadata:
  name: {APPNAME}
  namespace: {NAMESPACE}
  labels:
    app: {APPNAME}
    k8s.kuboard.cn/layer: {LAYER}
    k8s.kuboard.cn/name: {APPNAME}
  annotations:
    k8s.kuboard.cn/displayName: {APPNAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {APPNAME}
      k8s.kuboard.cn/layer: {LAYER}
      k8s.kuboard.cn/name: {APPNAME}
  template:
    metadata:
      labels:
        app: {APPNAME}
        k8s.kuboard.cn/layer: {LAYER}
        k8s.kuboard.cn/name: {APPNAME}
    spec:
      containers:
        - name: {CONTAINER_NAME}
          image: '{IMAGE}'
          resources:
            limits:
              cpu: '1'
              memory: 512Mi
            requests:
              cpu: 200m
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: {CHECK_PATH}
              port: {CHECK_PORT}
              scheme: {CHECK_SCHEMA}
            timeoutSeconds: 1
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: {CHECK_PATH}
              port: {CHECK_PORT}
              scheme: {CHECK_SCHEMA}
            timeoutSeconds: 1
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      imagePullSecrets:
        - name: {HARBOR_SECRET}
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: {APPNAME}
                    k8s.kuboard.cn/layer: {LAYER}
                    k8s.kuboard.cn/name: {APPNAME}
                topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

2.4 .gitlab-ci.yml文件

gitlab-ci文件主要包括如下三个阶段

• build阶段
  该阶段主要功能：对java项目进行编译打包，生成jar包资源，供下一步进行docker build。

• package阶段
  该阶段主要功能：为java测试项目docker镜像制作，并上传至harbor私有仓库。

• deploy_to_k8s
  该阶段主要功能：部署项目至k8s集群，因为本次是测试项目，只部署了deployment，并未涉及service/ingress等配置，方法与部署deployment类同。

• cleanup
  该阶段主要功能：清理文件

variables:
  SPRING_PROFILES_ACTIVE: gitlab-ci
  MAVEN_CLI_OPTS: "-Dmaven.test.skip=true -s settings.xml --batch-mode"
  MAVEN_OPTS: "-Dmaven.repo.local=repository"
  GIT_SSL_NO_VERIFY: "true"
  KUBE: "$KUBECONFIG"
  APPNAME: "demo"
  LAYER: "web"
  NAMESPACE: "web"
  CONTAINER_NAME: "c-demo"
  IMAGE: "192.168.1.230/testrepo/demo1:$CI_COMMIT_REF_SLUG"
  CHECK_PATH: "/hello"
  CHECK_PORT: "8000"
  CHECK_SCHEMA: "HTTP"
  HARBOR_SECRET: "harbor-secret"
  USER: "admin"
  PASSWORD: "*******"

stages:
  - build
  - package
  - deploy_to_k8s
  - cleanup

.build_image: &build_image |
  echo $CI_COMMIT_REF_SLUG > tag.txt
  echo "======================================"
  sed -i "/COPY/aADD tag.txt ." Dockerfile
  echo "192.168.1.230 yuhaozhi.com" >> /etc/hosts
  cat /etc/hosts
  ping -c 1 yuhaozhi.com
  docker build -t ${IMAGE} .
  docker login -u ${USER} -p ${PASSWORD} http://192.168.1.230
  docker push ${IMAGE}
  echo $CI_COMMIT_REF_SLUG

.build_code: &build_code |
  echo "192.168.1.230 gitlab.example.com" >> /etc/hosts
  cat /etc/hosts
  mvn --version
  pwd
  ls ~/.m2/repository
  mvn $MAVEN_CLI_OPTS clean package
  mvn $MAVEN_CLI_OPTS -f ./pom.xml clean compile package
  mvn $MAVEN_CLI_OPTS -f ./pom.xml clean compile package spring-boot:repackage

.deploy_k8s: &dep_k8s |
  echo "heihei"
  echo $CI_COMMIT_REF_SLUG
  sed -i "s+{VAR}+${kubeconfig}+g" test.txt
  cat test.txt
  echo $kubeconfig
  echo $KUBE
  mkdir -p ~/.kube
  cat $kubeconfig > ~/.kube/config
  kubectl get node
  kubectl get pod --all-namespaces
  echo "done"
  sed -i "s+{APPNAME}+${APPNAME}+g" k8s/app.yaml
  sed -i "s+{LAYER}+${LAYER}+g" k8s/app.yaml
  sed -i "s+{NAMESPACE}+${NAMESPACE}+g" k8s/app.yaml
  sed -i "s+{CONTAINER_NAME}+${CONTAINER_NAME}+g" k8s/app.yaml
  sed -i "s+{IMAGE}+${IMAGE}+g" k8s/app.yaml
  sed -i "s+{CHECK_PATH}+${CHECK_PATH}+g" k8s/app.yaml
  sed -i "s+{CHECK_PORT}+${CHECK_PORT}+g" k8s/app.yaml
  sed -i "s+{CHECK_SCHEMA}+${CHECK_SCHEMA}+g" k8s/app.yaml
  sed -i "s+{HARBOR_SECRET}+${HARBOR_SECRET}+g" k8s/app.yaml
  cat k8s/app.yaml
  kubectl apply -f k8s/app.yaml
  kubectl get pod -n ${NAMESPACE}

build:
  image: molgenis/maven-jdk17:latest
  stage: build
  script:
    - *build_code
  tags:
    - "231"
  artifacts:
    paths:
      - ./target/*.jar

package:
  image: docker:latest
  services:
    - name: docker:24.0.5-dind
      #为dockerd服务端容器配置insecure-registry,否则docker login会报错
      command: ["--insecure-registry=yuhaozhi.com","--insecure-registry=192.168.1.230"]
      #和下面的DOCKER_HOST中的tcp://docker:2376对应
      alias: docker
  variables:
    #docker:24.0.5-dind dockerd默认使用了2376端口
    DOCKER_HOST: tcp://docker:2376
    DOCKER_DRIVER: overlay2
    #证书目录配置，启动容器时会自动生成
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "/certs/client"
    # GIT_CHECKOUT  设置为false，本job任务不下载代码
    GIT_CHECKOUT: "false"
  stage: package
  dependencies:
    - build
  tags:
    - "231"
  before_script:
    - docker info
    - pwd
    - ls
  script:
    - *build_image


deploy_to_k8s:
  stage: deploy_to_k8s
  image:
    name: bitnami/kubectl:1.28
    entrypoint: [ "" ]
  tags:
    - "231"
  variables:
    kubeconfig: "$KUBECONFIG"
  script:
    - *dep_k8s


#artifacts自动清理任务
cleanup:
  stage: cleanup
  tags:
    - "231"
  script:
    - pwd
    - ls -al
    - rm -rf artifacts/*

2.5 gitlab全局变量

该file类型的变量，存放了连接k8s集群的config文件，在gitlab-ci.yml中会进行引用。

三、验证

3.1 提交代码

提交代码触发gitlab ci，截图如下：

3.2 k8s集群验证

#demo deployment输出信息如下：
[root@node1 ~]# kubectl -n web get deployments.apps demo -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "6"
    k8s.kuboard.cn/displayName: demo
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"k8s.kuboard.cn/displayName":"demo"},"labels":{"app":"demo","k8s.kuboard.cn/layer":"web","k8s.kuboard.cn/name":"demo"},"name":"demo","namespace":"web"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app":"demo","k8s.kuboard.cn/layer":"web","k8s.kuboard.cn/name":"demo"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app":"demo","k8s.kuboard.cn/layer":"web","k8s.kuboard.cn/name":"demo"}},"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"demo","k8s.kuboard.cn/layer":"web","k8s.kuboard.cn/name":"demo"}},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"image":"192.168.1.230/testrepo/demo1:demon1-v4","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/hello","port":8000,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"c-demo","readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/hello","port":8000,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"harbor-secret"}],"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}}}
  creationTimestamp: "2024-03-08T01:39:09Z"
  generation: 8
  labels:
    app: demo
    k8s.kuboard.cn/layer: web
    k8s.kuboard.cn/name: demo
  name: demo
  namespace: web
  resourceVersion: "524770"
  uid: c75ce6d2-e075-4076-840c-14202cde9896
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: demo
      k8s.kuboard.cn/layer: web
      k8s.kuboard.cn/name: demo
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: demo
        k8s.kuboard.cn/layer: web
        k8s.kuboard.cn/name: demo
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: demo
                  k8s.kuboard.cn/layer: web
                  k8s.kuboard.cn/name: demo
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - image: 192.168.1.230/testrepo/demo1:demon1-v4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /hello
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: c-demo
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /hello
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 200m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: harbor-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2024-03-08T02:29:44Z"
    lastUpdateTime: "2024-03-08T02:29:44Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-03-08T02:13:21Z"
    lastUpdateTime: "2024-03-08T02:29:44Z"
    message: ReplicaSet "demo-6dd7668dbc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 8
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

四、部署中需要注意的事项

4.1、containerd部署

配置insecure-registry。

4.2 job的docker-dind配置–insecure-registry

4.3 kubectl 镜像job报错

报错信息如下：

Using docker image sha256:031311091d2ef4bb68876c6b347105977cf68ecfcfef4478abb91353fb47d391 for bitnami/kubectl:1.28 with digest bitnami/kubectl@sha256:b3a89d418ff417274267e3c29b7777e966444908d674741fd7d2a0bf794db791 ...
error: unknown command "sh" for "kubectl"
Did you mean this?
	set
	cp

解决方法：在job中添加entrypoint: [ “” ]