【K8S】【Jenkins】【CI/CD】【一】交付CI/CD工具至k8s

1 安装Harbor镜像仓库(之前已部署 ,略)

可参考之前的《Kubernetes业务迁移.pdf》

网站-账号密码

http://gitlab.oldxu.net:30080/users/sign_in    ( root/ admin123 )
http://sonar.oldxu.net:30080/                  (admin / admin12345)  #初始 admin / admin
http://jenkins.oldxu.net:30080/                (admin / admin12345)

2 交付GitLab至K8S (sts、svc、ingress)

Gitlab以容器方式运行,需要持久化如下几个目录中的数据
在这里插入图片描述

#拉取 推送
docker pull gitlab/gitlab-ce:14.6.0-ce.0
docker tag gitlab/gitlab-ce:14.6.0-ce.0 harbor.oldxu.net/ops/gitlab-ce:14.6.0
docker push harbor.oldxu.net/ops/gitlab-ce:14.6.0
#创建 ns 、和docker-registry
kubectl create ns ops

kubectl create secret docker-registry harbor-admin \
 --docker-username=admin \
 --docker-password=Harbor12345 \
 --docker-server=harbor.oldxu.net \
 -n ops

1、 gitlab-sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlib
  namespace: ops
spec:
  serviceName: "gitlab-svc"
  selector:
    matchLabels:
      app: gitlab
  template:
    metadata:
      labels:
        app: gitlab
    spec:
      imagePullSecrets: 
      - name: harbor-admin
      
      containers:
      - name: gitlab-ce
        image: harbor.oldxu.net/ops/gitlab-ce:14.6.0
        imagePullPolicy: IfNotPresent
        env:
        - name: GITLAB_ROOT_PASSWORD
          value: "admin123"
        - name: GITLAB_OMNIBUS_CONFIG
          value: |
            external_url "http://gitlab.oldxu.net"
            gitlab_rails['time_zone'] = 'Asia/Shanghai'
            node_exporter['enable'] = false
            redis_exporter['enable'] = false
            postgres_exporter['enable'] = false
            gitlab_exporter['enable'] = false
            grafana['enable'] = false
            grafana['reporting_enabled'] = false
            prometheus['enable'] = false
            prometheus['monitor_kubernetes'] = false
        
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        volumeMounts:
        - name: data
          mountPath: /etc/gitlab
          subPath: config
          
        - name: data
          mountPath: /var/opt/gitlab
          subPath: data
               
        - name: data
          mountPath: /var/log/gitlab
          subPath: logs
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteMany"]
      storageClassName: "nfs"
      resources:
        requests:
          storage: 25Gi 

2、 gitlab-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: gitlab-svc
  namespace: ops
spec:
  clusterIP: None
  selector:
    app: gitlab
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443  

3、 gitlab-ingress.yaml

#apiVersion: networking.k8s.io/v1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gitlab-ingress
  namespace: ops
spec:
  ingressClassName: "nginx"
  rules:
  - host: "gitlab.oldxu.net"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: gitlab-svc
          servicePort: 80
          #service:
          #  name: gitlab-svc
          #  port: 
          #    name: http
    

设置gitlab的界面语言为中文
在这里插入图片描述

3 交付PostgreSQL至K8S (sts 、 svc)

部署说明:
在这里插入图片描述

Sonarqube扫描流程:
1、使用SonarScanner客户端工具将代码源文件以http/https方式推送给Sonarqube服务端;
2、Sonarqube服务端基于ElasticSerach对代码进行分析,而后将分析结果存储至Database;
3、Sonarqube服务端读取Database数据,然后将扫描结果进行前端展示;
所以,安装Sonarqube之前需要先安装依赖的数据库,后期进行漏洞扫描时还需要借助SonarScanner客户端;
#Sonarqube需要PostgreSQL
#下载postgresql镜像
docker pull postgres:13.8
docker tag  621268accecf harbor.oldxu.net/ops/postgres:13.8
docker push harbor.oldxu.net/ops/postgres:13.8

1、 pgsql-sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql
  namespace: ops
spec:
  serviceName: "pgsql-svc"
  selector:
    matchLabels:
      app: pgsql
  template:
    metadata:
      labels:
        app: pgsql
    spec:
      imagePullSecrets:
      - name: harbor-admin
      
      containers:
      - name: postgresql
        image: harbor.oldxu.net/ops/postgres:13.8
        imagePullPolicy: IfNotPresent
        env:
        - name: POSTGRES_DB
          value: sonardb
        - name: POSTGRES_USER
          value: sonar
        - name: POSTGRES_PASSWORD
          value: "123456"
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: db
          mountPath: /var/lib/postgresql/data
          
  volumeClaimTemplates:
  - metadata:
      name: db
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "nfs"
      resources:
        requests:
          storage: 20Gi

2、 pgsql-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: pgsql-svc
  namespace: ops
spec:
  clusterIP: None
  selector:
    app: pgsql
  ports:
  - port: 5432

3、检查postgresql

kubectl exec -it -n ops postgresql-0 -- bash
root@postgresql-0:/# psql -Usonar -d sonardb

sonardb=# \l+

在这里插入图片描述

4 交付Sonarqube至K8S (sts、svc、ingress)

#下载sonarqube镜像
docker pull sonarqube:9.7-community
docker tag sonarqube:9.7-community harbor.oldxu.net/ops/sonarqube:9.7
docker push harbor.oldxu.net/ops/sonarqube:9.7

1、 sonarqube-sts.yaml

#需要借助busybox调整内核参数

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sonarqube
  namespace: ops
spec:
  serviceName: "sonarqube-svc"
  selector:
    matchLabels:
      app: sonarqube
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      imagePullSecrets:
      - name: harbor-admin
      
      initContainers:
      - name: set-kernel
        image: busybox
        command: ["sh","-c","sysctl -w vm.max_map_count=524288 ; sysctl -w fs.file-max=131072 ; ulimit -n 131072 ; ulimit -u 8192"] 
        securityContext:
          privileged: true
      containers:
      - name: sonarqube
        image: harbor.oldxu.net/ops/sonarqube:9.7
        imagePullPolicy: IfNotPresent
        
        env:
        - name: JAVA_OPTS
          value: -Duser.timezone=Asia/Shanghai
        - name: SONARQUBE_JDBC_USERNAME
          value: sonar
        - name: SONARQUBE_JDBC_PASSWORD
          value: "123456"
        - name: SONARQUBE_JDBC_URL
          value: jdbc:postgresql://pgsql-svc:5432/sonardb
        
        resources:
          limits:
            cpu: 1500m
            memory: 2048Mi
        ports:
        - name: web
          containerPort: 9000
        volumeMounts:
        - name: data
          mountPath: /opt/sonarqube/data
          subPath: data
        - name: data
          mountPath: /opt/sonarqube/logs
          subPath: logs
        - name: data
          mountPath:  /opt/sonarqubee/extensions
          subPath: extensions

  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "nfs"
      resources:
        requests:
          storage: 20Gi

2、 sonarqube-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: sonarqube-svc
  namespace: ops
spec:
  clusterIP: None
  selector:
    app: sonarqube
  ports:
  - name: web
    port: 9000

3、 sonarqube-ingress.yaml

#apiVersion: networking.k8s.io/v1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: sonarqube-ingress
  namespace: ops
spec:
  ingressClassName: "nginx"
  rules:
  - host: "sonar.oldxu.net"
    http:
      paths: 
      - path: /
        pathType: Prefix
        backend:
          serviceName: sonarqube-svc
          servicePort: 9000
        #service:
        #  name: sonarqube-svc
        #  port: 
        #    name: web

4、 访问sonarqube

安装中文插件,随后出现install pending, 随后点击 “restart server”
在这里插入图片描述

5 交付Jenkins至K8S (rbac 、 sts 、svc 、 ingress)

#下载 ,打tab ,推送
docker pull jenkins/jenkins:2.346.3-2-lts
docker tag jenkins/jenkins:2.346.3-2-lts harbor.oldxu.net/ops/jenkins:2.346
docker push harbor.oldxu.net/ops/jenkins:2.346

创建RBAC (Jenkins) 01-jenkins-rbac.yaml

# serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: ops

---
# clusterRole
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "ingresses"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
# clusterrolebinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: ops

02-jenkins-sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  namespace: ops
spec:
  serviceName: "jenkins-svc"
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins
      imagePullSecrets:
      - name: harbor-admin

      containers:
        - name: jenkins
          image: harbor.oldxu.net/ops/jenkins:2.346
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 0  #root身份运行
          env:
          - name: JAVA_OPTS
            value: -Duser.timezone=Asia/Shanghai

          ports:
            - name: http
              containerPort: 8080
            - name: agent
              containerPort: 50000
          resources:
            limits:
              cpu: 1500m
              memory: 2048Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: data
              mountPath: /var/jenkins_home
  volumeClaimTemplates:
  - metadata: 
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "nfs"
      resources:
        requests:
          storage: 25Gi     

03-jenkins-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-svc
  namespace: ops
spec:
  clusterIP: None
  selector:
    app: jenkins
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: agent
      port: 50000
      targetPort: 50000

04-jenkins-ingress.yaml

#apiVersion: networking.k8s.io/v1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: ops
spec:
  ingressClassName: "nginx"
  rules:
  - host: jenkins.oldxu.net
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: jenkins-svc 
          servicePort: 8080
          #service:
          #  name: jenkins-svc 
          #  port:
          #    name: http


访问Jenkins 。修改密码。安装插件。

[root@master01 04-jenkins]# kubectl exec -it -n ops jenkins-0 -- bash
root@jenkins-0:/# cat /var/jenkins_home/secrets/initialAdminPassword
9c6f0d23cc194970a3e8326708dbabbf

http://jenkins.oldxu.net:30080/pluginManager/available

安装Jenkins插件
中文插件: Localization: Chinese
Git插件: git、gitlab
Sonar插件: SonarQube Scanner
Pipeline插件: pipeline、Stage View、Blue Ocean
Kubernetes插件: Kubernetes

在这里插入图片描述

6 制作Jenkins pod template

6.1、maven

 wget https://linux.oldxu.net/settings_docker.xml

[root@node4 maven]# cat Dockerfile 
FROM  maven:3.8.6-openjdk-8

ADD ./settings_docker.xml /usr/share/maven/conf/settings.xml
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime



[root@node4 maven]# ls
Dockerfile  settings_docker.xml

#构建推送
docker build -t harbor.oldxu.net/ops/maven:3.8.6 .
docker push harbor.oldxu.net/ops/maven:3.8.6

6.2、sonar

docker pull emeraldsquad/sonar-scanner:2.3.0
docker tag emeraldsquad/sonar-scanner:2.3.0 harbor.oldxu.net/ops/sonar-scanner:2.3.0
docker push harbor.oldxu.net/ops/sonar-scanner:2.3.0

6.3、NodeJs

cat Dockerfile

FROM centos:7
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN curl --silent --location https://rpm.nodesource.com/setup_14.x |bash -
RUN yum install nodejs gcc-c++ make vim -y && \
    yum clean all
[root@node4 nodejs]# ls
Dockerfile

docker build -t harbor.oldxu.net/ops/nodejs:14.20 .
docker push harbor.oldxu.net/ops/nodejs:14.20

6.4、Docker

docker pull docker:20.10
docker tag docker:20.10 harbor.oldxu.net/ops/docker:20.10
docker push harbor.oldxu.net/ops/docker:20.10

6.5、kubelet

[root@node4 kubelet]# cat kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
FROM centos:7
# 1、调整时区
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo 'Asia/Shanghai' >/etc/timezone

# 2、添加yum源
ADD ./kubernetes.repo /etc/yum.repos.d/kubernetes.repo

# 3、安装Kubectl
RUN yum makecache && yum install kubectl-1.22.3 -y && \
    yum clean all
[root@node4 kubelet]# ls
Dockerfile  kubernetes.repo

docker build -t harbor.oldxu.net/ops/kubectl:1.22.3 .
docker push harbor.oldxu.net/ops/kubectl:1.22.3

#最好使用宿主机的k8s集群的 kubectl版本

6.6 编写Pipeline

pipeline{
  agent{
    kubernetes{
      cloud 'kubernetes'
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          imagePullSecrets:
          - name: harbor-admin
          volumes:
          - name: data
            nfs:
              server: 192.168.79.33
              path: /data/maven
          - name: dockersocket
            hostPath:
              path: /run/docker.sock            
              
          containers:
          - name: maven
            image: harbor.oldxu.net/ops/maven:3.8.6
            imagePullPolicy: IfNotPresent
            command: ["cat"]
            tty: true
            volumeMounts:
            - name: data
              mountPath: /root/.m2
          
          - name: nodejs
            image: harbor.oldxu.net/ops/nodejs:14.20
            imagePullPolicy: IfNotPresent
            command: ["cat"]
            tty: true
            
          - name: sonar
            image: harbor.oldxu.net/ops/sonar-scanner:2.3.0
            imagePullPolicy: IfNotPresent
            command: ["cat"]
            tty: true
            
          - name: docker
            image: harbor.oldxu.net/ops/docker:20.10
            imagePullPolicy: IfNotPresent
            command: ["cat"]
            tty: true
            volumeMounts:
            - name: dockersocket
              mountPath: /run/docker.sock
              
          - name: kubectl
            image: harbor.oldxu.net/ops/kubectl:1.18.0
            imagePullPolicy: IfNotPresent
            command: ["cat"]
            tty: true
      '''
    } // kubernetes end
  }//agent end
  
  stages{
    stage('maven测试'){
        steps{
            container('maven'){
                sh 'mvn --version'
            }
                
        }          
    }
    
    stage('docker'){
        steps{
            container('docker'){
               sh 'docker ps'
            }
                
        }          
    }
    
    stage('nodejs 测试'){
        steps{
            container('nodejs'){
                sh 'node -v'
            }
                
        }          
    }
    
    stage('kubectl测试'){
        steps{
            container('kubectl'){
                sh 'kubectl version'
            }
        }
    
    }        
 
  }//stages end
} //pipeline end

6.7 运行Pipeline

在这里插入图片描述

7 、 Jenkins pipeline

7.1 pipeline示例demo

新建流水线-demo

pipeline {
  agent any
  stages {
    stage('下载代码') {
        steps {
            echo "get gitlab 的 代码"
        }
    }

    stage('检测代码'){
       steps{
         echo "sonarqube Unit TEST。。。"
       }
    }

    stage('编译代码'){
        steps{
            echo "maven  build code"
    }
    }

    stage('制作镜像'){
       steps{
           echo "build docker"
       }
    
    }

    stage('部署应用'){
       steps{
         echo "Deplot code....."
       }
    }

  }//stages end

}//pipeline end

blue ocean界面:
在这里插入图片描述
在这里插入图片描述

一些概念:

7.2 Jenkins Slave架构

JenkinsMaster/Slave 架构,及在Master上进行任务分配。然后由Slave来完成,不过Slave运行方式有两种:
静态SLave:需要固定的节点,配置其对应环境,手动注册到Master,然后执行任务,任务完成节点处于空闲等待状态;

动态Slave:由Master动态创建Slave的Pod,自动注册到Master,然后执行任务,任务结束Pod自动销毁;
动态Jenkins Slave:

在这里插入图片描述

8.Jenkins动态Slave配置

8.1 配置Kubernetes段

http://jenkins.oldxu.net:30080/configureClouds/

系统管理 -> 节点管理 -> configure Clouds ->Add a New Cloud -> kubernetes

Kubernetes地址: https://kubernetes.default.svc.cluster.local
Kubernetes命名空间: ops

在这里插入图片描述

#配置Jenkins段
Jenkins地址: http://jenkins-svc.ops.svc.cluster.local:8080
Jenkins通道: jenkins-svc.ops.svc.cluster.local:50000

点击“连接测试”,查看是否能连到k8s

在这里插入图片描述

8.2 运行流水线测试

pipeline{
    agent{
        kubernetes{ //自动创建一个slavePod 执行如下动作
            cloud 'Kubernetes'
        }
    }//agent end

    stages{
        stage('Hello'){
            steps{
                echo 'hello 133'
                sh 'hostname'
                sh 'pwd'
            }
        }
    }

}//pipeline end

END

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值