k8s+ceph部署Jenkins

1.Jenkins部署

jenkins官方文档(点击跳转)

github上的两个文件模板点击跳转到GitHub

1.我这里已经把我要用到的东西都准备好了,
[root@k8s-master jenkins]#        ll
总用量 446984
-rw-r--r-- 1 root root       995 8月  23 15:36 ingress.yaml
-rw-r--r-- 1 root root      1016 8月  23 11:22 jenkins-account.yaml (认证相关)
-rw-r--r-- 1 root root 195139032 8月  23 11:26 jenkins.tar.gz (jenkins的镜像)
-rw-r--r-- 1 root root      2832 8月  26 15:27 jenkins.yaml (jenkins的文件)
-rw-r--r-- 1 root root       220 8月  26 15:20 pvc_tools.yaml (pvc存储卷)
-rw-r--r-- 1 root root       212 8月  23 10:57 pvc.yaml
2.创建个存储卷用来存储Jenkins的数据目录
[root@k8s-master jenkins]# cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-fs-pvc
  namespace: sa ###我这里是把所有资源都放在sa的命名空间里,不需要的可以删除
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: rook-cephfs

3.这里能看到所需要的镜像,可以单独下载下来,我是单独下载下来然后上传到自己harbor仓库里的
[root@k8s-master jenkins]# $grep image jenkins.yaml     
          image: jenkins/jenkins:lts-alpine
          imagePullPolicy: Always

修改成自己的存储卷名字
修改前
[root@k8s-master jenkins]# $cat -n  jenkins.yaml |grep -A 10
'volumeClaimTemplates'
    68    volumeClaimTemplates:
    69    - metadata:
    70        name: jenkins-home
    71        # annotations:
    72        #   volume.beta.kubernetes.io/storage-class: anything
    73      spec:
    74        accessModes: [ "ReadWriteOnce" ]
    75        resources:
    76          requests:
    77            storage: 1Gi
修改后
[root@k8s-master jenkins]# grep -A 8  'volumes' jenkins.yaml   
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-fs-pvc

这个jenkins的数据目录,不需要调整,name对应上一步存储卷的的name
[root@k8s-master jenkins]#cat -n  jenkins.yaml |grep -A 2 'volumeM'
    49            volumeMounts:
    50              - name: jenkins-home
    51                mountPath: /var/jenkins_home

4.创建account
[root@k8s-master jenkins]# kubectl apply -f  jenkins-account.yaml    
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created

5.创建Jenkins
[root@k8s-master jenkins]# kubectl apply -f jenkins.yaml                            
statefulset.apps/jenkins created
service/jenkins created
ingress.networking.k8s.io/jenkins created

6.查看是否创建成功
[root@k8s-master jenkins]#  kubectl -n sa get statefulsets.apps  -o wide
NAME      READY   AGE    CONTAINERS   IMAGES
jenkins   1/1     3d3h   jenkins      harbor.yh.com/devops/jenkins:v1

7.这里可以把service改成nodeport,把端口映射出来，也可以自己配置ingress
[root@k8s-master jenkins]# kubectl -n sa get service
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
jenkins   NodePort   10.100.177.226   <none>        80:30918/TCP,50000:32591/TCP   3d3h

然后浏览器就可以通过 http://ip:30918访问jenkins
登陆密码在pod里面的/var/jenkins/xx里面

2.jenkins集成k8s

可以通过在jenkins里面安装插件,但我这里是内网环境下载插件比较麻烦,所以我选择把命令直接放到pod里面,并配置持久化存储卷

2.1先测试直接把命令跟admin.conf丢进去能不能使用
[root@k8s-master ~]# kubectl cp /etc/kubernetes/admin.conf  sa/jenkins-0:/root/
[root@k8s-master ~]# kubectl cp /bin/kubectl  sa/jenkins-0:/root/ 
[root@k8s-master jenkins]# kubectl -n sa exec -ti jenkins-0 /bin/sh
~ # mkdir -p $HOME/.kube
~ #  sudo cp -i admin.conf $HOME/.kube/config
2.2测试可以使用
~ # ./kubectl get pod
NAME                   READY   STATUS    RESTARTS      AGE
demo-b4cb4cd78-9kcfx   1/1     Running   0             4h13m
jenkins-0              1/1     Running   2 (20m ago)   2d20h

2.3创建存储卷存储k8s的文件
[root@k8s-master jenkins]# kubectl apply -f  pvc_tools.yaml    
persistentvolumeclaim/jenkins-k8stools-fs-pvc created
[root@k8s-master jenkins]# kubectl -n sa get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-fs-pvc            Bound    pvc-947d9343-6d4f-43d9-8470-90735143a5d5   10Gi       RWX            rook-cephfs    3d4h
jenkins-k8stools-fs-pvc   Bound    pvc-1ab5e3ce-b87d-4ffd-93e1-0a304a5e3635   1Gi        RWX            rook-cephfs    12s
2.4修改yaml配置文件
[root@k8s-master jenkins]# grep -A 8  'volumes' jenkins.yaml   
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-fs-pvc
      - name: jenkins-k8s-tools
        persistentVolumeClaim:
          claimName: jenkins-k8stools-fs-pvc
[root@k8s-master jenkins]# grep -A 4  'volumeMounts' jenkins.yaml  
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
            - name: jenkins-k8s-tools
              mountPath: /root/.kube  
[root@k8s-master jenkins]# kubectl apply -f jenkins.yaml 
statefulset.apps/jenkins configured
service/jenkins unchanged
2.5因为重新apply了,刚刚传进去的文件都没了,重新传一波
[root@k8s-master ~]#  kubectl cp /etc/kubernetes/admin.conf  sa/jenkins-0:/root/.kube/config
[root@k8s-master ~]# kubectl cp  /bin/kubectl sa/jenkins-0:/root/.kube/
[root@k8s-master jenkins]# kubectl -n sa exec -ti jenkins-0 /bin/sh
/ # df -h
Filesystem                                                                                                                                            Size  Used Avail Use% Mounted on
overlay                                                                                                                                               867G   35G  788G   5% /
tmpfs                                                                                                                                                  64M     0   64M   0% /dev
tmpfs                                                                                                                                                  55G     0   55G   0% /sys/fs/cgroup
172.16.4.211:6789,172.16.4.212:6789,172.16.4.213:6789:/volumes/csi/csi-vol-44020f41-228f-11ed-92ae-0000001ef824/291f5890-3854-409f-9a77-94bc6d2153e8   10G  420M  9.6G   5% /var/jenkins_home
172.16.4.211:6789,172.16.4.212:6789,172.16.4.213:6789:/volumes/csi/csi-vol-890748e5-250f-11ed-97f0-0000003331d1/d37db400-2cc6-4b4c-ab53-04bcaa56a941  1.0G     0  1.0G   0% /root/.kube
/ # /root/.kube/kubectl get pod -n sa
NAME                   READY   STATUS    RESTARTS       AGE
demo-b4cb4cd78-9kcfx   1/1     Running   0              4h30m
jenkins-0              1/1     Running   1 (4m6s ago)   7m14s

2.1jenkins创建流水线测试调用k8s命令

测试没问题