Jenkins+k8s部署Jenkins slave

一.构建镜像

下载基础镜像，这里使用openvz的包,下载centos7的镜像

1.1 下载镜像后导入到本地 JENKINS VERSION为: 2.222.4 --------------------------------------- [root@node-1 jenkins]# ls centos-7-x86_64.tar.gz  jenkinsci.tar  jenkins-slave [root@node-16 jenkins]# docker import centos-7-x86_64.tar.gz openvz-centos7 sha256:e27fdd2e207904422fecb26e2f8596bed10db7ea0961d930edf813cb8f172778 [root@node-1 jenkins]# [root@node-1 jenkins]# [root@node-1 jenkins]# docker images REPOSITORY                                  TAG                   IMAGE ID            CREATED             SIZE openvz-centos7                              latest                e27fdd2e2079        11 seconds ago      589MB

从官方下载最新版的Jenkins的war包和适配的jdk，这里使用jdk8。 [root@node-1 software]# ls jenkins.war jenkins.war [root@node-1 software]# [root@node-1 software]# ls jdk-8u77-linux-x64.tar.gz jdk-8u77-linux-x64.tar.gz [root@node-1 software]#   1.2 构建Jenkins的dockerfile文件dockerfile-jenkins   [root@node-1 jenkins]# [root@node-1 jenkins]# cat dockerfile-jenkins FROM openvz-centos7 ADD jdk-8u77-linux-x64.tar.gz  /home ADD jenkins.war /home/jenkins.war ENV JAVA_HOME=/usr/local/java ENV PATH=$PATH:/usr/local/java/bin ENV JENKINS_HOME=/var/jenkins_home WORKDIR /home CMD java -jar jenkins.war --httpPort=8088   [root@node-1 jenkins]#   [root@node-1jenkins]#

1.3: 构建后推送到本地仓库 [root@node-1 jenkins]# docker login 100.100.100.11 Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store   Login Succeeded [root@node-1 jenkins]# [root@node-1 jenkins]# docker build -t  100.100.100.11/library/jenkins-self:2.222.4 -f dockerfile-jenkins .   [root@node-1 jenkins]# docker build -t  100.100.100.11/library/jenkins-self:2.222.4 -f dockerfile-jenkins . Sending build context to Docker daemon  1.196GB Step 1/8 : FROM openvz-centos7  ---> 78f2ed06a6c3 Step 2/8 : ADD jdk-8u77-linux-x64.tar.gz  /home  ---> e9137dec97bf Step 3/8 : ADD jenkins.war /home/jenkins.war  ---> ae55e40474b1 Step 4/8 : ENV JAVA_HOME=/usr/local/java  ---> Running in 9db3d99fdd60 Removing intermediate container 9db3d99fdd60  ---> 863a0540193c Step 5/8 : ENV PATH=$PATH:/usr/local/java/bin  ---> Running in 255f91b46b82 Removing intermediate container 255f91b46b82  ---> d1f66c55525b Step 6/8 : ENV JENKINS_HOME=/var/jenkins_home  ---> Running in 81b269c393e9 Removing intermediate container 81b269c393e9  ---> 682e163084e7 Step 7/8 : WORKDIR /home  ---> Running in a0c991355449 Removing intermediate container a0c991355449  ---> 808924159ee5 Step 8/8 : CMD java -jar jenkins.war --httpPort=8088  ---> Running in f75e74dfd0b8 Removing intermediate container f75e74dfd0b8  ---> 5cc8f40ad092 Successfully built 5cc8f40ad092 Successfully tagged 10.83.35.11/library/jenkins-self:2.222.4 [root@node-1jenkins]#     [root@node-1 jenkins]# docker images |grep jenkins 100.100.100.11/library/jenkins-self            2.222.4               5cc8f40ad092        2 minutes ago       1.02GB [root@node-1 jenkins]#     [root@node-1 jenkins]# [root@node-1 jenkins]# docker push 100.100.100.11/library/jenkins-self:2.222.4 The push refers to repository [100.100.100.11/library/jenkins-self] e233a814813a: Pushed e2b9f19efa22: Pushed 25710d453e5e: Pushed 2.222.4: digest: sha256:d0588b058d8a48df8a40d6a3de3f9a56b8e607d350bca2eee0beb902dad1c01c size: 955 [root@node-1 jenkins]#

1.4: 构建Jenkins-agent的dockerfile-agent   [root@node-1 jenkins]# ls agent.jar agent.jar [root@node-1 jenkins]# ls slave-agent.jnlp slave-agent.jnlp [root@node-1 jenkins]#   [root@node-1 jenkins]# cat dockerfile-agent FROM openvz-centos7 ADD jdk-8u77-linux-x64.tar.gz  /home ADD agent.jar /home ENV JAVA_HOME=/usr/local/java ENV PATH=$PATH:/usr/local/java/bin RUN yum -y install docker kubernetes-client kubernetes* WORKDIR /home CMD exec /usr/local/java/bin/java -Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true -cp /home/agent.jar hudson.remoting.jnlp.Main -headless -url ${JENKINS_URL} -workDir ${JENKINS_AGENT_WORKDIR} ${JENKINS_SECRET} ${JENKINS_AGENT_NAME} [root@node-1 jenkins]#   构建后推送到本地仓库 [root@node-1 jenkins]# docker import centos-7-x86_64.tar.gz openvz-centos7 sha256:9ff5144003abed64ddf8508417866d1f379c6900ef4472c6c709487ff3ceafd8 [root@node-1 jenkins]# docker build -t  100.100.100.11/library/jenkins-agent-self:2.222.4 -f dockerfile-agent . Sending build context to Docker daemon  1.197GB Step 1/8 : FROM openvz-centos7  ---> 9ff5144003ab Step 2/8 : ADD jdk-8u77-linux-x64.tar.gz  /home  ---> 8ad24fdbd943 Step 3/8 : ADD agent.jar /home  ---> f79dfeea5e0b Step 4/8 : ENV JAVA_HOME=/usr/local/java  ---> Running in 7c94f9849806 Removing intermediate container 7c94f9849806  ---> 3979bc6e1da1 Step 5/8 : ENV PATH=$PATH:/usr/local/java/bin  ---> Running in b02540d0e6b7 Removing intermediate container b02540d0e6b7  ---> 0087a373ca89 Step 6/8 : RUN yum -y install docker kubernetes-client kubernetes*  ---> Running in 3e9cca805163 Loaded plugins: fastestmirror Complete! Removing intermediate container 3e9cca805163  ---> 497b57fe08e8 Step 7/8 : WORKDIR /home  ---> Running in e3e350e53466 Removing intermediate container e3e350e53466  ---> cc0885a443a7 Step 8/8 : CMD exec /usr/local/java/bin/java -Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true -cp /home/agent.jar hudson.remoting.jnlp.Main -headless -url ${JENKINS_URL} -workDir ${JENKINS_AGENT_WORKDIR} ${JENKINS_SECRET} ${JENKINS_AGENT_NAME}  ---> Running in 93454e655ab1 Removing intermediate container 93454e655ab1  ---> 9932a57a6a34 Successfully built 9932a57a6a34 Successfully tagged 10.83.35.11/library/jenkins-agent-self:2.222.4 [root@node-1 jenkins]#   推送到镜像仓库 [root@node-16 jenkins]# docker push 100.100.100.11/library/jenkins-agent-self:2.222.4 The push refers to repository [10.83.35.11/library/jenkins-agent-self] e8e0c72b3559: Pushed 5e40f3cc9fc1: Pushed df4a455e9cce: Pushed 25710d453e5e: Mounted from library/jenkins-self 2.222.4: digest: sha256:ac4bbaa55c2396327893c9d5fc7967d0ba2d97cfb79520f38ecaca501acd4496 size: 1167 [root@node-1 jenkins]#     ###################################################################################################   agent.jar可以从Jenkins中下载。manager Jenkins --> 节点管理 --> 新建节点 --> 输入test，选择固定节点 --> 远程工作目录输入 /home --> 启动方式选择通过web web启动代理后点击保存。 这里这个节点是不在线的，点击下面界面显示的agent.jar后就可以下载对应的agent.jar包

二.部署Jenkins到k8s中

[root@node-1 jenkins]# cat class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata:   name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters:   archiveOnDelete: "true" [root@node-1 jenkins]#       [root@node-1 jenkins]# cat jenkins-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata:   name: jenkins-admin       #ServiceAccount名   namespace: default    #指定namespace，一定要修改成你自己的namespace   labels:     name: jenkins --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata:   name: jenkins-admin   labels:     name: jenkins subjects:   - kind: ServiceAccount     name: jenkins-admin     namespace: default roleRef:   kind: ClusterRole   name: cluster-admin   apiGroup: rbac.authorization.k8s.io [root@node-1 jenkins]#   [root@node-1 jenkins]# cat jenkins-deployment.yaml apiVersion: v1 kind: Service metadata:   name: jenkins   labels:     app: jenkins spec:   type: NodePort   ports:   - name: http     port: 8080         #服务端口     targetPort: 8080     nodePort: 32001   #NodePort方式暴露 Jenkins 端口   - name: jnlp     port: 50000         #代理端口     targetPort: 50000     nodePort: 32002   selector:     app: jenkins --- apiVersion: apps/v1 kind: Deployment metadata:   name: jenkins   labels:     app: jenkins spec:   selector:     matchLabels:       app: jenkins   replicas: 1   template:     metadata:       labels:         app: jenkins     spec:       serviceAccountName: jenkins-admin       containers:       - name: jenkins         image: jenkins/jenkins:lts         securityContext:                               runAsUser: 0       #设置以ROOT用户运行容器           privileged: true   #拥有特权         ports:         - name: http           containerPort: 8080         - name: jnlp           containerPort: 50000         resources:           limits:             memory: 2Gi             cpu: "1000m"           requests:             memory: 1Gi             cpu: "500m"         env:         - name: LIMITS_MEMORY           valueFrom:             resourceFieldRef:               resource: limits.memory               divisor: 1Mi         - name: "JAVA_TOOL_OPTIONS"           value: "                   -Dfile.encoding=UTF-8                   -Dsun.jnu.encoding=UTF-8                  "              - name: "JAVA_OPTS"  #设置变量，指定时区和 jenkins slave 执行者设置           value: "                   -Xmx$(LIMITS_MEMORY)m                   -XshowSettings:vm                   -Dhudson.slaves.NodeProvisioner.initialDelay=0                   -Dhudson.slaves.NodeProvisioner.MARGIN=50                   -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85                   -Duser.timezone=Asia/Shanghai                  "            - name: "JENKINS_OPTS"           value: "--prefix=/jenkins"         #设置路径前缀加上 Jenkins         volumeMounts:                        #设置要挂在的目录         - name: data           mountPath: /var/jenkins_home       volumes:       - name: data         persistentVolumeClaim:           claimName: jenkins      #设置PVC --- kind: PersistentVolumeClaim apiVersion: v1 metadata:   name: jenkins spec:   storageClassName: "managed-nfs-storage"   accessModes:     - ReadWriteOnce   resources:     requests:       storage: 2Gi      #生产环境空间一定要设置比较大点 #  selector: #    matchLabels: #      app: jenkins [root@node-1 jenkins]#       [root@node-1 jenkins]# cat ingress.yaml apiVersion: v1 kind: List items: - apiVersion: extensions/v1beta1   kind: Ingress   metadata:     name: jenkins     annotations:       nginx.ingress.kubernetes.io/ssl-redirect: "true"       nginx.ingress.kubernetes.io/proxy-body-size: 100m   spec:     rules:     - host: jenkins.test.com       http:         paths:         - path:           backend:             serviceName: jenkins             servicePort: 8080 [root@node-1jenkins]#     [root@node-1 jenkins]# kubectl get pod,svc,ing |grep jenkins pod/jenkins-677485858c-m2wng                  1/1     Running   0          3d service/jenkins            NodePort    10.0.0.112   <none>        8080:32001/TCP,50000:32002/TCP   3d ingress.extensions/jenkins            jenkins.ctnrs.com             80      3d [root@node-1 jenkins]#

三.需要安装插件：

 

Jenkins需要安装插件Kubernetes plugin插件才可以动态调用k8s的api接口完成pod的创建

 

4.配置Jenkins调用k8s配置

 

配置Jenkins代理为固定端口50000，因为service定义是对外暴露的端口是50000。

点击Jenkins的节点管理

配置Kubernetes地址为http://kubernetes.default:433，命名空间为default，和Jenkins是同一个命名空间，然后点击连接测试。因为Jenkins的pod在提交是配置了rbac授权，所以可以访问k8s。

配置Jenkins地址为http://jenkins.kube-ops.svc.cluster.local:32001

配置pod模版，名字配置为jenkins-slave-001，命名空间为default，标签列表为haimaxy-jnlp，这个标签列表 很重要，后面需要用到。容器列表配置jnlp， 镜像就是自己构建的100.100.100.11/library/jenkins-agent-self:2.222.4，工作目录写/home/jenkins           下面的运行的命令和参数都不要写，否则会覆盖掉镜像中定义的启动命令。

[root@node-1 ~]# ls /var/run/docker.sock /var/run/docker.sock [root@node-1 ~]# ls /home/jenkins/.kube ls: cannot access /home/jenkins/.kube: No such file or directory [root@node-1 ~]# ls /root/.kube cache  http-cache [root@node-1 ~]#     这里把/var/run/docker.sock和/home/jenkins/.kube挂载到容器中，这样可以使用docker和kubectl命令。注意kubectl必须在每个节点都可以执行。

五: 创建slave测试

添加一个项目来测试。

这里选择节点标签就是前面的定义的haimaxy-jnlp。

写入shell命令。 sleep 10 hostname echo "---------" ls / echo "---------" env echo "----sleep 100-----" sleep 100 date docker info kubectl get pods date

添加完成后开始构建

构建过程，就是在k8s中创建了一个pod来运行。

查看构建日志

此时在k8s中查看命名空间kube-ops下有自动创建的pod。

执行完成后查看日志，发现同样获取了命名空间下的pod信息。