在k8s上面运行jenkins

最新推荐文章于 2023-08-14 16:28:57 发布

月夜楓

最新推荐文章于 2023-08-14 16:28:57 发布

阅读量1.3k

点赞数

分类专栏：自动化部署文章标签： jenkins

本文链接：https://blog.csdn.net/cyxinda/article/details/107254673

版权

自动化部署专栏收录该内容

2 篇文章 0 订阅

订阅专栏

自定义jenkins镜像

使用官方的基础镜像
在k8s上面拉起jenkins
jenkins-slave
在jenkins上面配置jenkins：
- 异常处理：
- - 错误信息

使用官方的基础镜像

先写一个总的，使用官方提供的jenkins master的基础镜像，构建自己的dockerfile，如下：

FROM jenkins/jenkins:2.256
LABEL maintainer=caoyong1
USER root
WORKDIR /usr/local/
COPY src/ .
ENV JENKINS_UC=https://updates.jenkins.io \
    JENKINS_UC_DOWNLOAD=http://mirror.xmission.com/jenkins \
    JENKINS_OPTS="-Dhudson.model.UpdateCenter.updateCenterUrl=https://cdn.jsdelivr.net/gh/jenkins-zh/update-center-mirror/tsinghua/current/update-center.json"
RUN echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    cat /etc/resolv.conf && apt-get update && apt-get install -y --allow-unauthenticated --assume-yes apt-utils lsb-core bridge-utils traceroute ca-certificates curl netbase wget vim git curl unzip net-tools iputils-ping && ln -sf /usr/local/Asia/Shanghai /etc/localtime && sed -i "20i set -e" /usr/local/bin/install-plugins.sh && mkdir -p /usr/share/jenkins/ref/plugins/ && mv /usr/local/plugins.txt /usr/share/jenkins/ref/plugins/ && /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins/plugins.txt && date
EXPOSE 8080
EXPOSE 50000
ENTRYPOINT ["/sbin/tini","--","/usr/local/bin/jenkins.sh"]

引入工具集

引入jdk、nodejs、maven、sonarqube等
在dockerfile中是这个位置：
在这里插入图片描述
目录结构是这样的：

src/
├── Asia
├── jdk
│   ├── jdk-11.0.7
│   └── jdk1.8.0_181
├── maven
│   └── 3.6.3
├── node
│   └── node-v12
├── plugins.txt
└── sonar
    └── sonar-scanner-4.4

jenkins镜像生成过程中安装插件

需要在jenkins 插件的官网的镜像上面，找到插件id，将需要安装的插件的id以及版本号写入到插件的列表文件plugins.txt中，如下：

cloudbees-folder:6.14
antisamy-markup-formatter:2.1
build-timeout:1.20
credentials-binding:1.23
timestamper:1.11.5
ws-cleanup:0.38
ant:1.11
gradle:1.36
pipeline-stage-view:2.15
pipeline-github-lib:1.0
github-branch-source:2.9.0
workflow-aggregator:2.6
git:4.3.0
subversion:2.13.1
ssh-slaves:1.31.2
matrix-auth:2.6.2
mailer:1.32
email-ext:2.75
ldap:1.24
pam-auth:1.6
localization-zh-cn:1.0.18
generic-webhook-trigger:1.67
pipeline-utility-steps:2.6.1
kubernetes:1.27.1
kubernetes-credentials:0.7.0
kubernetes-pipeline-devops-steps:1.6
kubernetes-cli:1.8.3
kubernetes-client-api:4.9.2-2
nodejs:1.3.7
sonar:2.12

将插件的更新网址和下载地址进行环境变量的配置，如上面的dockerfile中：

ENV JENKINS_UC=https://updates.jenkins.io \
    JENKINS_UC_DOWNLOAD=http://mirror.xmission.com/jenkins \
    JENKINS_OPTS="-Dhudson.model.UpdateCenter.updateCenterUrl=https://cdn.jsdelivr.net/gh/jenkins-zh/update-center-mirror/tsinghua/current/update-center.json"

使用debian源

国外的源下载忒慢了，更新为阿里源：

RUN echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \

安装必要的工具软件，并且将插件安装在镜像构建的时候创建完成：

RUN apt-get update && apt-get install -y --allow-unauthenticated --assume-yes apt-utils lsb-core bridge-utils traceroute ca-certificates curl netbase wget vim git curl unzip net-tools iputils-ping && ln -sf /usr/local/Asia/Shanghai /etc/localtime && sed -i "20i set -e" /usr/local/bin/install-plugins.sh && mkdir -p /usr/share/jenkins/ref/plugins/ && mv /usr/local/plugins.txt /usr/share/jenkins/ref/plugins/ && /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins/plugins.txt && date

在k8s上面拉起jenkins

master.yaml如下：

apiVersion: v1
kind: Namespace
metadata:
  name: devops
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: master-deploy
  namespace: devops
  labels:
    app: jenkins-deploy
spec:
  serviceName: master
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
      jenkins: master
  template:
    metadata:
      labels:
        app: jenkins
        jenkins: master
    spec:
      containers:
      - name: master
        image: 10.110.152.173/jenkins/master:v1.0.0.3
        imagePullPolicy:  Always
        ports:
          - containerPort: 8080
            name: http-port
          - containerPort: 50000
            name: jnlp-port
        env:
        - name: JAVA_OPTS
          value: -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: maven-cache
          mountPath: /usr/local/.repository
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: master-home
      - name: maven-cache
        persistentVolumeClaim:
          claimName: maven-cache
---
apiVersion: v1
kind: Service
metadata:
  name: ui
  namespace: devops
spec:
  selector:
    app: jenkins
    jenkins: master
  ports:
  - name: http-port
    protocol: TCP
    port: 80
    targetPort: 8080
    nodePort: 32456
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: master
  namespace: devops
spec:
  selector:
    app: jenkins
    jenkins: master
  ports:
  - name: jnlp-port
    port: 80
    targetPort: 50000
    protocol: TCP

jenkins-slave

构建jenkins-slave镜像，添加必要的工具集，如：docker-cli,kubectl,pip(python)等
如下：

FROM jenkins/jnlp-slave:4.3-4
LABEL maintainer=caoyong1
WORKDIR /usr/local/
COPY src/ .
USER root
RUN rm -rf /etc/apt/sources.list.d/stretch-backports.list && echo ""> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian stretch-updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list && \
    echo "deb-src http://mirrors.aliyun.com/debian-security stretch/updates main contrib non-free" >> /etc/apt/sources.list
ARG DEBIAN_FRONTEND=noninteractive
RUN ln -sf /usr/local/Asia/Shanghai /etc/localtime && uname -a && cat /etc/debian_version &&  dpkg -i containerd.io_1.2.13-2_amd64.deb && dpkg -i docker-ce-cli_19.03.12_3-0_debian-stretch_amd64.deb &&  echo "10.110.149.169  harbor.aipo.lenovo.com" >> /etc/hosts && apt-get update && python -V && apt-get install python-pip -y && pip install nose coverage nosexcover pylint&& pip install s3cmd && cp /usr/local/.s3cfg /root/ && dpkg -i kubectl_1.18.6-00_amd64_339673592303e280949d52b2f4e842ad365529a5d656dd9f6483517651e2ded8.deb && mkdir -p /root/.kube/ && touch /root/.kube/config && chown $(id -u):$(id -g) /root/.kube/config &&  cat /usr/local/admin.conf >> /root/.kube/config && kubectl version && date
ENTRYPOINT ["jenkins-agent"]

src目录结构如下：

src
├── admin.conf
├── .s3cfg
├── Asia
├── containerd.io_1.2.13-2_amd64.deb
├── docker-ce-cli_19.03.12_3-0_debian-stretch_amd64.deb
├── jdk
│   ├── jdk-11.0.7
│   └── jdk1.8.0_181
├── kubectl_1.18.6-00_amd64_339673592303e280949d52b2f4e842ad365529a5d656dd9f6483517651e2ded8.deb
├── maven
│   └── 3.6.3
├── node
│   └── node-v12
├── plugins.txt
└── sonar
    └── sonar-scanner-4.4

在jenkins上面配置jenkins：

jenkins-k8s plugin配置
设置两个label
在这里插入图片描述
添加jenkins slave镜像：

在这里插入图片描述
当然，也有实践中，使用initContainer来进行工具集的添加，那样更具有扩展性，本文主要是为了减少每次启动时候，都要启动initContainer工具镜像的消耗，所以，在slave上面预先安装了不少工具集。

异常处理：

发现启动job的过程中，不能启动slave pod的情况，原因可能是devops（默认账号没有创建pod的权限），将账号的权限（角色）设置为最大：cluster-admin，可以解决该问题。

错误信息

2021-10-12 07:24:32.295+0000 [id=31]    INFO    hudson.slaves.NodeProvisioner#lambda$update$6: slave-6n6gq provisioning successfully completed. We have now 2 computer(s)
Disconnected computer slave-6n6gq
2021-10-12 07:24:32.326+0000 [id=554]   WARNING o.c.j.p.k.KubernetesSlave#_terminate: Slave pod slave-6n6gq was not deleted due to retention policy On Failure.
2021-10-12 07:24:32.326+0000 [id=554]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer slave-6n6gq
2021-10-12 07:24:42.263+0000 [id=35]    INFO    o.c.j.p.k.KubernetesCloud#provision: Excess workload after pending Kubernetes agents: 1
2021-10-12 07:24:42.263+0000 [id=35]    INFO    o.c.j.p.k.KubernetesCloud#provision: Template for label null: slave
2021-10-12 07:24:52.272+0000 [id=554]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: slave-scj1d, template=PodTemplate{inheritFrom='', name='slave', namespace='devops', hostNetwork=false, slaveConnectTimeout=180, label='jenkins-slave', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim@a77c69f], containers=[ContainerTemplate{name='jnlp', image='10.110.147.181:89/jenkins/slave:v4.3.4.13', workingDir='/home/jenkins/agent', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', envVars=[KeyValueEnvVar [getValue()=http://ui.devops.svc.cluster.local:80, getKey()=JENKINS_URL], KeyValueEnvVar [getValue()=/etc/kubernetes/ssl, getKey()=KUBE_CONFIG]], livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@5e7fa1ad}]}
java.lang.NullPointerException
        at org.csanchez.jenkins.plugins.kubernetes.PodUtils.cancelQueueItemFor(PodUtils.java:76)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:138)
        at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2021-10-12 07:24:52.272+0000 [id=554]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent slave-scj1d

kubectl create clusterrolebinding devops-permissive-binding --clusterrole=cluster-admin  --serviceaccount=devops:default

本文引用：
https://cloud.tencent.com/developer/article/1590856
https://blog.csdn.net/u014249394/article/details/90272768
https://github.com/jenkin/docker/blob/master/README.md

月夜楓

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
在k8s上面运行jenkins

自定义jenkins镜像jenkins镜像生成过程中安装插件新的改变功能快捷键合理的创建标题，有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列表创建一个表格设定内容居中、居左、居右SmartyPants创建一个自定义列表如何创建一个注脚注释也是必不可少的KaTeX数学公式新的甘特图功能，丰富你的文章UML 图表FLowchart流程图导出与导入导出导入jenkins镜像生成过程中安装插件https://cloud.tencent.com/developer/ar
复制链接

扫一扫