CI/CD常用服务部署
Jenkins && Github
docker-compose文件
version: "3"
services:
jenkins:
container_name: jenkins
image: jenkins:latest
restart: always
user: root
ports:
- 8080:8080
- 50000:50000
volumes:
- /data/jenkins/jenkins_home:/var/jenkins_home
- /etc/localtime:/etc/localtime
deploy:
resources:
limits:
cpus: '2'
memory: 2G
gitlab:
container_name: gitlab
image: gitlab/gitlab-ce:latest
restart: always
environment:
TZ: 'Asia/Shanghai'
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://192.168.2.10'
ports:
- 33022:22/tcp
- 80:80
- 443:443
volumes:
- /data/gitlab/config:/etc/gitlab
- /data/gitlab/data:/var/opt/gitlab
- /data/gitlab/logs:/var/log/gitlab
- /etc/localtime:/etc/localtime
deploy:
resources:
limits:
cpus: '2'
memory: 2G
Harbor镜像仓库
前置条件: 已安装docker-compose
下载地址
https://github.com/goharbor/harbor/releases
tar -zxf harbor-offline-installer-v2.1.0.tgz
mv harbor /usr/local/
cp harbor.yml.tmpl harbor.yml
## 编辑配置
vim harbor.yml
hostname: 192.168.2.10
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 85
# https related config
#https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path
# 修改data目录
data_volume: /data/harbor/data
# 修改log目录
location: /data/harbor/log/harbor
## 安装
./install.sh
访问: ip:85
默认账户密码：admin/Harbor12345
配置项目,用户
docker登录私有仓库
docker login --username=wangshui898 http://192.168.2.10:85
启动关闭
进入目录下
docker-compose up -d 启动
docker-compose stop 停止
docker-compose restart 重新启动
上传镜像
配置daemon.json,把镜像仓库地址设置为可信地址
vi /etc/docker/daemon.json
{
"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.2.10:85"]
}
重启docker生效
systemctl restart docker
Jenkins添加Kubenetes集群
生成所需K8S集群证书
admin帐号请求证书json文件
cat >> admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "hangzhou",
"L": "XS",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
生成admin证书
- 前置条件: 已有k8s证书生成工具,有集群ca相关配置和公钥私钥(说白了就是部署集群时候用的那套工具以及ca配置和证书)
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
ls
admin.csr admin-csr.json admin-key.pem admin.pem
生成pfx格式证书
生成pfx格式证书,并下载到电脑
openssl pkcs12 -export -out ./jenkins-admin.pfx -inkey ./admin-key.pem -in ./admin.pem -passout pass:pfx_passwd
ls jenkins-admin.pfx
jenkins-admin.pfx
注意: -passout 参数后的pass:pfx_passwd ,pfx_passwd是fpx文件格式的密码,上传jenkins时候需要
证书上传到jenkins
在凭据管理中选择Certificate类型–>Upload PKCS#12 certificate证书,并根据提示上传之前的jenkins-admin.pfx
注意填写密码,点击Change Password,填入刚刚设置的pfx_passwd
配置jenkins云节点,添加kubenetes集群
插件: kubenetes
方法一:
系统管理-> 系统配置 -> cloud;点击跳转连接a separate configuration page,会跳转到k8s集群配置页面
方法二:
系统管理-> 节点管理 -> Configure Clouds
选择 Add a new cloud -> kubenetes,点击Kubernetes Cloud details…进行配置
- 集群名称随意填写
- Kubernetes 地址填写apiserver对外访问地址
- Kubernetes 服务证书 key复制ca.pem公钥内容
- 凭据选择上传的pfx证书
点击连接测试,如出现success(老版本)或者Connected to Kubernetes v1.18.10,表示连接k8s集群成功
如果jenkins部署在k8s集群内部,则Kubernetes地址只需要填写k8s集群名称即可,可通过kubectl get svc获取集群名称
制作jenkins-slave镜像
获取agent.jar
也叫slave.jar,实际jenkins中的remoting
官网地址
https://www.jenkins.io/zh/projects/remoting/
jar包下载地址
https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/
获取jenkins-slave
git地址
https://github.com/jenkinsci/docker-inbound-agent/blob/master/jenkins-agent
编辑Dockerfile
这里只加了jdk环境和maven环境,可以根据实际情况添加
ls -1
apache-maven-3.6.3-bin.tar.gz
Dockerfile
jdk-8u251-linux-x64.tar.gz
jenkins-agent
remoting-3.40.1.jar
remoting-4.6.jar
settings.xml
kubectl
FROM centos:7
RUN yum install git crul wget openssl openssl-devel\
&& mkdir -p /usr/share/jenkins
ADD jdk-8u251-linux-x64.tar.gz /usr/local/
ADD apache-maven-3.6.3-bin.tar.gz /usr/local/
COPY settings.xml /usr/local/apache-maven-3.6.3/conf/settings.xml
COPY jenkins-agent /usr/local/bin/jenkins-slave
COPY remoting-3.40.1.jar /usr/share/jenkins/agent.jar
COPY kubectl /sbin/kubectl
RUN chmod 755 /usr/share/jenkins/agent.jar \
&& chmod +x /usr/local/bin/jenkins-slave
USER root
WORKDIR /home/jenkins
ENV M3_HOME=/usr/local/apache-maven-3.6.3
ENV JAVA_HOME=/usr/local/jdk1.8.0_251
ENV PATH=${PATH}:${JAVA_HOME}/bin:${M3_HOME}/bin
ENTRYPOINT ["jenkins-slave"]
#ENTRYPOINT ["/bin/ping"]
#CMD ["www.baidu.com"]
remoting.jar具体重命名和所放目录,要参照jenkins-slave脚本的执行命令参数来修改
使用jenkins部署项目到K8S
生成kube-admin.kubeconfig文件
插件: Kubernetes Continuous Deploy
生成admin用户的kubeconfig文件
ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem
ls -1 /opt/k8s/ssl/ca*
ca-key.pem ca.pem
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.2.100:6443 \
--kubeconfig=kube-admin.kubeconfig
# 设置客户端认证参数-设置证书信息
kubectl config set-credentials admin \
--client-certificate=/data/TLS/k8s/admin.pem \
--client-key=/data/TLS/k8s/admin-key.pem \
--embed-certs=true \
--kubeconfig=kube-admin.kubeconfig
# 设置上下文参数-绑定账号和管理的集群
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=kube-admin.kubeconfig
# 设置默认上下文
kubectl config use-context kubernetes \
--kubeconfig=kube-admin.kubeconfig
–embed-certs=true表示把证书内容写入到文件中
将kube-admin.kubeconfig保存到jenkins
安装好Kubernetes Continuous Deploy插件后,凭据管理中选择Kubernetes configuration (kubeconfig)–>Enter directily,将kube-admin.kubeconfig内容拷贝到输入框,保存.
jenkins发布到K8S逻辑说明
jenkins生成动态jenkins-slave节点,里面有kubectl命令,通过凭据保存的kube-admin.kubeconfig证书文件,在节点内保存到/root/.kube/config,动态节点即可操作集群,通过项目代码内的deployment.yaml发布到集群
配置文件示例
deployment.yaml
apiVersion: v1
kind: Namespace
metadata:
name: NS_NAME
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: APP_NAME
namespace: NS_NAME
labels:
app: APP_NAME
spec:
replicas: RS_NUM
selector:
matchLabels:
app: APP_NAME
template:
metadata:
labels:
app: APP_NAME
spec:
imagePullSecrets:
- name: SECRET_NAME
containers:
- image: IMAGE_NAME
name: APP_NAME
---
apiVersion: v1
kind: Service
metadata:
labels:
app: APP_NAME
name: APP_NAME-service
namespace: NS_NAME
spec:
ports:
- port: 8091
protocol: TCP
targetPort: 8080
selector:
app: APP_NAME
Jenkinsfile
// 插件: pipeline / git / Git Parameter / Kubernetes / Kubernetes Continuous Deploy
// 项目信息
def project_name = "apps"
def app_name = "book"
// 镜像信息
def harbor_url = "192.168.2.10:85"
def harbor_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
def image_tag = "${BUILD_NUMBER}"
def image_name = "${harbor_url}/${project_name}/${app_name}:${image_tag}"
// Git信息
def git_url = "http://192.168.2.10/ipanel/book.git"
def git_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
// k8s集群信息
def k8s_auth = ""
def k8s_kubeconfig_auth = "a37784ad-f7f2-49df-96f5-f1dc9b5ebd6b"
def Secret_Name = "jenkins-slave-secret"
pipeline {
agent {
kubernetes {
cloud 'kubernetes'
defaultContainer 'jnlp'
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: jnlp
image: ${harbor_url}/library/jenkins-slave:v1
imagePullPolicy: Always
volumeMounts:
- mountPath: /etc/resolv.conf
name: resolv
readOnly: false
- mountPath: /var/run/docker.sock
name: docker-sock
readOnly: false
- mountPath: /data/maven/repos
name: maven-date
readOnly: false
- mountPath: /usr/bin/docker
name: docker-bin
readOnly: false
- mountPath: /home/jenkins/agent
name: agent-home
readOnly: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: Never
volumes:
- hostPath:
path: /usr/bin/docker
name: docker-bin
- hostPath:
path: /etc/resolv.conf
name: resolv
- hostPath:
path: /var/run/docker.sock
name: docker-sock
- emptyDir:
name: agent-home
- name: maven-date
nfs:
path: /data/NFS/maven
readOnly: false
server: 192.168.2.10
"""
}
}
parameters {
gitParameter name: 'Branch',
type: 'PT_BRANCH',
branchFilter: '.*',
defaultValue: 'master',
selectedValue: 'NONE',
sortMode: 'NONE',
description: 'Select your branch or tag.'
choice(name: 'ReplicasNum', choices: ['1', '3', '5'], description: '请选择副本数')
choice(name: 'NameSpace', choices: ['default', 'dev', 'apps'], description: '请选择命名空间')
}
stages {
stage('源码拉取') {
steps {
container('jnlp') {
checkout([$class: 'GitSCM',
branches: [[name: "${Branch}"]],
doGenerateSubmoduleConfigurations: false,
extensions: [], submoduleCfg: [],
userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]])
sh 'ls'
}
}
}
stage('代码构建') {
steps {
container('jnlp') {
sh "mvn clean package -Dmaven.test.skip=true"
}
}
}
stage('镜像构建') {
steps {
container('jnlp') {
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
sh """
echo "Docker Login"
docker login -u "$user" -p "$passwd" "${harbor_url}"
echo "Add Dockfile"
echo '
FROM ${harbor_url}/devops/centos-jdk1.8:v1
COPY target/*.jar /home/app/book.jar
USER root
WORKDIR /home/app
EXPOSE 8080
ENTRYPOINT ["java","-jar","book.jar"]
' > Dockerfile
echo "Docker Build"
docker build -t ${image_name} .
echo "Docker push"
docker push ${image_name}
echo "Docker rm images"
docker rmi ${image_name}
"""
}
}
}
}
stage('发布到K8S') {
steps {
container('jnlp') {
withCredentials([kubeconfigContent(credentialsId: "${k8s_kubeconfig_auth}", variable: 'KUBECONFIG_CONTENT')]) {
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
sh """
mkdir /root/.kube
echo "${KUBECONFIG_CONTENT}" > /root/.kube/config
sed -i 's#IMAGE_NAME#${image_name}#g' deployment.yaml
sed -i 's#APP_NAME#${app_name}#g' deployment.yaml
sed -i 's#NS_NAME#${params.NameSpace}#g' deployment.yaml
sed -i 's#RS_NUM#${params.ReplicasNum}#g' deployment.yaml
sed -i 's#SECRET_NAME#${Secret_Name}#g' deployment.yaml
cat deployment.yaml
kubectl create secret docker-registry "${Secret_Name}" --docker-server="${harbor_url}" --docker-username="$user" --docker-password="$passwd" -n "${params.NameSpace}" |true
kubectl apply -f deployment.yaml --record
kubectl get node
kubectl get deploy -o wide -n "${params.NameSpace}"
kubectl get pod -o wide -n "${params.NameSpace}"
kubectl rollout history deploy/${app_name} -n "${params.NameSpace}"
"""
}
}
}
}
}
}
}
如果使用NFS挂载方式,node节点需要安装nfs-utils
1493
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
