环境准备
首先需要装好一个k8s集群和kubesphere,安装这些的过程这里不做演示了
[root@k8s-node3 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node2 Ready control-plane,master 11d v1.23.17
k8s-node3 Ready <none> 11d v1.23.17
k8s-node5 Ready <none> 11d v1.23.17
DevOps项目
首先需要调试好一个完整的DevOps部署的系统,里面包含了前端、后端、中间件(Redis、Nacos)
部署之前要创建好专门的工作空间和企业空间,工作空间里要提前设置好存储和配置
像Dockerfile、deployment.yaml、Jenkinsfile这种东西,要尽量做成通用版,把一些需要替换的东西都改成获取变量,减少后期重复操作
存储
创建了工作空间以后,创建几个存储类,我创建了两个分别给redis存储数据和日志文件,存储量给小一点,2g一般都够用了
配置
工作空间配置
配置分保密字典和配置字典,保密字典用来放一些账号密码,配置字典用来存放服务相关配置
工作空间的配置哟配置字典就够了,这里我创建了这样几个配置,有nginx、redis、前端的配置文件,kube-root-ca.crt是本身就有的不用管
系统配置
除了工作空间的配置,还要改下系统的配置,这个第一次用流水线的时候要改,改好了以后就不需要改了
如果公司用了私有的编译库,就要改下maven配置,在集群管理选择配置字典,搜索devops,选择ks-devops-agent,里面有个MavenSetting,这里就是maven的配置文件,把私库的地址加进去
用maven下载的依赖如果不用存储来保存的话,每次都要重新下载依赖包,这样很浪费时间,所以要保存每次下载的依赖
搜索jenkins,在jenkins-casc-config里把jenkins_user.yaml配置的maven里的路径重新设置一下,/data/nfs/maven_repo/这个路径是我在nfs里创建的
凭证
创建DevOps项目,先创建凭证,这个和保密字典类似,用来在流水线里连Git和Harbor的用户
我这里创建了GitLab和Harbor的管理员用户
admin-kubeconfig是kubeconfig类型,不是用户密码的类型,创建的时候选好类型就可以了
Dockerfile和deployment.yaml
Dockerfile是生成容器镜像用的编排文件,deployment.yaml是将服务部署到集群的编排文件
两个文件都要添加到Git相关分支内,前端和后端都需要
deployment要注意,如果是不需要映射到外网的服务,就把倒数两行nodePort相关的注释掉
deployment里面#开头的内容都是命名空间、pod名、端口号等参数,这些都会在jenkinsfile里用sed替换掉
deployment要添加副本数就改replicas后面的值,这个只适用后端程序,如果是nacos或者redis这种是不能通过直接改这个值来实现负载均衡的,中间件需要在这个工作空间里部署集群
FROM 192.168.0.192/basic_images/gm_centos79_jdk:latest
LABEL CHENWENCONG="cwc@123.com"
ARG SERVICE_NAME
ARG PORT
ENV JVM_OPT -Dnacos.standalone=true
RUN mkdir -p /data/log/${SERVICE_NAME} && mkdir -p /data/register4/app
COPY lib/${SERVICE_NAME}.jar /data/register4/app/app.jar
CMD ["sh", "-c", "java $JVM_OPT -jar /data/register4/app/app.jar"]
EXPOSE ${PORT}
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: #service_name
namespace: #namespace
labels:
app: #service_name
annotations:
deployment.kubernetes.io/revision: '1'
kubesphere.io/creator: cwc
spec:
replicas: 1
selector:
matchLabels:
app: #service_name
template:
metadata:
creationTimestamp: null
labels:
app: #service_name
annotations:
kubesphere.io/creator: cwc
kubesphere.io/imagepullsecrets: '{}'
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
containers:
- name: #service_name
image: '#registry/#project_name/#service_name:#version'
ports:
- name: #service_name
containerPort: #service_port
protocol: TCP
resources:
limits:
memory: 2Gi
volumeMounts:
- name: host-time
readOnly: true
mountPath: /etc/localtime
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
---
apiVersion: v1
kind: Service
metadata:
namespace: #namespace
labels:
app: #service_name
name: #service_name
spec:
sessionAffinity: None
selector:
app: #service_name
ports:
- name: #service_name
protocol: TCP
targetPort: #service_port
port: #service_port
nodePort: #nat_port
type: NodePort
工作负载
我需要创建一个Redis的pod给系统使用,因为Redis基本上不会更新,所以不需要流水线
在工作负载里创建Redis的工作负载(pod),在服务里创建Redis对应的服务(service)
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: redis
namespace: register4-dev
labels:
app: redis
annotations:
deployment.kubernetes.io/revision: '1'
kubesphere.io/creator: chenwencong
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
creationTimestamp: null
labels:
app: redis
annotations:
kubesphere.io/creator: chenwencong
kubesphere.io/imagepullsecrets: '{}'
kubesphere.io/restartedAt: '2023-08-04T03:33:16.040Z'
logging.kubesphere.io/logsidecar-config: '{}'
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
- name: volume-kqm5pe
persistentVolumeClaim:
claimName: redis-data
- name: volume-ilw6vd
persistentVolumeClaim:
claimName: redis-log
- name: volume-fe8kw6
configMap:
name: register4-redis
items:
- key: redis.conf
path: redis.conf
defaultMode: 420
containers:
- name: redis
image: '192.168.0.192/basic_images/gm_redis:7.0.8'
command:
- redis-server
args:
- /etc/redis/redis.conf
ports:
- name: redis
containerPort: 6379
protocol: TCP
resources:
limits:
memory: 1Gi
requests:
memory: 512Mi
volumeMounts:
- name: host-time
readOnly: true
mountPath: /etc/localtime
- name: volume-kqm5pe
mountPath: /data
- name: volume-ilw6vd
mountPath: /logs
- name: volume-fe8kw6
readOnly: true
mountPath: /etc/redis/
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
---
kind: Service
apiVersion: v1
metadata:
name: redis
namespace: register4-dev
labels:
app: redis
annotations:
kubesphere.io/creator: chenwencong
spec:
ports:
- name: redis
protocol: TCP
port: 6379
targetPort: 6379
selector:
app: redis
type: NodePort
sessionAffinity: None
流水线
开始创建流水线,目前这个系统里需要部署3个前端服务,2个后端服务,两个后端服务其中一个是nacos,第一次要一个个调试好
- 后端
- 先做jdk的基础镜像打包到harbor
- maven的相关配置要提前设置好
- 前端
- 用nginx的镜像作为前端的基础镜像,配置里把要代理的服务用pod名替换掉
- 前端如果.git文件过大,可以在git步骤设置浅克隆,超时时间也改长一点,前端研发自行打包压缩文件上传到Git里,流水线跳过编译的步骤
最好调试完成,每个都能使用了,这里贴了部署后端和前端的jenkinsfile作为参考
def createVersion() {
// 定义一个版本号作为当次构建的版本,输出结果 20191210175842_69
return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}
pipeline {
agent {
kubernetes {
inheritFrom 'maven'
}
}
stages {
stage('Checkout scm') {
agent none
steps {
container('maven') {
git(url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq', changelog: true, poll: false, branch: "${params.BRANCH_NAME}")
}
}
}
stage('Compile') {
agent none
steps {
container('maven') {
sh "mvn clean install -pl ${params.SERVICE_NAME} -am -amd -Dmaven.test.skip=true"
}
}
}
stage('Archive the artifacts') {
agent none
steps {
container('maven') {
archiveArtifacts "${params.SERVICE_NAME}/target/*jar"
}
}
}
stage('Make && Push images') {
agent none
steps {
container('maven') {
withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
sh "docker image build --build-arg SERVICE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.SERVICE_NAME}/"
sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
sh "docker push ${registry}/${project_name}/${service_name1}:latest"
}
}
}
}
stage('Deploy to server') {
agent none
steps {
container('maven') {
withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
sh """cd ${params.SERVICE_NAME}
sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
sed -i \'s/#registry/${registry}/g\' deployment.yaml
sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
sed -i \'s/#service_port/${port1}/g\' deployment.yaml
sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
sed -i \'s/#version/${version}/g\' deployment.yaml
sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
cat deployment.yaml
mkdir ~/.kube
echo "$ADMIN_KUBECONFIG" > ~/.kube/config
cat ~/.kube/config
kubectl apply -f .
"""
}
}
}
}
}
environment {
service_name1 = 'estate-server' //服务1的名字
port1 = '8085' // 服务1的端口
nat_port1 = 'random' // 服务1的外网端口,如果是randon就是随机端口
registry = '192.168.0.192' // 内网harbor镜像仓库地址
project_name = 'register4.0_dev' // 服务在harbor的项目名
namespace = 'register4-dev' // 要把服务部署在哪个命名空间
version = createVersion() // 获取镜像构建的时间为构建版本
}
parameters {
string(name: 'BRANCH_NAME', defaultValue: 'devops', description: '请输入要构建的分支名称')
string(name: 'SERVICE_NAME', defaultValue: 'estate-lowcode-plateform-cloud-server', description: '请输入服务名')
string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/estate-lowcode-platform.git', description: 'GIT仓库后缀')
}
}
def createVersion() {
// 定义一个版本号作为当次构建的版本,输出结果 20191210175842_69
return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}
pipeline {
agent {
kubernetes {
inheritFrom 'nodejs'
}
}
stages {
stage('Clone repository') {
agent none
steps {
checkout([$class: 'GitSCM', doGenerateSubmoduleConfigurations: false, submoduleCfg: [], extensions: [[$class: 'CloneOption', noTags: false, timeout: 1000, shallow: true, depth: 1]],
branches: [[name: "${params.BRANCH_NAME}"]],userRemoteConfigs: [[url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq']]])
}
}
stage('Archive the artifacts') {
agent none
steps {
container('nodejs') {
archiveArtifacts "${params.PACKAGE_DIR}/${params.SERVICE_NAME}.zip"
}
}
}
stage('Make && Push images') {
agent none
steps {
container('nodejs') {
withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
sh "docker image build --build-arg SERVICE_NAME=${service_name1} --build-arg PACKAGE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.PACKAGE_DIR}/"
sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
sh "docker push ${registry}/${project_name}/${service_name1}:latest"
}
}
}
}
stage('Deploy to server') {
agent none
steps {
container('nodejs') {
withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
sh """cd ${params.PACKAGE_DIR}
sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
sed -i \'s/#registry/${registry}/g\' deployment.yaml
sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
sed -i \'s/#service_port/${port1}/g\' deployment.yaml
sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
sed -i \'s/#version/${version}/g\' deployment.yaml
sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
cat deployment.yaml
mkdir ~/.kube
echo "$ADMIN_KUBECONFIG" > ~/.kube/config
cat ~/.kube/config
kubectl apply -f .
"""
}
}
}
}
}
environment {
service_name1 = 'low-code-ui' //服务1的名字
port1 = '81' // 服务1的端口
nat_port1 = '30022' // 服务1的外网端口,如果是randon就是随机端口
registry = '192.168.0.192' // 内网harbor镜像仓库地址
project_name = 'register4.0_sx' // 服务在harbor的项目名
namespace = 'register4-sx' // 要把服务部署在哪个命名空间
version = createVersion() // 获取镜像构建的时间为构建版本
}
parameters {
string(name: 'BRANCH_NAME', defaultValue: 'shanxi', description: '请输入要构建的分支名称')
string(name: 'PACKAGE_DIR', defaultValue: 'package', description: '请输入打包文件夹')
string(name: 'SERVICE_NAME', defaultValue: 'low-code-ui', description: '请输入服务名')
string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/portal/low-code/low-code-ui.git', description: 'GIT仓库后缀')
}
}
第N次创建DevOps项目
保存好第一次配置的所有东西,它们基本上都能保存成yaml文件,它们就是你这个系统的DevOps项目的模板了
如果这个系统还需要部署多套环境,就可以用这些模板进行快速的部署
创建好新的工作空间→导入存储、配置字典→创建DevOps凭证→添加Dockerfile和deployment→导入jenkinsfile
其中配置字典里的前端配置肯定需要修改,因为端口号变了
jenkinsfile里要改工作空间名、harbor组织名、外网端口号等,都在最下面的变量里进行修改