Devops
Devops:是让技术团队,运维,测试等团队实现一体式自动化流程,从而繁衍出CI/CD持续集成和持续交付
持续集成:从开发-编译-测试-发布的完成自动化流程。{}
持续交付:包含持续集成,并且增加将项目部署到对应环境的自动化流程
流程图参考
(https://devops-1.oss.cn-north-3.inspurcloudoss.com/Devops.png)
环境搭建
安装gitlab
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm --no-check-certificate
安装gitlab
[root@server1 package]# rpm -i gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm
........
Thank you for installing GitLab!
GitLab was unable to detect a valid hostname for your instance.
Please configure a URL for your GitLab instance by setting `external_url`
configuration in /etc/gitlab/gitlab.rb file.
Then, you can start your GitLab instance by running the following command:
sudo gitlab-ctl reconfigure
...........
安装gitlab的依赖
[root@master package]# yum -y install openssh-server
修改配置文件
[root@master package]# vim /etc/gitlab/gitlab.rb
external_url 'http://172.31.0.16:28080'
gitlab_rails['time_zone'] = 'Asia/Shanghai'
puma['worker_processes'] = 2 ##工作节点数量
sidekiq['max_concurrency'] = 8 ##最大兵并发数量
postgresql['shared_buffers'] = "128MB" ##修改pgSQL缓存大小
postgresql['max_worker_processes'] = 4 ##pgSQL的进程数量
prometheus_monitoring['enable'] = false ##关闭prometheus
gitlab重新读取配置文件
[root@master package]# gitlab-ctl reconfigure
gitlab重新启动
[root@master package]# gitlab-ctl restart
查看自动生成的密码
cat /etc/gitlab/initial_root_password
.....
Password: rGdiXBNjz1xF697+QnXGrI8lBHqxRn88ZG83+sKSMp4=
......
访问gitlab页面
http://172.31.0.16:28080 这里的端口是「external_url」设置的端口
创建一个加密的secret文件,来存储账号和密码
[root@server1 gitlab]# cat username
root
[root@server1 gitlab]# cat password
admin123456
[root@server1 gitlab]# kubectl create secret generic git-user-pass --from-file=./username --from-file=./password -n kube-devops
# 查看默认密码
cat /etc/gitlab/initial_root_password
# 登录后修改默认密码 > 右上角头像 > Perferences > Password
# 修改系统配置:点击左上角三横 > Admin
# Settings > General > Account and limit > 取消 Gravatar enabled > Save changes
# 关闭用户注册功能
# Settings > General > Sign-up restrictions > 取消 Sign-up enabled > Save changes
# 开启 webhook 外部访问
# Settings > Network > Outbound requests > Allow requests to the local network from web hooks and services 勾选
# 设置语言为中文(全局)
# Settings > Preferences > Localization > Default language > 选择简体中文 > Save changes
# 设置当前用户语言为中文
# 右上角用户头像 > Preferences > Localization > Language > 选择简体中文 > Save changes
创建K8S中的serect文件
[root@master gitlab]# echo root > ./username ##创建用户名
[root@master gitlab]# echo wolfcode > password ##创建密码
[root@master gitlab]# kubectl create namespace kube-devops ##创建命名空间
[root@master gitlab]# kubectl create secret generic git-user-pass --from-file=./username --from-file=./password -n kube-devops
PS:注释
# 停止服务
gitlab-ctl stop
# 卸载 rpm 软件(注意安装的软件版本是 ce 还是 ee)
rpm -e gitlab-ce
# 查看进程
ps -ef|grep gitlab
# 干掉第一个 runsvdir -P /opt/gitlab/service log 进程
# 删除 gitlab 残余文件
find / -name *gitlab* | xargs rm -rf
find / -name gitlab | xargs rm -rf
安装Harbor
服务器上需要安装docker和docker-compose
1.安装docekr
.......省略
2.安装docker-compose
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.20.3/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
docker compose version
Docker Compose version v2.20.3
2.下载harbor的安装包
wget https://github.com/goharbor/harbor/releases/download/v2.4.3/harbor-offline-installer-v2.4.3.tgz
3.解压安装
tar xf harbor-offline-installer-v2.4.3.tgz -C /usr/local/
4.进入harbor目录,修改配置文件
[root@master harbor]# vim /mydata/harbor/harbor.yml
hostname: master ##修改为主机的主机名
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 81 ##修改harbor界面的访问端口
# https related config
#https: ##如果部署过程中涉及到https,则需要将这里注释掉
# https port for harbor, default is 443
#port: 443
# The path of cert and key files for nginx
certificate: /your/certificate/path
private_key: /your/private/key/path
........
4.修改完成后进行安装和部署
[root@master harbor]# ./install.sh
5.安装完成后,进行查看个验证
查看镜像:
[root@master harbor]# docker images | grep harbor
goharbor/harbor-exporter v2.4.3 776ac6ee91f4 12 months ago 81.5MB
goharbor/chartmuseum-photon v2.4.3 f39a9694988d 12 months ago 172MB
goharbor/redis-photon v2.4.3 b168e9750dc8 12 months ago 154MB
goharbor/trivy-adapter-photon v2.4.3 a406a715461c 12 months ago 251MB
goharbor/notary-server-photon v2.4.3 da89404c7cf9 12 months ago 109MB
goharbor/notary-signer-photon v2.4.3 38468ac13836 12 months ago 107MB
goharbor/harbor-registryctl v2.4.3 61243a84642b 12 months ago 135MB
goharbor/registry-photon v2.4.3 9855479dd6fa 12 months ago 77.9MB
goharbor/nginx-photon v2.4.3 0165c71ef734 12 months ago 44.4MB
goharbor/harbor-log v2.4.3 57ceb170dac4 12 months ago 161MB
goharbor/harbor-jobservice v2.4.3 7fea87c4b884 12 months ago 219MB
goharbor/harbor-core v2.4.3 d864774a3b8f 12 months ago 197MB
goharbor/harbor-portal v2.4.3 85f00db66862 12 months ago 53.4MB
goharbor/harbor-db v2.4.3 7693d44a2ad6 12 months ago 225MB
goharbor/prepare v2.4.3 c882d74725ee 12 months ago 268MB
查看容器
[root@master harbor]# docker ps | grep harbor
e3cd968cbc18 goharbor/nginx-photon:v2.4.3 "nginx -g 'daemon of…" 16 hours ago Up 16 hours (healthy) 0.0.0.0:81->8080/tcp, :::81->8080/tcp nginx
dbeebe444509 goharbor/harbor-jobservice:v2.4.3 "/harbor/entrypoint.…" 16 hours ago Up 16 hours (healthy) harbor-jobservice
13102f4bdabe goharbor/harbor-core:v2.4.3 "/harbor/entrypoint.…" 16 hours ago Up 16 hours (healthy) harbor-core
083bbda22496 goharbor/registry-photon:v2.4.3 "/home/harbor/entryp…" 16 hours ago Up 16 hours (healthy) registry
fcc925410e36 goharbor/harbor-db:v2.4.3 "/docker-entrypoint.…" 16 hours ago Up 16 hours (healthy) harbor-db
8257616c02ba goharbor/redis-photon:v2.4.3 "redis-server /etc/r…" 16 hours ago Up 16 hours (healthy) redis
b634f9eb780a goharbor/harbor-portal:v2.4.3 "nginx -g 'daemon of…" 16 hours ago Up 16 hours (healthy) harbor-portal
71091d4f736d goharbor/harbor-registryctl:v2.4.3 "/home/harbor/start.…" 16 hours ago Up 16 hours (healthy) registryctl
98c2dfeaaa62 goharbor/harbor-log:v2.4.3 "/bin/sh -c /usr/loc…" 16 hours ago Up 16 hours (healthy) 127.0.0.1:1514->10514/tcp harbor-log
安装sonarqube
3.1通过nfs创建两个pv
三台节点上安装nfs-client,master节点/其他存储节点上安装nfs-server端
.........
创建pv「devops-nfs」
[root@server1 sonarqube]# cat devops-nfs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: devops-nfs
namespace: kube-devops
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: "managed-nfs-storage"
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /mydata/nfs/pgSQL/DB
server: 172.31.0.16
[root@server1 sonarqube]# cat sonarqube-nfs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: sonarqube-nfs
namespace: kube-devops
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: "managed-nfs-storage"
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /mydata/nfs/pgSQL/DB
server: 172.31.0.16
创建完成后,查看两个pv的状态
[root@server1 sonarqube]# kubectl get pv -n kube-devops
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
devops-nfs 10Gi RWX Retain Bound kube-devops/postgress-data managed-nfs-storage 19m
sonarqube-nfs 5Gi RWX Retain Bound kube-devops/sonarqube-data managed-nfs-storage 17m
3.2创建sonarqube的yaml文件
[root@server1 sonarqube]# cat pgsql.yaml
apiVersion: v1
kind: PersistentVolumeClaim ##创建一个pvc
metadata:
name: postgress-data
namespace: kube-devops
spec:
accessModes:
- ReadWriteMany
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-sonar
namespace: kube-devops
spec:
replicas: 1
selector:
matchLabels:
app: postgres-sonar
template:
metadata:
labels:
app: postgres-sonar
spec:
#imagePullsecrets:
#- name: harbor-secret
containers:
- name: postgres-sonar
image: postgres:14.2
ports:
- containerPort: 5432
env: ##创建环境变量,让sonarqube通过账号和密码来连接创建的pgsql数据库
- name: POSTGRES_DB
value: "sonarDB"
- name: POSTGRES_USER
value: "sonarUser"
- name: POSTGRES_PASSWORD
value: "123456"
volumeMounts:
- name: data
mountPath: /mydata/postgressql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: postgress-data
---
apiVersion: v1
kind: Service
metadata:
name: postgres-sonar
namespace: kube-devops
spec:
type: NodePort
ports:
- name: postgres-sonar
port: 5432
targetPort: 5432
protocol: TCP
selector:
app: postgres-sonar
创建sonarqube应用
[root@server1 sonarqube]# cat sonarqube.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sonarqube-data
namespace: kube-devops
spec:
accessModes:
- ReadWriteMany
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sonarqube
namespace: kube-devops
labels:
app: sonarqube
spec:
replicas: 1
selector:
matchLabels:
app: sonarqube
template:
metadata:
labels:
app: sonarqube
spec:
#imagePullsecrets:
#- name: harbor-secret
initContainers: ##初始化容器
- name: init-sysctl
image: busybox:latest
command: ["sysctl","-w","vm.max_map_count=262144"]
securityContext:
privileged: true
containers:
- name: sonarqube
image: sonarqube:latest
ports:
- containerPort: 9000
env:
- name: SONARQUBE_JDBC_USERNAME
value: "sonarUser"
- name: SONARQUBE_JDBC_PASSWORD
value: "123456"
- name: SONARQUBE_JDBC_URL
value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"
livenessProbe:
httpGet:
path: /sessions/new
port: 9000
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
httpGet:
path: /sessions/new
port: 9000
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 6
volumeMounts:
- mountPath: /opt/sonarqube/conf
name: data
- mountPath: /opt/sonarqube/data
name: data
- mountPath: /opt/sonarqube/extensions
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: sonarqube-data
---
apiVersion: v1
kind: Service
metadata:
name: sonarqube
namespace: kube-devops
spec:
type: NodePort
ports:
- name: sonarqube
port: 9000
targetPort: 9000
protocol: TCP
selector:
app: sonarqube
创建的资源如下
service:
[root@server1 ~]# kubectl get svc -n kube-devops -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
postgres-sonar NodePort 10.100.233.40 <none> 5432:30802/TCP 38m app=postgres-sonar
sonarqube NodePort 10.96.125.70 <none> 9000:30830/TCP 35m app=sonarqube
Deployment
[root@server1 ~]# kubectl get deploy -n kube-devops -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
postgres-sonar 1/1 1 1 39m postgres-sonar postgres:14.2 app=postgres-sonar
sonarqube 1/1 1 1 36m sonarqube sonarqube:latest app=sonarqube
Pod:
[root@server1 ~]# kubectl get po -n kube-devops -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
postgres-sonar-67f955cc9f-xwds8 1/1 Running 0 39m 10.244.141.210 server3 <none> <none>
sonarqube-6f8c7c84cb-jvht4 1/1 Running 2 (28m ago) 36m 10.244.141.211 server3 <none> <none>
pv:
[root@server1 ~]# kubectl get pv -n kube-devops -o wide
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE
devops-nfs 10Gi RWX Retain Bound kube-devops/postgress-data managed-nfs-storage 38m Filesystem
sonarqube-nfs 5Gi RWX Retain Bound kube-devops/sonarqube-data managed-nfs-storage 36m Filesystem
pvc:
[root@server1 ~]# kubectl get pvc -n kube-devops -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
postgress-data Bound devops-nfs 10Gi RWX managed-nfs-storage 39m Filesystem
sonarqube-data Bound sonarqube-nfs 5Gi RWX managed-nfs-storage 36m Filesystem
安装jenkins
制作带有maven的jenkins的镜像
4.1.编写Dockerfile文件
[root@server1 jenkins]# cat Dockerfile
FROM jenkins/jenkins:2.392-jdk11
ADD ./apache-maven-3.9.4-bin.tar.gz /usr/local/
ADD ./sonar-scanner-4.8.0.2856-linux /usr/local/sonar-scanner
ENV MAVEN_HOME=/usr/local/apache-maven-3.9.4
ENV PATH=$JAVA_HOME/bin/:$MAVEN_HOME/bin/:$PATH
USER root
RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
USER jenkinsjenkins ALL=NOPASSWD: ALL
制作镜像:
[root@server1 jenkins]# docker build -t 172.31.0.16:28081/devops/jenkins-maven:v1 .
[root@server1 jenkins]# docker images | grep jenkins
172.31.0.16:28081/devops/jenkins-maven v1 6aa0a1f1fe61 21 seconds ago 611MB
jenkins/jenkins
上传到harbor镜像仓库中
[root@server1 jenkins]# docker login -uadmin 172.31.0.16:28081 ##登录到harbor镜像仓库
[root@server1 jenkins]# docker push 172.31.0.16:28081/devops/jenkins-maven:v1 ##上传镜像
部署jenkins
jenkins-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mvn-settings
namespace: kube-devops
labels:
app: jenkins-server
data:
settings.xml: |-
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">
<localRepository>/var/jenkins_home/repository</localRepository> #本地存储仓库的地址
<mirrors>
<mirror>
<id>aliyun-central</id>
<mirrorOf>*</mirrorOf>
<name>aliyun maven</name>
<url>https://maven.aliyun.com/repository/central</url>
</mirror>
<mirror>
<id>aliyun-public</id>
<name>aliyun maven</name>
<mirrorOf>*</mirrorOf>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
<mirror>
<id>aliyun-apache-snapshots</id>
<name>aliyun maven</name>
<mirrorOf>*</mirrorOf>
<url>https://maven.aliyun.com/repository/apache-snapshots</url>
</mirror>
<mirror>
<id>aliyun-google</id>
<name>aliyun maven</name>
<mirrorOf>*</mirrorOf>
<url>https://maven.aliyun.com/repository/google</url>
</mirror>
</mirrors>
<pluginGroups>
<pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
</pluginGroups>
<profiles>
<profile>
<id>sonar</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<!-- Optional URL to server. Default value is http://localhost:9000 -->
<sonar.host.url>http://sonarqube:9000</sonar.host.url>
</properties>
</profile>
</profiles>
</settings>
jenkins-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: kube-devops
spec:
replicas: 1
selector:
matchLabels:
app: jenkins-server
template:
metadata:
labels:
app: jenkins-server
spec:
serviceAccountName: jenkins-admin
imagePullSecrets:
- name: harbor-secret
containers:
- name: jenkins
image: 172.31.0.16:28081/devops/jenkins-maven:v1
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
runAsUser: 0
resources:
limits:
memory: "2Gi"
cpu: "1000m"
requests:
memory: "500Mi"
cpu: "500m"
ports:
- name: httpport
containerPort: 8080
- name: jnlpport
containerPort: 50000
livenessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 5
volumeMounts:
- name: jenkins-data
mountPath: /var/jenkins_home
- name: docker
mountPath: /run/docker.sock
- name: docker-home
mountPath: /usr/bin/docker
- name: mvn-setting
mountPath: /usr/local/apache-maven-3.9.4-bin/conf/settings.xml
subPath: settings.xml
- name: daemon
mountPath: /etc/docker/daemon.json
subPath: daemon.json
- name: kubectl
mountPath: /usr/bin/kubectl
volumes:
- name: kubectl
hostPath:
path: /usr/bin/kubectl
- name: jenkins-data
persistentVolumeClaim:
claimName: jenkins-pvc
- name: docker
hostPath:
path: /run/docker.sock
- name: docker-home
hostPath:
path: /usr/bin/docker
- name: mvn-setting
configMap:
name: mvn-settings
items:
- key: settings.xml
path: settings.xml
- name: daemon
hostPath:
path: /etc/docker/
jenkins-nfs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-nfs
namespace: kube-devops
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: managed-nfs-storage
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /mydata/nfs/jenkins/jenkinsdb
server: 172.31.0.16
jenkins-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
namespace: kube-devops
spec:
storageClassName: managed-nfs-storage
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
jenkins-serviceAccout.yaml
##创建一个名称为jenkins-admin的角色
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
namespace: kube-devops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: jenkins-admin
namespace: kube-devops
jenkins-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: jenkins-svc
namespace: kube-devops
spec:
type: NodePort
ports:
- name: jenkins-svc
port: 8080
targetPort: 8080
protocol: TCP
selector:
app: jenkins-server
资源总和
[root@server1 yaml]# kubectl get configmap -n kube-devops
NAME DATA AGE
kube-root-ca.crt 1 2d
mvn-settings 1 28m
[root@server1 yaml]# kubectl get po -n kube-devops
NAME READY STATUS RESTARTS AGE
jenkins-86995676d8-qwg5s 1/1 Running 3 (24m ago) 28m
[root@server1 yaml]# kubectl get svc -n kube-devops
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-svc NodePort 10.101.71.120 <none> 8080:31654/TCP 24m
[root@server1 yaml]# kubectl get pv -n kube-devops
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jenkins-nfs 10Gi RWX Retain Bound kube-devops/jenkins-pvc managed-nfs-storage 64m
[root@server1 yaml]# kubectl get pvc -n kube-devops
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-pvc Bound jenkins-nfs 10Gi RWX managed-nfs-storage 64m