基于k8s的综合的web服务器构建

目录

项目架构图：

项目环境:

项目描述：

项目步骤：

ip规划：

一.在三台k8s机器上安装部署好k8s，一台作为master，两台node

安装部署k8s

node节点加入集群：

master节点初始化：

 安装Calico网络插件：

二，部署nfs服务，让所有的web业务pod都取访问，通过pv，pvc和卷挂载实现

1.搭建好nfs服务器

2.设置共享目录

3.创建pv使用nfs服务器上的共享目录

测试：

三.启动nginx和MySQL的pod，采用HPA技术，cpu使用率高时进行水平扩缩,使用ab进行压力测试。

k8s部署mysql pod：

安装metrics

启动开启了HPA功能的nginx的部署控制器，启动nginx的pod

下面在nfs上用ab工具测试

四.使用ingress给web业务做基于域名的负载均衡，基于url的负载均衡的实现

第1大步骤：  安装ingress controller

第2大步骤：  创建pod和暴露pod的服务

第3大步骤： 启用ingress 关联ingress controller 和service

在nfs服务器上进行测试，需要在/etc/hosts文件里添加域名解析记录

五.使用探针对web业务pod进行监控, 一旦出现问题马上重启, 增强业务pod的可靠性。

六.构建CI/CD环境, k8smaster上安装部署Jenkins,一台机器上安装部署harbor仓库。

安装jenkins：

部署harbor：

测试harbor的上传和拉取

七，使用dashboard对整个集群资源进行掌控

八，安装部署Prometheus+grafana：

安装Prometheus

在master或者被监听的机器上操作（安装相应的exporter，常规的是node_exporter）：

安装部署Grafana和Prometheus安装在一起：

九.安装部署firewalld防火墙和jumpserver堡垒机，对web集群进行保护。

jumpserver 的安装部署

部署firewall服务器，对内网进行保护

十.安装部署ansible机器，写好主机清单,便于日后的自动化运维。

1.建立免密通道 在ansible主机上生成密钥对，上传公钥到所有服务器的root用户家目录下

 2.编写主机清单

3.测试

项目心得：

---

项目架构图：

项目环境:

centos7,k8s,docker,prometheus,nfs,JumpServer,harbor,ansible,Jenkins等

项目描述：

模拟企业里的k8s生产环境,,构建一个高可用高性能的系统,

项目步骤：

ip规划：

k8smaster：192.168.220.100
k8snode1：192.168.220.101
k8snode2：192.168.220.102
nfs：192.168.220.103
harbor：192.168.220.104
Prometheus：192.168.220.105
jumpserver：192.168.220.106
firewalld： 192.168.220.107
ansible：192.168.220.108

一.在三台k8s机器上安装部署好k8s，一台作为master，两台node

安装部署k8s

###下面操作每台机器都要操作，建议xshell上开启输入到所有会话
1.配置静态ip地址和设置主机名和关闭selinux和firewalld

hostnamectl set-hostname master && bash
hostnamectl set-hostname node1 && bash
hostnamectl set-hostname node2 && bash

#关闭firewalld防火墙服务，并且设置开机不要启动

service firewalld stop
systemctl  disable  firewalld
 

#临时关闭selinux

setenforce 0

#永久关闭selinux

sed -i '/^SELINUX=/ s/enforcing/disabled/'  /etc/selinux/config

添加域名解析

[root@master ~]# vim /etc/hosts
[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.220.100 master
192.168.220.101 node1
192.168.220.102 node2

  #####  注意！！！下面为三台机器都要操作；
关闭交换分区
k8s设计的时候为了能提升性能，默认是不允许使用交换分区的。

[root@master ~]# swapoff -a 临时关闭
 sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@master ~]# vim /etc/fstab
加注释：
#/dev/mapper/centos-swap swap      swap    defaults        0 0    

加载网桥和地址地址转发功能（实现容器之间通信）：

[root@master ~]# modprobe br_netfilter
[root@master ~]# modprobe overlay

#如果文件不存在，tee 会创建它；如果文件已经存在，tee 会覆盖它的内容

[root@master ~]#  cat << EOF | tee /etc/modules-load.d/k8s.conf 
br_netfilter
overlay
EOF

查看内核模板是否成功
#lsmod是Linux系统中用于列出已加载内核模块（Kernel Modules）的命令

[root@master ~]#  lsmod |grep -e br_netfilter -e overlay
overlay                91659  0
br_netfilter           22256  0
bridge                151336  1 br_netfilter

[root@master ~]#  cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

 更新和配置软件源
# 添加阿里云yum源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

# 重新生成yum元数据缓存
yum clean all && yum makecache
# 安装基础软件包
yum install -y vim wget
 
# 配置阿里云Docker yum仓库源
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

配置ipvs功能：（实现负载均衡的软件）：
# 安装ipset和ipvsadm
yum install -y ipset ipvsadm

# 添加需要加载的模块写入脚本文件，保证在节点重启后能自动加载所需模块
cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

# 为脚本文件添加执行权限
chmod +x /etc/sysconfig/modules/ipvs.modules

# 执行脚本文件
/bin/bash /etc/sysconfig/modules/ipvs.modules

# 查看对应的模块是否加载成功
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4      15053  24
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0
ip_vs_wrr              12697  0
ip_vs_rr               12600  105
ip_vs                 145497  111 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          139264  10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack

 配置时间同步：

# 启用chronyd服务
systemctl start chronyd && systemctl enable chronyd

# 设置时区
timedatectl set-timezone Asia/Shanghai

docker的配置


yum install -y docker-ce-20.10.24-3.el7 docker-ce-cli-20.10.24-3.el7 containerd.io
[root@master ~]# systemctl start docker
[root@master ~]# systemctl enable docker

# 创建文件夹
mkdir -p /etc/docker
# 编辑配置
cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": [
    "https://youraddr.mirror.aliyuncs.com",
    "http://hub-mirror.c.163.com",
    "https://reg-mirror.qiniu.com",
    "https://docker.mirrors.ustc.edu.cn"
  ],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

systemctl daemon-reload && systemctl restart docker
[root@master ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@master ~]# getenforce
Disabled
[root@master ~]# swap
-bash: swap: 未找到命令
[root@master ~]# cat /proc/s
sched_debug    self/          stat           sysrq-trigger
schedstat      slabinfo       swaps          sysvipc/
scsi/          softirqs       sys/
[root@master ~]# cat /proc/swaps
Filename                Type        Size  Used    Priority

node节点加入集群：

下面步骤三台机器都要做

配置k8s集群环境：

# 配置组件源
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 构建本地yum缓存
yum makecache

# 安装
yum install -y kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0 --disableexcludes=kubernetes

[root@master ~]# cat <<EOF > /etc/sysconfig/kubelet

KUBELET_CGROUP_ARGS="--cgroup-driver=systemd"
KUBE_PROXY_MODE="ipvs"
EOF


# 启动并设置开机自启
systemctl enable --now kubelet

master节点初始化：

#只需要master上面做
kubeadm init \
    --kubernetes-version=v1.23.17 \
    --pod-network-cidr=10.224.0.0/16 \
    --service-cidr=10.96.0.0/12 \
    --apiserver-advertise-address=192.168.220.100 \
    --image-repository=registry.aliyuncs.com/google_containers
（不能有空格）
结果：
kubeadm join 192.168.220.100:6443 --token v7a9n5.ppfursc2nbica1fg \
    --discovery-token-ca-cert-hash sha256:6a4863a28201e03ee1b8083fc6fc08b6f7f39b44899b9c8bd6b627ab044b77ea 

journalctl -u kubelet 看错误

在node机器上：
kubeadm join 192.168.220.100:6443 --token v7a9n5.ppfursc2nbica1fg  --discovery-token-ca-cert-hash sha256:6a4863a28201e03ee1b8083fc6fc08b6f7f39b44899b9c8bd6b627ab044b77ea 

master：
[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES                  AGE   VERSION
master   NotReady   control-plane,master   22m   v1.23.17
node1    NotReady   <none>                 50s   v1.23.17
node2    NotReady   <none>                 44s   v1.23.17

给node节点打上标签
# 在master上执行
kubectl label node node1 node-role.kubernetes.io/worker=worker
kubectl label node node2 node-role.kubernetes.io/worker=worker
[root@master ~]# kubectl  get nodes
NAME     STATUS     ROLES                  AGE     VERSION
master   NotReady   control-plane,master   28m     v1.23.17
node1    NotReady   worker                 6m3s    v1.23.17
node2    NotReady   worker                 5m57s   v1.23.17

kubeadm  reset 可以清除初始化的操作

 安装Calico网络插件：

# master执行

wget --no-check-certificate https://docs.projectcalico.org/archive/v3.25/manifests/calico.yaml   #用的wget下载
calico 是k8s里的一个网络组件，用于实现master和node节点之间的通信的，底层使用overlay网络

# master执行
kubectl apply -f https://docs.projectcalico.org/archive/v3.25/manifests/calico.yaml  # k8s 1.23适用此版本
[root@master ~]# kubectl  get node
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   53m   v1.23.17
node1    Ready    worker                 30m   v1.23.17
node2    Ready    worker                 30m   v1.23.17

k8s配置ipvs：
master上：
kubectl edit configmap kube-proxy -n kube-system
mode: "ipvs"
# 删除所有kube-proxy pod使之重启
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
看k8s容器（pod）：
kubectl  get pod -n kube-system

kubectl get pod 是查看有哪些pod在运行  --》docker ps
-n kube-system   是查看kube-system命名空间里运行的pod   namespace
kube-system  是k8s控制平面的pod所在的命名空间

pod 是运行容器的单元
以pod治理pod
住在kube-system  命名空间里的pod是控制平面的pod

kubectl get ns  = kubectl get namespace  查看有哪些命名空间
NAME              STATUS   AGE
default           Active   78m           #是创建的普通的pod运行的命名空间
kube-node-lease   Active   78m
kube-public       Active   78m
kube-system       Active   78m    #是管理相关的命名空间

二，部署nfs服务，让所有的web业务pod都取访问，通过pv，pvc和卷挂载实现

注意点：
nfs服务器上：
#关闭firewalld防火墙服务，并且设置开机不要启动
service firewalld stop
systemctl  disable  firewalld
 
#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/'  /etc/selinux/config

1.搭建好nfs服务器

# 在nfs服务器和k8s集群上安装nfs
[root@nfs ~]# yum install nfs-utils -y
[root@master ~]# yum install nfs-utils -y
[root@node1 ~]# yum install nfs-utils -y
[root@node2 ~]# yum install nfs-utils -y

2.设置共享目录

[root@nfs ~]# vim /etc/exports
[root@nfs ~]# cat /etc/exports
/web   192.168.220.0/24(rw,no_root_squash,sync)
[root@nfs ~]# mkdir /web
[root@nfs ~]# cd /web
[root@nfs web]# echo "tiantianming" >index.html
[root@nfs web]# ls
index.html
[root@localhost web]# exportfs -rv        #刷新nfs服务
exporting 192.168.220.0/24:/web

#重启服务并且设置开机启动
[root@nfs web]# systemctl restart nfs && systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

3.创建pv使用nfs服务器上的共享目录

[root@master ~]# mkdir /pv
[root@master ~]# cd /pv/
[root@master pv]# vim  nfs-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-web
  labels:
    type: pv-web
spec:
  capacity:
    storage: 10Gi 
  accessModes:
    - ReadWriteMany
  storageClassName: nfs         # pv对应的名字
  nfs:
    path: "/web"       # nfs共享的目录
    server: 192.168.220.103   # nfs服务器的ip地址
    readOnly: false   # 访问模式
[root@master pv]#  kubectl apply -f nfs-pv.yml
persistentvolume/pv-web created
[root@master pv]# kubectl get pv
NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-web   10Gi       RWX            Retain           Available           nfs                     12s

# 创建pvc使用pv
[root@master pv]# vim nfs-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-web
spec:
  accessModes:
  - ReadWriteMany      
  resources:
     requests:
       storage: 1Gi
  storageClassName: nfs #使用nfs类型的pv
[root@master pv]# kubectl apply -f nfs-pvc.yml
persistentvolumeclaim/pvc-web created
[root@master pv]# kubectl get pvc
NAME      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-web   Bound    pv-web   10Gi       RWX            nfs            13s
#创建pod使用pvc
[root@master pv]# vim nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: sc-pv-storage-nfs
          persistentVolumeClaim:
            claimName: pvc-web
      containers:
        - name: sc-pv-container-nfs
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              name: "http-server"
          volumeMounts:
            - mountPath: "/usr/share/nginx/html"
              name: sc-pv-storage-nfs
[root@master pv]# kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created

[root@master pv]# kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
nginx-deployment-794d8c5666-dsxkq   1/1     Running   0          17m   10.224.166.130   node1   <none>           <none>
nginx-deployment-794d8c5666-fsctm   1/1     Running   0          12m   10.224.104.4     node2   <none>           <none>
nginx-deployment-794d8c5666-spkzs   1/1     Running   0          12m   10.224.104.3     node2   <none>           <none>

测试：

[root@master pv]# curl 10.224.166.130
tiantianming

修改下nfs服务器上的index.html的内容
[root@nfs web]# vim index.html 
[root@nfs web]# cat index.html 
tiantianming

welcome to hangzhou!!!
访问也变了，表示已经成功！
[root@master pv]# curl 10.224.166.130
tiantianming

welcome to hangzhou!!!

三.启动nginx和MySQL的pod，采用HPA技术，cpu使用率高时进行水平扩缩,使用ab进行压力测试。

k8s部署mysql pod：

1.编写yaml文件，包括了deployment、service
[root@master ~]# mkdir /mysql
[root@master ~]# cd /mysql/
[root@master mysql]# vim mysql.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
    labels:
        app: mysql
    name: mysql
spec:
    replicas: 1
    selector:
        matchLabels:
            app: mysql
    template:
        metadata:
            labels: 
                app: mysql
        spec:
            containers:
            - image: mysql:latest
              name: mysql
              imagePullPolicy: IfNotPresent
              env:
              - name: MYSQL_ROOT_PASSWORD
                value: "123456"  #mysql的密码
              ports:
              - containerPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: svc-mysql
  name: svc-mysql
spec:
  selector:
    app: mysql
  type: NodePort
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
    nodePort: 30007

2.部署
[root@master mysql]# kubectl apply -f mysql.yaml 
deployment.apps/mysql created
service/svc-mysql created
[root@master mysql]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          23h
php-apache   ClusterIP   10.96.134.145   <none>        80/TCP           21h
svc-mysql    NodePort    10.109.190.20   <none>        3306:30007/TCP   9s
[root@master mysql]# kubectl  get pod
NAME                                READY   STATUS              RESTARTS      AGE
mysql-597ff9595d-tzqzl              0/1     ContainerCreating   0             27s
nginx-deployment-794d8c5666-dsxkq   1/1     Running             1 (15m ago)   22h
nginx-deployment-794d8c5666-fsctm   1/1     Running             1 (15m ago)   22h
nginx-deployment-794d8c5666-spkzs   1/1     Running             1 (15m ago)   22h
php-apache-7b9f758896-2q44p         1/1     Running             1 (15m ago)   21h

[root@master mysql]# kubectl exec -it mysql-597ff9595d-tzqzl    -- bash
root@mysql-597ff9595d-tzqzl:/# mysql -uroot -p123456    #容器内部进入mysql

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

安装metrics

下载配置文件

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml 

     args:
#        // 新增下面两行参数
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
# 替换image
        image: registry.aliyuncs.com/google_containers/metrics-server:v0.6.0
        imagePullPolicy: IfNotPresent
  

部署:
kubectl apply -f components.yaml

[root@master ~]# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6949477b58-tbkl8   1/1     Running   1          7h10m
calico-node-4t8kx                          1/1     Running   1          7h10m
calico-node-6lbdw                          1/1     Running   1          7h10m
calico-node-p6ghl                          1/1     Running   1          7h10m
coredns-7f89b7bc75-dxc9v                   1/1     Running   1          7h15m
coredns-7f89b7bc75-kw7ph                   1/1     Running   1          7h15m
etcd-master                                1/1     Running   1          7h15m
kube-apiserver-master                      1/1     Running   2          7h15m
kube-controller-manager-master             1/1     Running   1          7h15m
kube-proxy-87ptg                           1/1     Running   1          7h15m
kube-proxy-8gbsd                           1/1     Running   1          7h15m
kube-proxy-x4fbj                           1/1     Running   1          7h15m
kube-scheduler-master                      1/1     Running   1          7h15m
metrics-server-7787b94d94-jt9sc            1/1     Running   0          47s

[root@master hpa]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master   129m         6%     1111Mi          64%       
node1    61m          3%     608Mi           35%       
node2    59m          2%     689Mi           40%       
[root@master hpa]# 

启动开启了HPA功能的nginx的部署控制器，启动nginx的pod

[root@master hpa]# vim nginx-hpa.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ab-nginx
spec:
  selector:
    matchLabels:
      run: ab-nginx
  template:
    metadata:
      labels:
        run: ab-nginx
    spec:
      #nodeName: node-2  取消指定
      containers:
      - name: ab-nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 100m
          requests:
            cpu: 50m
---
apiVersion: v1
kind: Service
metadata:
  name: ab-nginx-svc
  labels:
    run: ab-nginx-svc
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 31000
  selector:
    run: ab-nginx
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ab-nginx
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ab-nginx
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50
[root@master hpa]# 
创建具有hpa功能的nginx的pod
[root@master hpa]# kubectl apply -f nginx-hpa.yaml 
deployment.apps/ab-nginx created
service/ab-nginx-svc created
horizontalpodautoscaler.autoscaling/ab-nginx created
[root@master hpa]# 

查看启动hpa和pod、deployment、service的情况
[root@master hpa]# kubectl get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
ab-nginx   1/1     1            1           55s
[root@master hpa]# kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
ab-nginx     Deployment/ab-nginx     0%/50%    1         10        1          58s
php-apache   Deployment/php-apache   0%/50%    1         10        1          20d
[root@master hpa]# kubectl get pod
NAME                        READY   STATUS    RESTARTS   AGE
ab-nginx-5f4c4b9558-xbxjb   1/1     Running   0          63s
configmap-demo-pod          1/1     Running   31         2d23h
[root@master hpa]# 

[root@master hpa]# kubectl get svc
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
ab-nginx-svc        NodePort    10.107.155.209   <none>        80:31000/TCP     2m26s


访问宿主机的31000端口

http://192.168.220.100:31000/

测试nginx pod是否启动成功

下面在nfs上用ab工具测试

安装http-tools工具得到ab软件
[root@nfs-server ~]# yum install httpd-tools -y

模拟访问
[root@nfs-server ~]# ab  -n 1000  -c50  http://192.168.220.100:31000/index.html

root@master hpa]# kubectl get hpa --watch

增加并发数和请求总数

[root@nfs-server ~]# ab  -n 5000  -c100  http://192.168.203.128:31000/index.html
[root@nfs-server ~]# ab  -n 10000  -c200  http://192.168.203.128:31000/index.html
[root@nfs-server ~]# ab  -n 20000  -c400  http://192.168.203.128:31000/index.html

[root@master hpa]# kubectl describe pod ab-nginx-5f4c4b9558-shtt5
Warning  OutOfmemory  98s   kubelet  Node didn't have enough resource: memory, requested: 268435456, used: 3584032768, capacity: 3848888320
[root@master hpa]# 
原因是node-2节点没有足够的内存去启动新的pod了

四.使用ingress给web业务做基于域名的负载均衡，基于url的负载均衡的实现

第1大步骤：  安装ingress controller

使用旧版本ingress controller v1.1完成
    准备工作：需要提前上传下面的这些镜像和yaml文件到k8s集群里的linux系统里，建议存放到master节点上，然后再scp到node节点上
[root@master .kube]# mkdir /ingress
[root@master .kube]# cd /ingress/
[root@master ingress]# 

ingress-controller-deploy.yaml   是部署ingress controller使用的yaml文件
ingress-nginx-controllerv1.1.0.tar.gz    ingress-nginx-controller镜像
kube-webhook-certgen-v1.1.0.tar.gz       kube-webhook-certgen镜像
#kube-webhook-certgen镜像主要用于生成Kubernetes集群中用于Webhook的证书。
#kube-webhook-certgen镜像生成的证书，可以确保Webhook服务在Kubernetes集群中的安全通信和身份验证

ingress.yaml 创建ingress的配置文件
nginx-svc-3.yaml    创建service3 和相关pod
nginx-svc-4.yaml    创建service4 和相关pod


1.将镜像scp到所有的node节点服务器上
[root@master ingress]# scp ingress-nginx-controllerv1.1.0.tar.gz node1:/root
ingress-nginx-controllerv1.1.0.tar.gz                                               100%  276MB  42.7MB/s   00:06    
[root@master ingress]# scp ingress-nginx-controllerv1.1.0.tar.gz node2:/root
ingress-nginx-controllerv1.1.0.tar.gz                                               100%  276MB  45.7MB/s   00:06    
[root@master ingress]# scp kube-webhook-certgen-v1.1.0.tar.gz node2:/root
kube-webhook-certgen-v1.1.0.tar.gz                                                  100%   47MB  40.5MB/s   00:01    
[root@master ingress]# scp kube-webhook-certgen-v1.1.0.tar.gz node1:/root
kube-webhook-certgen-v1.1.0.tar.gz                                                  100%   47MB  47.1MB/s   00:00    
[root@master ingress]# 
2.导入镜像，在所有的节点服务器（node-1和node-2）上进行
[root@k8smaster ingress]# docker load -i ingress-nginx-controllerv1.1.0.tar.gz 
[root@k8smaster ingress]# docker load -i kube-webhook-certgen-v1.1.0.tar.gz
[root@k8snode2 ~]# docker images
REPOSITORY                                                                     TAG        IMAGE ID       CREATED         SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller   v1.1.0     ae1a7201ec95   16 months ago   285MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen       v1.1.1     c41e9fcadf5a   17 months ago   47.7MB
[root@k8smaster new]# 
3.使用ingress-controller-deploy.yaml  文件去启动ingress  controller
[root@k8smaster 4-4]# kubectl apply -f ingress-controller-deploy.yaml 
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
[root@k8smaster 4-4]#
查看ingress controller的相关命名空间
[root@k8smaster 4-4]# kubectl get ns
NAME              STATUS   AGE
default           Active   11d
ingress-nginx     Active   52s
kube-node-lease   Active   11d
kube-public       Active   11d
kube-system       Active   11d

[root@k8smaster 4-4]# 
查看ingress controller的相关service
[root@k8smaster 4-4]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.99.160.10   <none>        80:30092/TCP,443:30263/TCP   91s
ingress-nginx-controller-admission   ClusterIP   10.99.138.23   <none>        443/TCP                      91s
[root@k8smaster 4-4]# 
查看ingress controller的相关pod
[root@k8smaster 4-4]# kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-k69t2        0/1     Completed   0          119s
ingress-nginx-admission-patch-zsrk8         0/1     Completed   1          119s
ingress-nginx-controller-6c8ffbbfcf-bt94p   1/1     Running     0          119s
ingress-nginx-controller-6c8ffbbfcf-d49kx   1/1     Running     0          119s
[root@k8smaster 4-4]# 

第2大步骤：  创建pod和暴露pod的服务

[root@master url]# cat sc-nginx-svc-3.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sc-nginx-deploy-3
  labels:
    app: sc-nginx-feng-3
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sc-nginx-feng-3
  template:
    metadata:
      labels:
        app: sc-nginx-feng-3
    spec:
      containers:
      - name: sc-nginx-feng-3
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name:  sc-nginx-svc-3
  labels:
    app: sc-nginx-svc-3
spec:
  selector:
    app: sc-nginx-feng-3
  ports:
  - name: name-of-service-port
    protocol: TCP
    port: 80
    targetPort: 80
[root@master url]# 

[root@master url]# cat sc-nginx-svc-4.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sc-nginx-deploy-4
  labels:
    app: sc-nginx-feng-4
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sc-nginx-feng-4
  template:
    metadata:
      labels:
        app: sc-nginx-feng-4
    spec:
      containers:
      - name: sc-nginx-feng-4
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name:  sc-nginx-svc-4
  labels:
    app: sc-nginx-svc-4
spec:
  selector:
    app: sc-nginx-feng-4
  ports:
  - name: name-of-service-port
    protocol: TCP
    port: 80
    targetPort: 80
[root@master url]# 

[root@master lb-url]# kubectl apply -f sc-nginx-svc-3.yaml 
deployment.apps/sc-nginx-deploy-3 created
service/sc-nginx-svc-3 created
[root@master lb-url]# kubectl apply -f sc-nginx-svc-4.yaml 
deployment.apps/sc-nginx-deploy-4 created
service/sc-nginx-svc-4 created

第3大步骤： 启用ingress 关联ingress controller 和service

[root@master url]# cat ingress-url.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-url-lb-example
  annotations:
    kubernets.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: www.guan.com
    http:
      paths:
      - path: /tian1
        pathType: Prefix
        backend:
          service:
            name: sc-nginx-svc-3  #必须与上面的service名字相同
            port:
              number: 80
      - path: /tian2
        pathType: Prefix
        backend:
          service:
            name: sc-nginx-svc-4
            port:
              number: 80

[root@master ingress]# kubectl apply -f sc-ingress-url.yaml 
ingress.networking.k8s.io/simple-fanout-example created

[root@master ingress]# kubectl get ingress
NAME                    CLASS   HOSTS          ADDRESS                           PORTS   AGE
simple-fanout-example   nginx   www.guan.com   192.168.220.101,192.168.220.102   80      29s

在nfs服务器上进行测试，需要在/etc/hosts文件里添加域名解析记录

[root@nfs-server ~]#  cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.220.101 www.guan.com #两个node节点上都添加
192.168.220.102 www.guan.com 


测试发现不能找到页面，到底是ingress controller的问题还是我们后端的pod的问题

进入service4，service3 对应的一个pod里，新建tian1和tian2文件夹以及index.html网页文件
[root@master ingress]# kubectl exec -it  sc-nginx-deploy-3-5c4b975ffc-6x4kc  -- bash
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/# cd /usr/share/nginx/html/
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# ls
50x.html  index.html
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# mkdir tian1
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# echo "tiantianming" > tian1/index.html
root@sc-nginx-deploy-3-5c4b975ffc-6x4kc:/usr/share/nginx/html# ls
50x.html  index.html  tian1

service4对应的一样创建文件夹tian2
[root@master ingress]# kubectl exec -it  sc-nginx-deploy-4-7d4b5c487f-2sdvf   -- bash
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/# cd /usr/share/nginx/html/
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# ls
50x.html  index.html
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# mkdir tian2
root@sc-nginx-deploy-4-7d4b5c487f-2sdvf:/usr/share/nginx/html# echo "tiantianming2222" > tian2/index.html 

再次在nfs服务器上测试，多测试几次，因为service 背后的ipvs的调度算法是轮询的，所以建议每个pod都建立对应的文件夹
 curl  www.guan.com/tian1/index.html
 curl  www.guan.com/tian2/index.html


效果如下：
[root@nfs ~]#  curl  www.guan.com/tian1/index.html
tiantianming
[root@nfs ~]#  curl  www.guan.com/tian2/index.html
tiantianming2222

五.使用探针对web业务pod进行监控, 一旦出现问题马上重启, 增强业务pod的可靠性。

        livenessProbe:
          exec:
            command:
            - ls
            - /tmp
          initialDelaySeconds: 5
          periodSeconds: 5
 
        readinessProbe:
          exec:
            command:
            - ls
            - /tmp
          initialDelaySeconds: 5
          periodSeconds: 5 
 
        startupProbe:
          httpGet:
            path: /
            port: 8000
          failureThreshold: 30
          periodSeconds: 10
 
[root@k8smaster probe]# vim my-web.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myweb
  name: myweb
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
      - name: myweb
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: 300m
          requests:
            cpu: 100m
        livenessProbe:
          exec:
            command:
            - ls
            - /tmp
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          exec:
            command:
            - ls
            - /tmp
          initialDelaySeconds: 5
          periodSeconds: 5   
        startupProbe:
          httpGet:
            path: /
            port: 8000
          failureThreshold: 30
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: myweb-svc
  name: myweb-svc
spec:
  selector:
    app: myweb
  type: NodePort
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
    nodePort: 30001
 
[root@k8smaster probe]# kubectl apply -f my-web.yaml 
deployment.apps/myweb created
service/myweb-svc created
 
[root@master probe]# kubectl get pod |grep -i  myweb
myweb-7df8f89d75-2c9v6               0/1     Running   0              69s
myweb-7df8f89d75-cf82r               0/1     Running   0              69s
myweb-7df8f89d75-fmbpn               0/1     Running   0              69s

 
[root@k8smaster probe]# kubectl describe pod myweb-6b89fb9c7b-4cdh9
。。。
   Liveness:     exec [ls /tmp] delay=5s timeout=1s period=5s #success=1 #failure=3
    Readiness:    exec [ls /tmp] delay=5s timeout=1s period=5s #success=1 #failure=3
    Startup:      http-get http://:8000/ delay=0s timeout=1s period=10s #success=1 #failure=30
。。。

六.构建CI/CD环境, k8smaster上安装部署Jenkins,一台机器上安装部署harbor仓库。

安装jenkins：

# Jenkins部署到k8s里
# 1.安装git软件
[root@k8smaster jenkins]# yum install git -y
 
# 2.下载相关的yaml文件
[root@k8smaster jenkins]# git clone https://github.com/scriptcamp/kubernetes-jenkins

[root@k8smaster jenkins]# ls
kubernetes-jenkins
[root@k8smaster jenkins]# cd kubernetes-jenkins/
[root@k8smaster kubernetes-jenkins]# ls
deployment.yaml  namespace.yaml  README.md  serviceAccount.yaml  service.yaml  volume.yaml
 
# 3.创建命名空间
[root@k8smaster kubernetes-jenkins]# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: devops-tools
[root@k8smaster kubernetes-jenkins]# kubectl apply -f namespace.yaml 
namespace/devops-tools created
 
[root@k8smaster kubernetes-jenkins]# kubectl get ns
NAME                   STATUS   AGE
default                Active   22h
devops-tools           Active   19s
ingress-nginx          Active   139m
kube-node-lease        Active   22h
kube-public            Active   22h
kube-system            Active   22h
 
# 4.创建服务账号，集群角色，绑定
[root@k8smaster kubernetes-jenkins]# cat serviceAccount.yaml 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-admin
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops-tools
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
 
[root@k8smaster kubernetes-jenkins]# kubectl apply -f serviceAccount.yaml 
clusterrole.rbac.authorization.k8s.io/jenkins-admin created
serviceaccount/jenkins-admin created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-admin created
 
# 5.创建卷，用来存放数据
[root@k8smaster kubernetes-jenkins]# cat volume.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv-volume
  labels:
    type: local
spec:
  storageClassName: local-storage
  claimRef:
    name: jenkins-pv-claim
    namespace: devops-tools
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: /mnt
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1   # 需要修改为k8s里的node节点的名字
 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
  namespace: devops-tools
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
 
[root@k8smaster kubernetes-jenkins]# kubectl apply -f volume.yaml 
storageclass.storage.k8s.io/local-storage created
persistentvolume/jenkins-pv-volume created
persistentvolumeclaim/jenkins-pv-claim created
 
[root@k8smaster kubernetes-jenkins]# kubectl get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS    REASON   AGE
jenkins-pv-volume   10Gi       RWO            Retain           Bound    devops-tools/jenkins-pv-claim   local-storage            33s
pv-web              10Gi       RWX            Retain           Bound    default/pvc-web                 nfs                      21h
 
[root@k8smaster kubernetes-jenkins]# kubectl describe pv jenkins-pv-volume
Name:              jenkins-pv-volume
Labels:            type=local
Annotations:       <none>
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-storage
Status:            Bound
Claim:             devops-tools/jenkins-pv-claim
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:     
  Required Terms:  
    Term 0:        kubernetes.io/hostname in [k8snode1]
Message:           
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /mnt
Events:    <none>
 
# 6.部署Jenkins
[root@k8smaster kubernetes-jenkins]# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
            fsGroup: 1000 
            runAsUser: 1000
      serviceAccountName: jenkins-admin
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home         
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pv-claim
 
[root@k8smaster kubernetes-jenkins]# kubectl apply -f deployment.yaml 
deployment.apps/jenkins created
 
[root@k8smaster kubernetes-jenkins]# kubectl get deploy -n devops-tools
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
jenkins   1/1     1            1           5m36s
 
[root@k8smaster kubernetes-jenkins]# kubectl get pod -n devops-tools
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-7fdc8dd5fd-bg66q   1/1     Running   0          19s
 
# 7.启动服务发布Jenkins的pod
[root@k8smaster kubernetes-jenkins]# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: devops-tools
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector: 
    app: jenkins-server
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 32000
 
[root@k8smaster kubernetes-jenkins]# kubectl apply -f service.yaml 
service/jenkins-service created
 
[root@k8smaster kubernetes-jenkins]# kubectl get svc -n devops-tools
NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
jenkins-service   NodePort   10.104.76.252   <none>        8080:32000/TCP   24s
 
# 8.在Windows机器上访问Jenkins，宿主机ip+端口号
http://192.168.220.100:32000/
 
# 9.进入pod里获取登录的密码
[root@master kubernetes-jenkins]# kubectl exec -it jenkins-b96f7764f-qkzvd -n devops-tools -- bash
jenkins@jenkins-b96f7764f-qkzvd:/$ cat /var/jenkins_home/secrets/initialAdminPassword
557fc27bdf4149bb824b3c6e21a7f823

# 修改密码

部署harbor：

# 前提是安装好 docker 和 docker compose
# 1.配置阿里云的repo源
yum install -y yum-utils
 
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
 
# 2.安装docker服务
yum install docker-ce-20.10.6 -y
 
# 启动docker，设置开机自启
systemctl start docker && systemctl enable docker.service
 
# 3.查看docker版本，docker compose版本
[root@harbor ~]# docker version
[root@harbor ~]# docker compose version


# 5.安装 harbor，到 harbor 官网或者 github 下载harbor源码包，上传到本地。

wget https://github.com/goharbor/harbor/releases/download/v2.8.3/harbor-offline-installer-v2.8.3.tgz

[root@localhost ~]# ls
anaconda-ks.cfg  harbor-offline-installer-v2.8.3.tgz

 
# 6.解压
[root@localhost ~]# tar xf harbor-offline-installer-v2.8.3.tgz
[root@harbor ~]# ls
anaconda-ks.cfg  harbor  harbor-offline-installer-v2.8.3.tgz
[root@harbor ~]# cd harbor
[root@harbor harbor]# ls
common.sh             harbor.yml.tmpl  LICENSE
harbor.v2.8.3.tar.gz  install.sh       prepare

 
# 7.修改配置文件
[root@harbor harbor]# vim harbor.yml.tmpl 

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.220.104 #修改为主机ip地址

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 5001 #可以进行修改端口

# https related config
#https:
  # https port for harbor, default is 443
 # port: 443
  # The path of cert and key files for nginx
  #certificate: /your/certificate/path
  #private_key: /your/private/key/path

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

 
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433
 
# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345  #登录密码
 
# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345 #登录密码，可以修改

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123  #这是 Harbor 数据库的 root 用户的密码
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 100 #这是空闲连接池中的最大连接数。
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 900 #这是到数据库的最大打开连接数。
 
# 8.执行部署脚本
[root@harbor harbor]# ./install.sh
看harbor相关容器运行情况：
 [root@harbor harbor]# docker compose ps |grep harbor
harbor-core         goharbor/harbor-core:v2.8.3          "/harbor/entrypoint.…"   core          About a minute ago   Up About a minute (healthy)   
harbor-db           goharbor/harbor-db:v2.8.3            "/docker-entrypoint.…"   postgresql    About a minute ago   Up About a minute (healthy)   
harbor-jobservice   goharbor/harbor-jobservice:v2.8.3    "/harbor/entrypoint.…"   jobservice    About a minute ago   Up About a minute (healthy)   
harbor-log          goharbor/harbor-log:v2.8.3           "/bin/sh -c /usr/loc…"   log           About a minute ago   Up About a minute (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       goharbor/harbor-portal:v2.8.3        "nginx -g 'daemon of…"   portal        About a minute ago   Up About a minute (healthy)   
nginx               goharbor/nginx-photon:v2.8.3         "nginx -g 'daemon of…"   proxy         About a minute ago   Up About a minute (healthy)   0.0.0.0:5001->8080/tcp, :::5001->8080/tcp
redis               goharbor/redis-photon:v2.8.3         "redis-server /etc/r…"   redis         About a minute ago   Up About a minute (healthy)   
registry            goharbor/registry-photon:v2.8.3      "/home/harbor/entryp…"   registry      About a minute ago   Up About a minute (healthy)   
registryctl         goharbor/harbor-registryctl:v2.8.3   "/home/harbor/start.…"   registryctl   About a minute ago   Up About a minute (healthy)   


# 9.测试登录
http://192.168.220.104:5001/

问题：prepare base dir is set to /root/harbor
no config file: /root/harbor/harbor.yml
解决方法：
[root@harbor harbor]# mv harbor.yml.tmpl  harbor.yml

 
# 账号：admin
# 密码：Harbor12345
登录后：
在harbor里创建一个项目k8s-harbor
并且新建一个用户 guan  密码是Gxx123456
授权k8s-harbor这个项目允许guan这个用户去访问，授予项目管理员权限 

#10.实现pod集群都用这个harbor仓库；
master机器：
[root@master ~]# vim /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://ruk1gp3w.mirror.aliyuncs.com"],
  "insecure-registries" : ["192.168.220.104:5001"] 
}
然后重启docker
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart docker

[root@master ~]# journalctl -xe
出现 "3月 14 14:49:02 master systemd[1]: Failed to start Docker Application Container Engine.
配置文件不标准。

测试harbor的上传和拉取

在原来安装harbor的宿主机上，重新启动harbor相关的容器
[root@harbor harbor]# cd /root
[root@harbor ~]# cd harbor
[root@harbor harbor]# docker compose up -d

pod集群机器上拉取一个镜像或者dockerfile制作一个镜像，修改镜像的名字
[root@master ~]# docker tag nginx:latest  192.168.220.104:5001/k8s-harbor/nginx:latest
[root@master ~]# docker images
192.168.220.104:5001/k8s-harbor/nginx                             latest     605c77e624dd   2 years ago     141MB
。。。）

本机上传
首先登陆私有仓库

登录使用guan这个用户，密码是Gxx123456
[root@master ~]# docker login 192.168.220.104:5001
Username: guan
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

上传到仓库
[root@master ~]# docker push 192.168.220.104:5001/k8s-harbor/nginx:latest
在浏览器检测已经收到

在nfs机器上拉取pod机器上传的镜像：
# 1.配置阿里云的repo源
yum install -y yum-utils
 
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
 
# 2.安装docker服务
yum install docker-ce-20.10.6 -y
 
# 启动docker，设置开机自启
systemctl start docker && systemctl enable docker.service

root@nfs ~]# vim /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://ruk1gp3w.mirror.aliyuncs.com"],
  "insecure-registries" : ["192.168.203.128:80"] 
}
然后重启docker
[root@nfs ~]# systemctl daemon-reload
[root@nfs ~]# systemctl restart docker

[root@nfs ~]# docker login 192.168.220.104:5001
Username: guan
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
拉取成功：
[root@nfs ~]#  docker pull  192.168.220.104:5001/k8s-harbor/nginx:latest
latest: Pulling from k8s-harbor/nginx
a2abf6c4d29d: Pull complete 
a9edb18cadd1: Pull complete 
589b7251471a: Pull complete 
186b1aaa4aa6: Pull complete 
b4df32aa5a72: Pull complete 
a0bcbecc962e: Pull complete 
Digest: sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3
Status: Downloaded newer image for 192.168.220.104:5001/k8s-harbor/nginx:latest
192.168.220.104:5001/k8s-harbor/nginx:latest
[root@nfs ~]# docker images
REPOSITORY                              TAG       IMAGE ID       CREATED       SIZE
192.168.220.104:5001/k8s-harbor/nginx   latest    605c77e624dd   2 years ago   141MB

七，使用dashboard对整个集群资源进行掌控

[root@master ~]# mkdir dashboard
[root@master ~]# cd dashboard/
[root@master dashboard]#  wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

[root@master dashboard]# ls
recommended.yaml

修改配置文件，因为访问用户需要RBAC鉴权才能访问资源。
 [root@master dashboard]# vim recommended.yaml 
---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort  #指定类型
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30088  #指定宿主机端口号
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: serviceaccount-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:serviceaccount:kubernetes-dashboard:kubernetes-dashboard

查看是否启动dashboard的pod
[root@master dashboard]# kubectl get pod --all-namespaces|grep dashboard
kubernetes-dashboard   dashboard-metrics-scraper-799d786dbf-lltbf   1/1     Running     0              39s
kubernetes-dashboard   kubernetes-dashboard-546cbc58cd-p4xlr        1/1     Running     0              39s

查看服务是否创建
[root@master dashboard]# kubectl get svc --all-namespaces|grep dash
kubernetes-dashboard   dashboard-metrics-scraper            ClusterIP   10.98.46.11      <none>        8000/TCP                     66s
kubernetes-dashboard   kubernetes-dashboard                 NodePort    10.109.239.147   <none>        443:30088/TCP                66s


在浏览器里访问，使用https协议去访问
https://192.168.220.100:30088/

点击继续访问

https://192.168.220.100:30088/#/login


出现一个登录画图，需要输入token


获取dashboard 的secret的名字
[root@master dashboard]# kubectl get secret -n kubernetes-dashboard|grep dashboard-token
kubernetes-dashboard-token-pnt2v   kubernetes.io/service-account-token   3      6m6s


获取secret里的token
[root@master dashboard]# kubectl describe secret kubernetes-dashboard-token-pnt2v  -n kubernetes-dashboard
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImhvV1g5cTQ1Q2F1N1A5RGxCQnhrTkVkeFNmczgtRG5WNlFMNWJ4SzcyaTQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1wbnQydiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjBkYmYwMGFkLWU1YTktNDc0Ny05YWZiLWM0MDk2N2RmY2I1MCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.l1U9GljbDpv3OpQcqijj10YaqymqLC18pY1Ut6-UPzNiY8sSyKvpnC9_6aCMFLz-mXV2x17TWmvOME5mK8DO0pV8QH3JJcsS-XCkn3RBxHygZfVYqcpSnoPibCA6QhaCCchSYDQ9a6fmriIztySgsXCtNV4Rfow49l5pkafTYLllV3dXp5SbxsL3TL46IOXw2uQ0iG0JD5QYj4pMfe_rZiwDQNwriaVqLb84K88AglDh3uniPg8XuYWs_nDIy3pztdwQOjWFLCy8NsQ1TGftZg6HRXD9pon2W8QeUj3vhKvA1B8L1MdzSfGpLPIojjHVLHB9C6aCnI3HqrjjvmrKjA

然后在浏览器上就可以访问资源了

八，安装部署Prometheus+grafana：

安装Prometheus

[root@prometheus ~]# ls
anaconda-ks.cfg
grafana-enterprise-9.1.2-1.x86_64.rpm
mysqld_exporter-0.12.1.linux-amd64 (1).tar.gz
prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus ~]# mkdir /prom
[root@prometheus ~]# mv prometheus-2.43.0.linux-amd64.tar.gz  /prom/prometheus-2.43.0.linux-amd64.tar.gz 
[root@prometheus ~]# cd /prom
[root@prometheus prom]# ls
prometheus-2.43.0.linux-amd64.tar.gz
[root@prometheus prom]# tar xf prometheus-2.43.0.linux-amd64.tar.gz 
[root@prometheus prom]# ls
prometheus-2.43.0.linux-amd64
[root@prometheus prom]# mv prometheus-2.43.0.linux-amd64 prometheus
[root@prometheus prom]# ls
prometheus  prometheus-2.43.0.linux-amd64.tar.gz

[root@prometheus prometheus]# PATH=/prom/prometheus:$PATH

[root@prometheus prometheus]# vim /etc/profile
添加到末尾：PATH=/prom/prometheus:$PATH

[root@prometheus prometheus]# nohup prometheus --config.file=/prom/prometheus/prometheus.yml  & #在后台运行
[1] 2137
[root@prometheus prometheus]# nohup: 忽略输入并把输出追加到"nohup.out"

看进程：
[root@prometheus prometheus]# ps aux |grep prome
root       2137  0.4  2.3 798956 44252 pts/0    Sl   12:38   0:00prometheus --config.file=/prom/prometheus/prometheus.yml
看端口：
[root@prometheus prometheus]# netstat -anplut |grep prom
tcp6       0      0 :::9090                 :::*                    LISTEN      2137/prometheus     
tcp6       0      0 ::1:48882               ::1:9090                ESTABLISHED 2137/prometheus     
tcp6       0      0 ::1:9090                ::1:48882               ESTABLISHED 2137/prometheus     
关闭防火墙：
[root@prometheus prometheus]# service firewalld stop
Redirecting to /bin/systemctl stop firewalld.service
[root@prometheus prometheus]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
把Prometheus做成一个服务进行管理，
[root@prometheus prometheus]# vim /usr/lib/systemd/system/prometheus.service

[Unit]
Description=prometheus
[Service]
ExecStart=/prom/prometheus/prometheus --config.file=/prom/promethe
us/prometheus.yml
ExecReload=/bin/kill -HUP $MAINPID
killMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target

[root@prometheus prometheus]# systemctl daemon-reload #重新加载systemd相关的服务

浏览器访问ip+9090端口

杀死Prometheus进程  ：
[root@prometheus prometheus]# ps aux|grep prom
root       2137  0.0  2.9 799212 54400 pts/0    Sl   12:38   0:00prometheus --config.file=/prom/prometheus/prometheus.yml
root       2346  0.0  0.0 112824   972 pts/0    S+   12:59   0:00 grep --color=auto prom
[root@prometheus prometheus]# kill -9 2137

在master或者被监听的机器上操作（安装相应的exporter，常规的是node_exporter）：

[root@master ~]# tar xf node_exporter-1.4.0-rc.0.linux-amd64.tar.gz 
[root@master ~]# mv node_exporter-1.4.0-rc.0.linux-amd64 /node_exporter
[root@master ~]# cd /node_exporter/
[root@master node_exporter]# ls
LICENSE  node_exporter  NOTICE

修改环境变量
[root@master node_exporter]# PATH=/node_exporter/:$PATH
[root@master node_exporter]# vim /root/.bashrc 
PATH=/node_exporter/:$PATH

[root@prometheus /]# nohup node_exporter --web.listen-address 0.0.0.0:8090 &
[1] 4844
[root@prometheus /]# nohup: 忽略输入并把输出追加到"nohup.out"


[root@prometheus /]# ps aux |grep node
root       4844  0.0  0.7 716544 13104 pts/0    Sl   13:55   0:00 node_exporter --web.listen-address 0.0.0.0:8090

关闭防火墙：
[root@prometheus /]# service firewalld stop
Redirecting to /bin/systemctl stop firewalld.service
[root@prometheus /]# systemctl disable  firewalld
查看selinux的状态：
[root@prometheus /]# getenforce
关闭selinux：
#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/'  /etc/selinux/config

浏览器访问：ip+8090

设置node_exporter开机启动
[root@prometheus /]# vim /etc/rc.local 
nohup /node_exporter/node_exporter/node_exporter --web.listen-address 0.0.0.0:8090 &
[root@prometheus node_exporter]# chmod +x /etc/rc.d/rc.local

在Prometheus里添加我们在哪些机器里安装了exporter程序，就可以去pull了


  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "master"
    static_configs:
      - targets: ["192.168.220.100:8090"]
（。。。）
 
~                                            
[root@prometheus prometheus]# service prometheus  restart
Redirecting to /bin/systemctl restart prometheus.service

安装部署Grafana和Prometheus安装在一起：

wget https://d1.grafana.com/enterprise/release/grafana-enterprise-9.1.2-1.x86_64.rpm

[root@prometheus ~]# yum install grafana-enterprise-9.1.2-1.x86_64.rpm  -y
启动
[root@prometheus ~]# service grafana-server start
Starting grafana-server (via systemctl):                   [  确定  ]

[root@prometheus ~]# ps aux |grep grafana
grafana    5115  2.1  3.6 1129728 68768 ?       Ssl  14:28   0:00 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
root       5124  0.0  0.0 112824   976 pts/0    S+   14:28   0:00 grep --color=auto grafana
[root@prometheus ~]# netstat -anlput |grep grafana
tcp        0      0 192.168.220.165:54852   34.120.177.193:443      ESTABLISHED 5115/grafana-server 
tcp        0      0 192.168.220.165:46122   185.199.109.133:443     ESTABLISHED 5115/grafana-server 
tcp6       0      0 :::3000                 :::*                    LISTEN      5115/grafana-server 
浏览器访问 ip+3000端口
默认用户admin
密码：admin

添加数据源：
管理——》数据源-》添加Prometheus -》http://192.168.220.105:9090

添加模板：1860  
仪表盘-》 导入

监听的是master的使用情况

九.安装部署firewalld防火墙和jumpserver堡垒机，对web集群进行保护。

jumpserver 的安装部署

准备一台 2核4G （最低）且可以访问互联网的 64 位 Linux 主机；
以 root 用户执行如下命令一键安装 JumpServer。

curl -sSL https://resource.fit2cloud.com/jumpserver/jumpserver/releases/latest/download/quick_start.sh | bash

部署firewall服务器，对内网进行保护

ens33的wan口ip为外面window的ip网段，连接外网

# 关闭虚拟机，增加一块网卡（ens36）用来连内网
vim /etc/sysconfig/network-scripts/ifcfg-ens36
关闭dhcp
添加：
IPADDR=192.168.220.107
GATEWAY=192.168.220.106 #jump server的ip地址
NETMASK=255.255.255.0
DNS1=114.114.114.114

而且内网的机器都需要把网关设置为firewalld的ip
GATEWAY=192.168.220.108
 #内网的机器出去都需要通过这台firewalld机器

# 编写脚本实现SNAT_DNAT功能
[root@firewalld ~]# cat snat_dnat.sh 
#!/bin/bash
 
# 开启路由功能
echo 1 >/proc/sys/net/ipv4/ip_forward
 
# stop firewall
systemctl   stop  firewalld
systemctl disable firewalld
 
# clear iptables rule
iptables -F
iptables -t nat -F
 
# enable snat
iptables -t nat  -A POSTROUTING  -s 192.168.220.0/24  -o ens33  -j  MASQUERADE
#内网来的192.168.220.0网段过来的ip地址全部伪装（替换）为ens33接口的公网ip地址,好处就是不需要考虑ens33接口的ip地址是多少，你是哪个ip地址，我就伪装成哪个ip地址
 
 iptables  -t filter  -P INPUT ACCEPT #默认是ACCPET

 
 
# web服务器上操作，开启相应的端口，默认为drop，实现对k8s集群的保护。
[root@k8smaster ~]# cat open.sh 
#!/bin/bash
 
# open ssh
iptables -t filter  -A INPUT  -p tcp  --dport  22 -j ACCEPT
 
# 打开 dns
iptables -t filter  -A INPUT  -p udp  --dport 53 -s 192.168.220.0/24 -j ACCEPT  可以dns解析来自于内网段的机器ip 
 
# 打开 dhcp 
iptables -t filter  -A INPUT  -p udp   --dport 67 -j ACCEPT
 
# 打开 http/https
iptables -t filter  -A INPUT -p tcp   --dport 80 -j ACCEPT
iptables -t filter  -A INPUT -p tcp   --dport 443 -j ACCEPT
 
# 打开mysql端口
iptables  -t filter  -A INPUT -p tcp  --dport 3306  -j ACCEPT
 
# 默认为drop
iptables  -t filter  -P INPUT DROP

 

十.安装部署ansible机器，写好主机清单,便于日后的自动化运维。

1.建立免密通道 在ansible主机上生成密钥对，上传公钥到所有服务器的root用户家目录下

#     所有服务器上开启ssh服务 ，开放22号端口，允许root用户登录
[root@ansible ~]# yum install -y epel-release
[root@ansible ~]# yum install ansible
[root@ansible ~]# ssh-keygen 
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.100
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.101
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.102
[root@ansible ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.220.103
（。。。）

 2.编写主机清单

[root@ansible ansible]# vim hosts 

[master]
192.168.220.100
[node]
192.168.220.101
192.168.220.102
[nfs]
192.168.220.103
[harbor]
192.168.220.104
[prometheus]
192.168.220.105
[jumpserver]
192.168.220.106

3.测试

[root@ansible ansible]# ansible all -m shell -a "ip add"

项目心得：

1.更加深入的了解了k8s的各个功能（service，pv，pvc，ingress）等。

2.更加了解开发和运维的关系。

3.对负载均衡和高可用， 自动扩缩有了认识。

4.对各个服务(Prometheus, nfs等) 深入了解