Kubernetes--二进制多节点部署

最新推荐文章于 2024-04-28 20:36:11 发布

置顶 Chenjian0824

最新推荐文章于 2024-04-28 20:36:11 发布

阅读量452

点赞数

分类专栏： kubernetes

本文链接：https://blog.csdn.net/weixin_47151650/article/details/108855439

版权

kubernetes 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

文章目录

一、群集理论基础
- 1.1 flannel网络理论介绍
- 1.2 各节点组件介绍
二、单节点部署
三、多节点部署
四、部署nginx负载均衡
- 4.2: keepalived热备部署

一、群集理论基础

1.1 flannel网络理论介绍

●Overlay Network：覆盖网络，在基础网络上叠加的一种虚拟化网络技术模式，该网络中的主机通过虚拟链路连接起来

●VXLAN：将源数据包封装到UDP中，并使用基础网络的IP/MAC作为外层报文头进行封装，然后在以太网上进行传输，到达目的地后由隧道端点解封装并将数据发送给目标地址

●Flannel：是Overlay网络的一种，也是将源数据包封装在另一种网络包里面进行路由转发和通信，目前已经支持UDP、VXLAN、AWS VPC和GCE路由等数据转发方式

●Flannel是CoreOS团队针对 Kubernetes设计的一个网络规划服务，简单来说，它的功能是让集群中的不同节点主机创建的 Docker容器都具有全集群唯一的虚拟IP地址。而且它还能在这些IP地址之间建立一个覆盖网络（overlay Network），通过这个覆盖网络，将数据包原封不动地传递到目标容器内

●ETCD在这里的作用：为Flannel提供说明

●存储管理 Flannel可分配的IP地址段资源
●监控ETCD中每个Pod的实际地址，并在内存中建立维护Pod节点路由表

1.2 各节点组件介绍

Master01:192.168.100.130/24 kube-apiserver kube-controller-manager kube-scheduler etcd
Masetr02:192.168.100.88/24 kube-apiserver kube-controller-manager kube-scheduler etcd
Node01:192.168.100.128/24 kubelet kube-proxy docker flannel etcd
Node02:192.168.100.129/24 kubelet kube-proxy docker flannel etcd
负载均衡nginx:192.168.100.77/24
负载均衡nginx02:192.168.100.66/24
VIP:192.168.100.99

●master组件

●kube-apiserver：是集群的统一入口，各个组件的协调者，所有对象资源的增删改查和监听操作都交给APIserver处理，再提交给etcd存储。

●kube-controller-manager：处理群集中常规的后台任务，一个资源对应一个控制器，而controller-manager就是负责管理这些控制器。

●kube-scheduler：根据调度算法为新创建的pod选择一个node节点，可以任意部署，可以部署同一个节点上，也可以部署在不同节点上

●node组件

●kubelet：kube是master在node节点上的Agent，管理本机运行容器的生命周期，比如创建容器、Pod挂载数据卷、下载secret、获取容器和节点状态等工作。kubelet将每个pod转换成一组容器

●kube-proxy：在node节点上实现pod网络代理，维护网络规划和四层负载均衡的工作

●docker：Docker引擎

●flannel：flannel网络

●etcd集群介绍：etcd集群在这里分布的部署到了三个节点上

●etcd是CoreOS团队于2013年6月发起的开源项目，基于go语言开发，目标是构建一个高可用的分布式键值（key-value）数据库。etcd内部采用raft协议作为一致性算法。

etcd集群数据无中心化集群，有如下特点：

1、简单：安装配置简单，而且提供了HTTP进行交互，使用也很简单

2、安全：支持SSL证书验证

3、快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作

4、可靠：采用raft算法，实现分布式数据的可用性和一致性

部署K8S集群中会用到的自签SSL证书

二、单节点部署

[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls
etcd-cert.sh  etcd.sh

下载证书制作工具

[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
下载cfssl官方包
[root@localhost k8s]# bash cfssl.sh
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo  给与权限

[root@localhost k8s]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson
开始制作证书
cfssl 生成证书工具   cfssljson通过传入json文件生成证书
cfssl-certinfo查看证书信息

定义ca证书

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"     
        ]  
      } 
    }         
  }
}
EOF

实现证书签名

cat > ca-csr.json <<EOF 
{   
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生产证书，生成ca-key.pem ca.pem

[root@localhost k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2020/09/28 19:02:44 [INFO] generating a new CA key and certificate from CSR
2020/09/28 19:02:44 [INFO] generate received request
2020/09/28 19:02:44 [INFO] received CSR
2020/09/28 19:02:44 [INFO] generating key: rsa-2048
2020/09/28 19:02:45 [INFO] encoded CSR
2020/09/28 19:02:45 [INFO] signed certificate with serial number 66900407245912790326618564412446238422596508246

指定etcd三个节点之间的通信验证

cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.100.130",
    "192.168.100.128",
    "192.168.100.129"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

生成ETCD证书 server-key.pem server.pem

[root@localhost k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2020/09/28 19:04:36 [INFO] generate received request
2020/09/28 19:04:36 [INFO] received CSR
2020/09/28 19:04:36 [INFO] generating key: rsa-2048
2020/09/28 19:04:36 [INFO] encoded CSR
2020/09/28 19:04:36 [INFO] signed certificate with serial number 592282415758029159373729005317564028574004065756
2020/09/28 19:04:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@localhost k8s]# ls
ca-config.json  ca-csr.json  ca.pem     etcd.sh     server-csr.json  server.pem
ca.csr          ca-key.pem   etcd-cert  server.csr  server-key.pem

ETCD 二进制包地址
https://github.com/etcd-io/etcd/releases
在这里插入图片描述

复制到centos7中

[root@localhost etcd-cert]# ls
ca-config.json  etcd-cert.sh                          server-csr.json
ca.csr          etcd-v3.3.10-linux-amd64.tar.gz       server-key.pem
ca-csr.json     flannel-v0.10.0-linux-amd64.tar.gz    server.pem
ca-key.pem      kubernetes-server-linux-amd64.tar.gz
ca.pem          server.csr

[root@localhost etcd-cert]# mv *.tar.gz ../
[root@localhost k8s]# ls
cfssl.sh   etcd.sh                          flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz

[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

配置文件，命令文件，证书

[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p  
[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
证书拷贝
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/
进入卡住状态等待其他节点加入
[root@localhost k8s]# bash etcd.sh etcd01 192.168.100.130 etcd02=https://192.168.100.128:2380,etcd03=https://192.168.100.129:2380

使用另外一个会话打开，会发现etcd进程已经开启

[root@localhost ~]# ps -ef | grep etcd

拷贝证书去其他节点

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.100.128:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.100.129:/opt
启动脚本拷贝其他节点
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.100.128:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.100.129:/usr/lib/systemd/system/

在node01节点修改

[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.128:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.128:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.128:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.128:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.100.130:2380,etcd02=https://192.168.100.128:2380,etcd03=https://192.168.100.129:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
启动
[root@localhost ssl]# systemctl start etcd
[root@localhost ssl]# systemctl status etcd

在node02节点修改

[root@localhost ~]# vim /opt/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.129:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.129:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.129:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.129:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.100.130:2380,etcd02=https://192.168.100.128:2380,etcd03=https://192.168.100.129:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
启动
[root@localhost ssl]# systemctl start etcd
[root@localhost ssl]# systemctl status etcd

检查群集状态

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.100.130:2379,https://192.168.100.128:2379,https://192.168.100.129:2379" cluster-health
member 4741bde0cf25b4fd is healthy: got healthy result from https://192.168.100.128:2379
member 710606be9e92d6f9 is healthy: got healthy result from https://192.168.100.129:2379
member fa0f0a82b9aff014 is healthy: got healthy result from https://192.168.100.130:2379
cluster is healthy

docker引擎部署可以看下我的博客关于部署docker：
https://blog.csdn.net/weixin_47151650/article/details/108691654

flannel网络配置
写入分配的子网段到ETCD中，供flannel使用

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.195.130:2379,https://192.168.100.129:2379,https://192.168.100.128:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

查看写入的信息

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.100.130:2379,https://192.168.100.128:2379,https://192.168.100.129:2379" get /coreos.com/network/config
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

拷贝到所有node节点（只需要部署在node节点即可）

[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.100.128:/root
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.100.129:/root
所有node节点操作解压
[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz 
flanneld
mk-docker-opts.sh
README.md

#!/bin/bash

ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld

开启flannel网络功能

[root@localhost ~]# bash flannel.sh https://192.168.100.130:2379,https://192.168.100.128:2379,https://192.168.100.129:2379

Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.

配置docker连接flannel

[root@localhost ~]# vim /usr/lib/systemd/system/docker.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/subnet.env    添加的环境变量
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

[root@localhost ~]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.42.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
说明：bip指定启动时的子网
DOCKER_NETWORK_OPTIONS=" --bip=172.17.42.1/24 --ip-masq=false --mtu=1450"

测试ping通对方docker0网卡证明flannel起到路由作用

[root@localhost ~]# docker run -it centos:7 /bin/bash
[root@5f9a65565b53 /]# yum install net-tools -y
[root@be5a2d954d22 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.80.2  netmask 255.255.255.0  broadcast 172.17.80.255
        ether 02:42:ac:11:50:02  txqueuelen 0  (Ethernet)
        RX packets 15862  bytes 12464939 (11.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7612  bytes 414637 (404.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

进入node02容器

[root@f284f66742b0 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 1575  bytes 11845405 (11.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1574  bytes 88522 (86.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

互相ping通

[root@f284f66742b0 /]# ping 172.17.80.2
PING 172.17.80.2 (172.17.80.2) 56(84) bytes of data.
64 bytes from 172.17.80.2: icmp_seq=1 ttl=62 time=0.549 ms
64 bytes from 172.17.80.2: icmp_seq=2 ttl=62 time=0.444 ms
64 bytes from 172.17.80.2: icmp_seq=3 ttl=62 time=0.399 ms
64 bytes from 172.17.80.2: icmp_seq=4 ttl=62 time=0.467 ms
^C
--- 172.17.80.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.399/0.464/0.549/0.060 ms

三、多节点部署

单节点部署完了，接着部署多节点

优先关闭防火墙和selinux服务
在master01上操作
复制kubernetes目录到master02
[root@localhost k8s]# scp -r /opt/kubernetes/ root@192.168.100.88:/opt
The authenticity of host '192.168.100.88 (192.168.100.88)' can't be established.
ECDSA key fingerprint is SHA256:IJ43xXlBWD7qPaL/uFG+4qW4qd7C8xBqUttHiYME8YE.
ECDSA key fingerprint is MD5:cf:3e:dc:e5:89:86:e9:43:38:ee:31:9d:8c:d4:75:9f.
Are you sure you want to continue connecting (yes/no)? yes

复制master中的三个组件启动脚本kube-apiserver.service kube-controller-manager.service kube-scheduler.service

[root@localhost k8s]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.100.88:/usr/lib/systemd/system/
root@192.168.100.88's password: 
kube-apiserver.service                                            100%  282   268.1KB/s   00:00    
kube-controller-manager.service                                   100%  317   294.2KB/s   00:00    
kube-scheduler.service                                            100%  281   257.5KB/s   00:00

master02上操作
修改配置文件kube-apiserver中的IP

[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# vim kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.100.130:2379,https://192.168.100.128:2379,https://192.168.100.129:2379 \
--bind-address=192.168.100.88 \    '修改成master02的IP'
--secure-port=6443 \
--advertise-address=192.168.100.88 \

特别注意：master02一定要有etcd证书
需要拷贝master01上已有的etcd证书给master02使用

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.100.88:/opt/
root@192.168.100.88's password: 
etcd                                                              100%  523   415.0KB/s   00:00    
etcd                                                              100%   18MB  42.7MB/s   00:00    
etcdctl                                                           100%   15MB  35.2MB/s   00:00    
ca-key.pem                                                        100% 1675   612.1KB/s   00:00    
ca.pem                                                            100% 1265     1.0MB/s   00:00    
server-key.pem                                                    100% 1679     1.7MB/s   00:00    
server.pem                                                        100% 1338     1.7MB/s   00:00

启动master02中的三个组件服务

[root@localhost cfg]# systemctl start kube-apiserver.service 
[root@localhost cfg]# systemctl start kube-controller-manager.service 
[root@localhost cfg]# systemctl start kube-scheduler.service

增加环境变量

[root@localhost cfg]# vim /etc/profile
#末尾添加
export PATH=$PATH:/opt/kubernetes/bin/
[root@localhost cfg]# source /etc/profile

多节点部署完成

[root@master02 cfg]# kubectl get nodes
NAME              STATUS   ROLES    AGE     VERSION
192.168.100.128   Ready    <none>   11h   v1.12.3
192.168.100.129   Ready    <none>   11h   v1.12.3

四、部署nginx负载均衡

两台nginx的操作是一样的，我这里展示的nginx01的操作

主机开局优化：关闭防火墙和核心防护，编辑nginx yum源

[root@nginx01 ~]# systemctl stop firewalld.service
[root@nginx01 ~]# vim /etc/yum.repos.d/nginx.repo 	
[nginx]
name=nginx.repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
enabled=1
gpgcheck=0
[root@nginx01 ~]# yum -y install nginx

编辑nginx主配置文件，进行负载均衡的配置

[root@nginx01 ~]# vim /etc/nginx/nginx.conf
 13 stream {
 14 
 15      log_format  main  '$remote_addr $upstream_addr - [$time_loc    al] $status $upstream_bytes_sent';
 16       access_log  /var/log/nginx/k8s-access.log  main;  '指定日志目录'
 17 
 18       upstream k8s-apiserver {
 19   #此处为master的ip地址和端口
 20           server 192.168.100.130:6443;     
 21   #此处为master2的ip地址和端口
 22           server 192.168.100.88:6443;
 23       }
 24       server {
 25                   listen 6443;
 26                   proxy_pass k8s-apiserver;
 27       }
 28       }

开启nginx服务

[root@nginx01 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successf
[root@nginx01 ~]# systemctl start nginx
[root@nginx01 ~]# systemctl status nginx
[root@nginx01 ~]# netstat -ntap |grep nginx
tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      11357/nginx: master 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      11357/nginx: maste

4.2: keepalived热备部署

两台nginx一样的操作，稍有不同处我会进行备注

安装keepalived服务

[root@nginx01 ~]# yum install keepalived -y
[root@nginx01 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"  'keepalived服务检查脚本的位置'
}

vrrp_instance VI_1 {
    state MASTER           'nginx02设置为BACKUP'
    interface ens33
    virtual_router_id 51   '这里的id两台nginx必须一样'
    priority 100            '优先级，nginx02可设置为90'
    advert_int 1            '指定VRRP 心跳包通告间隔时间，默认1秒 '
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.99/24    'VIP地址'
    }
    track_script {
        check_nginx
    }
}

创建监控脚本，启动keepalived服务，查看VIP地址

[root@nginx01 ~]# mkdir -p /usr/local/nginx/sbin/
[root@nginx01 ~]# vim /usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@nginx01 ~]# chmod +x /usr/local/nginx/sbin/check_nginx.sh
[root@nginx01 ~]# systemctl start keepalived.service 
[root@nginx01 ~]# systemctl status keepalived.service 

[root@nginx01 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:27:0a:08 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.66/24 brd 192.168.100.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.100.99/24 scope global secondary ens33      'VIP出现'
       valid_lft forever preferred_lft forever
    inet6 fe80::ad2c:2952:74ec:d22a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:60:d3:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:60:d3:98 brd ff:ff:ff:ff:ff:ff


[root@nginx02 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ee:e6:9a brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.77/24 brd 192.168.100.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5772:e862:c78b:57da/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:f3:bb:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:f3:bb:59 brd ff:ff:ff:ff:ff:ff

对漂移地址进行验证，在node01中关闭nginx服务，已查看不到VIP，此时VIP已飘到了node02中

nginx01已经没有VIP
[root@nginx01 keepalived]# pkill nginx
[root@nginx01 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:27:0a:08 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.66/24 brd 192.168.100.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::ad2c:2952:74ec:d22a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:60:d3:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:60:d3:98 brd ff:ff:ff:ff:ff:ff



nginx02出现VIP
[root@nginx02 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ee:e6:9a brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.77/24 brd 192.168.100.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.100.99/24 scope global secondary ens33     '出现VIP'
       valid_lft forever preferred_lft forever
    inet6 fe80::5772:e862:c78b:57da/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:f3:bb:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:f3:bb:59 brd ff:ff:ff:ff:ff:ff

将node01中服务恢复后，地址会飘回来

[root@nginx01 ~]# systemctl start nginx
[root@nginx01 ~]# systemctl start keepalived.service

修改node的VIP以及pod的创建
两台node节点都要修改VIP，我这里只展示了node01的操作

[root@node01 ~]# cd /opt/kubernetes/cfg/
[root@node01 cfg]# vim bootstrap.kubeconfig 
server: https://192.168.100.99:6443     '将地址修改为VIP地址'
[root@node01 cfg]# vim kubelet.kubeconfig 
server: https://192.168.100.99:6443     '将地址修改为VIP地址'
[root@node01 cfg]# vim kube-proxy.kubeconfig 
server: https://192.168.100.99:6443      '将地址修改为VIP地址'
[root@node01 cfg]# grep 99 *
bootstrap.kubeconfig:    server: https://192.168.100.99:6443
kubelet.kubeconfig:    server: https://192.168.100.99:6443
kube-proxy.kubeconfig:    server: https://192.168.100.99:6443

[root@node01 cfg]# systemctl restart kubelet.service 
[root@node01 cfg]# systemctl restart kube-proxy.service

此时查看k8s的日志，可以看到访问信息，是两台master轮流访问的，大大缓解了master的压力

[root@nginx01 keepalived]# tail /var/log/nginx/k8s-access.log 
192.168.100.129 192.168.100.130:6443 - [06/Oct/2020:22:22:35 +0800] 200 171
192.168.100.129 192.168.100.88:6443 - [06/Oct/2020:22:22:35 +0800] 200 171
192.168.100.129 192.168.100.130:6443 - [06/Oct/2020:22:22:35 +0800] 200 171
192.168.100.128 192.168.100.88:6443 - [06/Oct/2020:22:22:36 +0800] 200 171

在master上创建pod

[root@master ~]# kubectl run nginx --image=nginx  '创建nginx的pod资源'
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx
[root@master ~]# kubectl get pods     '此时查看正在创建中'
NAME                    READY   STATUS              RESTARTS   AGE
nginx-dbddb74b8-cnkd6   0/1     ContainerCreating   0          49s
[root@master ~]# kubectl get pods     '创建完毕了'
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-cnkd6   1/1     Running   0          4m43s

此时查看pod日志是没有权限的

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-cnkd6)

指定集群中的匿名用户有管理员权限

[root@master ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created

此时可以查看pod日志，只能看到一些启动日志

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

查看pod的IP网络信息，在对应的节点进行访问

[root@master ~]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE              NOMINATED NODE
nginx-dbddb74b8-cnkd6   1/1     Running   0          10m   172.17.71.3   192.168.100. 66  <none>
[root@node02 cfg]# curl 172.17.71.3

此时再次查看pod的日志信息，可以看到访问信息了

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
172.17.71.1 - - [29/Sep/2020:12:04:29 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
[root@master2 ~]# kubectl logs nginx-dbddb74b8-cnkd6
172.17.71.1 - - [29/Sep/2020:12:04:29 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"