kubernetes 1.18.8 高可用安装

1 集群规划

角色	ip地址
k8s-vip	192.168.109.150
master1	192.168.109.151
master2	192.168.109.152
master3	192.168.109.153
node1	192.168.109.154

2 安装要求

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：

1. 一台或多台机器，操作系统 CentOS7.x-86_x64
2. 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
3. 可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
4. 禁止swap分区

3 准备环境

关闭防火墙
systemctl stop firewalld.service && systemctl disable firewalld.service

设置 SELinux 为 disabled 模式
setenforce 0 && sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

禁用交换分区
swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab

修改每个系统的主机名 永久修改
hostnamectl set-hostname xxx


#将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

安装ntp:
yum install -y ntp
ntpdate time.windows.com && hwclock -w  同步时间且写入硬件

在所有的节点添加hosts
cat >> /etc/hosts << EOF
192.168.109.150 k8s-vip
192.168.109.151 master1
192.168.109.152 master2
192.168.109.153 master3
192.168.109.154 node1
EOF

4 所有master节点部署keepalived

4.1 安装相关包和keepalived

yum install -y conntrack-tools libseccomp libtool-ltdl

yum install -y keepalived

4.2 配置master节点

master1、master2、master3节点配置

cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER 
    interface ens33 
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        192.168.109.150
    }
    track_script {
        check_haproxy
    }

}
EOF

注：
1、virtual_ipaddress 是 VIP的地址
2、interface 网络请注意，ifconfig查看
3、priority 优先级,备lvs要比主lvs稍小
4、state 主MASTER 备BACKUP

4.3 启动和检查

# 启动keepalived && 设置开机启动
systemctl start keepalived.service && systemctl enable keepalived.service

# 查看启动状态
systemctl status keepalived.service

# 重启
systemctl restart keepalived.service

启动后查看master1的网卡信息

# 查询网卡信息 ens33 对应自己的网卡
ip a s ens33

[root@master1 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:28:58:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.109.151/24 brd 192.168.109.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::e119:2c13:fa0a:3953/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@master2 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:3e:cd:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.109.152/24 brd 192.168.109.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::e119:2c13:fa0a:3953/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9f0e:5697:e453:e0b4/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::4faf:a02d:4291:70ae/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

[root@master3 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:4b:35:44 brd ff:ff:ff:ff:ff:ff
    inet 192.168.109.153/24 brd 192.168.109.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.109.150/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::e119:2c13:fa0a:3953/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9f0e:5697:e453:e0b4/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

注：3个有不同，在master3节点上有150的地址，其他的都没有，只有master3发生故障之后VIP才会漂移到 master1 或者 master2上

5 所有master节点部署haproxy

5.1 安装

yum install -y haproxy

5.2 配置

注：三台master节点的配置均相同，配置中声明了后端代理的三个master节点服务器，指定了haproxy运行的端口为16443等，因此16443端口为集群的入口
修改自己的IP地址 backend kubernetes-apiserver 下的 server

cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon 
       
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------  
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#--------------------------------------------------------------------- 
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver    
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server      master1   192.168.109.151:6443 check
    server      master2   192.168.109.152:6443 check
    server      master3   192.168.109.153:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats
EOF

5.3 启动和检查

三台master都启动

# 开启haproxy && 设置开机启动 
systemctl start haproxy && systemctl enable haproxy

# 查看启动状态
systemctl status haproxy

检查端口

netstat -lntup|grep haproxy

6 所有节点安装Docker/kubeadm/kubelet

Kubernetes默认CRI（容器运行时）为Docker，因此先安装Docker。

6.1 安装docker

卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

第一步
yum install -y yum-utils device-mapper-persistent-data lvm2

第二步
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

第三部
推荐19版本
yum install docker-ce-19.03.9-3.el7


镜像加速
data-root 是 存储位置

mkdir -p /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
  "registry-mirrors": ["xxxxx"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "storage-driver": "overlay2",
  "storage-opts": ["overlay2.override_kernel_check=true"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
    },
    "insecure-registries":["127.0.0.1"],
    "data-root":"/home/docker-data"
}
EOF

# 修改docker.service文件，使用-g参数指定存储位置
vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --graph /home/docker-data


开机自动启动
systemctl start docker && systemctl enable docker

registry-mirrors=对应阿里加速地址

6.2 添加阿里云YUM软件源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

6.3 安装kubeadm，kubelet和kubectl

yum install -y kubelet-1.18.8 kubeadm-1.18.8 kubectl-1.18.8
systemctl enable kubelet

7 部署Kubernetes Master

7.1 创建kubeadm配置文件

看上面的标题 4.3 文档
在具有vip的master上操作，这里为master3
以下操作是master3上

$ mkdir /usr/local/kubernetes/manifests -p

$ cd /usr/local/kubernetes/manifests/

$ vi kubeadm-config.yaml

内容如下
apiServer:
  certSANs:
    - master1
    - master2
    - master3
    - k8s-vip
    - 192.168.109.150
    - 192.168.109.151
    - 192.168.109.152
    - 192.168.109.153
    - 127.0.0.1
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "k8s-vip:16443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
  local:    
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.8
networking: 
  dnsDomain: cluster.local  
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.1.0.0/16
scheduler: {}

注：certSANs说明，是master的所有节点ip和hostname，在加上VIP的hostname和VIP的ip地址,在加上 127.0.0.1

7.2 在master3节点执行

cd /usr/local/kubernetes/manifests/

# 查看所需镜像列表
kubeadm config images list --config kubeadm-config.yaml

# 拉取镜像
kubeadm config images pull --config kubeadm-config.yaml

# kubeadm初始化
kubeadm init --config kubeadm-config.yaml

按照提示配置环境变量，使用kubectl工具：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get nodes
$ kubectl get pods -n kube-system

注：kubectl get nodes 会出现notReady ,这是正常的，因为没安装网络插件

按照提示保存以下内容，一会要使用：

kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
    --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
    --control-plane

kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
    --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d

查看集群状态

kubectl get cs

kubectl get pods -n kube-system

8 安装集群网络

8.1 安装 flannel（弃用）

从官方地址获取到flannel的yaml，在master3上执行

cd /usr/local/kubernetes/manifests/
mkdir flannel
cd flannel
wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

安装flannel网络

cd /usr/local/kubernetes/manifests/flannel
kubectl apply -f kube-flannel.yml 

检查

kubectl get pods -n kube-system

8.2 安装 calico （推荐）

下载

cd /usr/local/kubernetes/manifests/
mkdir calico
cd calico
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml

安装calico网络

cd /usr/local/kubernetes/manifests/calico
kubectl apply -f calico-3.13.1.yaml

检查

kubectl get pods -n kube-system

9 ssh免密登录

在master3执行

ssh-keygen -t rsa
一路回车

$IPs 是master的hostname
ssh-copy-id master1
ssh-copy-id master2
ssh-copy-id $IPs 按照提示输入yes 和root密码

10 master1 和 master2 节点加入集群

10.1 复制密钥

从master3复制密钥及相关文件到master1

ssh root@master1 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master1:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master1:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master1:/etc/kubernetes/pki/etcd

从master3复制密钥及相关文件到master2

ssh root@master2 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master2:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master2:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master2:/etc/kubernetes/pki/etcd

10.2 master1 加入集群

kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
    --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
    --control-plane

10.3 master2 加入集群

kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
    --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
    --control-plane

检查状态

kubectl get node

kubectl get pods --all-namespaces

11 加入Kubernetes Node

在node1上执行
向集群添加新节点，执行在kubeadm init输出的kubeadm join命令：

kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
    --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d

检查状态

kubectl get node

kubectl get pods --all-namespaces

12 测试集群

在Kubernetes集群中创建一个pod，验证是否正常运行：

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc

13 测试集群

在Kubernetes集群中创建一个pod，验证是否正常运行：

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc

访问地址：http://NodeIP:Port

14 获取join命令参数

kubeadm token create --print-join-command

得到结果
[root@master3 ~]# kubeadm token create --print-join-command
W1015 17:26:12.916625  117107 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join k8s-vip:16443 --token vqqtvv.we08sbuxqjk63uk3     --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d

有效时间
该 token 的有效时间为 2 个小时，2小时内，您可以使用此 token 初始化任意数量的 worker 节点。

15 添加node节点

# 只在 worker 节点执行
# 替换 x.x.x.x 为 master 节点的内网 IP
export MASTER_IP=192.168.109.150
# 替换 apiserver.demo 为初始化 master 节点时所使用的 APISERVER_NAME
export APISERVER_NAME=k8s-vip
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

获取join命令参数
kubeadm token create --print-join-command

执行得到的命令


检查初始化结果
在 master 节点上执行
# 只在 master 节点执行
kubectl get nodes -o wide

16 移除node节点

WARNING
正常情况下，您无需移除 worker 节点

在准备移除的 worker 节点上执行
kubeadm reset

    
在第一个 master 节点 demo-master-a-1 上执行
kubectl delete node node1


将 node1 替换为要移除的 worker 节点的名字
worker 节点的名字可以通过在第一个 master 节点 node1 上执行 kubectl get nodes 命令获得

17 helm安装

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

查看是否安装成功
$ helm version

17.1 helm常用命令

添加仓库
helm repo add stable http://mirror.azure.cn/kubernetes/charts/
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

刷新
helm repo update
查看列表
helm repo list
删除仓库
helm repo remove aliyun

查询chart
helm search repo prometheus

helm install stable/prometheus

helm list 
helm status prometheus

把参数行成文件
helm show values stable/prometheus > prometheus.yaml

安装
helm install prometheus --namespace monitor -f prometheus.yaml stable/prometheus

18 测试pod

# 运行完就删除掉
kubectl run busybox --rm -it --image=busybox /bin/sh

根据service名称访问pod
[root@ken ~]# kubectl run busybox --rm -it --image=busybox /bin/sh
/ # wget gateway-junban-gateway.jb-dev:15000
wget: bad address 'httpd2-svc:8080'
/ # wget gateway-junban-gateway.jb-dev:15000/swagger-ui.html
Connecting to gateway-junban-gateway.jb-dev:15000 (10.1.52.115:15000)
saving to 'swagger-ui.html'
swagger-ui.html      100% |************|  3318  0:00:00 ETA
'swagger-ui.html' saved

访问格式 [service名称].[命名空间名称]:[端口]
命名空间是default 可以不用写
因为属于不同的 namespace，必须使用 gateway-junban-gateway.jb-dev 才能访问到。

19 k8s volume 挂载踩坑

https://www.jianshu.com/p/f60bb78bd90b