kubernetes 1.18.8 高可用安装
1 集群规划
| 角色 | ip地址 |
|---|---|
| k8s-vip | 192.168.109.150 |
| master1 | 192.168.109.151 |
| master2 | 192.168.109.152 |
| master3 | 192.168.109.153 |
| node1 | 192.168.109.154 |
2 安装要求
在开始之前，部署Kubernetes集群机器需要满足以下几个条件：
- 一台或多台机器，操作系统 CentOS7.x-86_x64
- 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
- 可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
- 禁止swap分区
3 准备环境
关闭防火墙
systemctl stop firewalld.service && systemctl disable firewalld.service
设置 SELinux 为 disabled 模式
setenforce 0 && sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
禁用交换分区
swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
修改每个系统的主机名 永久修改
hostnamectl set-hostname xxx
#将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system # 生效
安装ntp:
yum install -y ntp
ntpdate time.windows.com && hwclock -w 同步时间且写入硬件
在所有的节点添加hosts
cat >> /etc/hosts << EOF
192.168.109.150 k8s-vip
192.168.109.151 master1
192.168.109.152 master2
192.168.109.153 master3
192.168.109.154 node1
EOF
4 所有master节点部署keepalived
4.1 安装相关包和keepalived
yum install -y conntrack-tools libseccomp libtool-ltdl
yum install -y keepalived
4.2 配置master节点
master1、master2、master3节点配置
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.109.150
}
track_script {
check_haproxy
}
}
EOF
注：
1、virtual_ipaddress 是 VIP的地址
2、interface 网络请注意，ifconfig查看
3、priority 优先级,备lvs要比主lvs稍小
4、state 主MASTER 备BACKUP
4.3 启动和检查
# 启动keepalived && 设置开机启动
systemctl start keepalived.service && systemctl enable keepalived.service
# 查看启动状态
systemctl status keepalived.service
# 重启
systemctl restart keepalived.service
启动后查看master1的网卡信息
# 查询网卡信息 ens33 对应自己的网卡
ip a s ens33
[root@master1 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:28:58:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.109.151/24 brd 192.168.109.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::e119:2c13:fa0a:3953/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@master2 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:3e:cd:de brd ff:ff:ff:ff:ff:ff
inet 192.168.109.152/24 brd 192.168.109.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::e119:2c13:fa0a:3953/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::9f0e:5697:e453:e0b4/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::4faf:a02d:4291:70ae/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
[root@master3 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:4b:35:44 brd ff:ff:ff:ff:ff:ff
inet 192.168.109.153/24 brd 192.168.109.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.109.150/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::e119:2c13:fa0a:3953/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::9f0e:5697:e453:e0b4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
注：3个有不同，在master3节点上有150的地址，其他的都没有，只有master3发生故障之后VIP才会漂移到 master1 或者 master2上
5 所有master节点部署haproxy
5.1 安装
yum install -y haproxy
5.2 配置
注：三台master节点的配置均相同，配置中声明了后端代理的三个master节点服务器，指定了haproxy运行的端口为16443等，因此16443端口为集群的入口
修改自己的IP地址 backend kubernetes-apiserver 下的 server
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master1 192.168.109.151:6443 check
server master2 192.168.109.152:6443 check
server master3 192.168.109.153:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
5.3 启动和检查
三台master都启动
# 开启haproxy && 设置开机启动
systemctl start haproxy && systemctl enable haproxy
# 查看启动状态
systemctl status haproxy
检查端口
netstat -lntup|grep haproxy
6 所有节点安装Docker/kubeadm/kubelet
Kubernetes默认CRI（容器运行时）为Docker，因此先安装Docker。
6.1 安装docker
卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
第一步
yum install -y yum-utils device-mapper-persistent-data lvm2
第二步
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
第三部
推荐19版本
yum install docker-ce-19.03.9-3.el7
镜像加速
data-root 是 存储位置
mkdir -p /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
"registry-mirrors": ["xxxxx"],
"exec-opts": ["native.cgroupdriver=systemd"],
"storage-driver": "overlay2",
"storage-opts": ["overlay2.override_kernel_check=true"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m",
"max-file": "3"
},
"insecure-registries":["127.0.0.1"],
"data-root":"/home/docker-data"
}
EOF
# 修改docker.service文件，使用-g参数指定存储位置
vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --graph /home/docker-data
开机自动启动
systemctl start docker && systemctl enable docker
registry-mirrors=对应阿里加速地址
6.2 添加阿里云YUM软件源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
6.3 安装kubeadm，kubelet和kubectl
yum install -y kubelet-1.18.8 kubeadm-1.18.8 kubectl-1.18.8
systemctl enable kubelet
7 部署Kubernetes Master
7.1 创建kubeadm配置文件
看上面的标题 4.3 文档
在具有vip的master上操作，这里为master3
以下操作是master3上
$ mkdir /usr/local/kubernetes/manifests -p
$ cd /usr/local/kubernetes/manifests/
$ vi kubeadm-config.yaml
内容如下
apiServer:
certSANs:
- master1
- master2
- master3
- k8s-vip
- 192.168.109.150
- 192.168.109.151
- 192.168.109.152
- 192.168.109.153
- 127.0.0.1
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "k8s-vip:16443"
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.8
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.1.0.0/16
scheduler: {}
注：certSANs说明，是master的所有节点ip和hostname，在加上VIP的hostname和VIP的ip地址,在加上 127.0.0.1
7.2 在master3节点执行
cd /usr/local/kubernetes/manifests/
# 查看所需镜像列表
kubeadm config images list --config kubeadm-config.yaml
# 拉取镜像
kubeadm config images pull --config kubeadm-config.yaml
# kubeadm初始化
kubeadm init --config kubeadm-config.yaml
按照提示配置环境变量，使用kubectl工具：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes
$ kubectl get pods -n kube-system
注：kubectl get nodes 会出现notReady ,这是正常的，因为没安装网络插件
按照提示保存以下内容，一会要使用：
kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
--discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
--control-plane
kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
--discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d
查看集群状态
kubectl get cs
kubectl get pods -n kube-system
8 安装集群网络
8.1 安装 flannel（弃用）
从官方地址获取到flannel的yaml，在master3上执行
cd /usr/local/kubernetes/manifests/
mkdir flannel
cd flannel
wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
安装flannel网络
cd /usr/local/kubernetes/manifests/flannel
kubectl apply -f kube-flannel.yml
检查
kubectl get pods -n kube-system
8.2 安装 calico （推荐）
下载
cd /usr/local/kubernetes/manifests/
mkdir calico
cd calico
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
安装calico网络
cd /usr/local/kubernetes/manifests/calico
kubectl apply -f calico-3.13.1.yaml
检查
kubectl get pods -n kube-system
9 ssh免密登录
在master3执行
ssh-keygen -t rsa
一路回车
$IPs 是master的hostname
ssh-copy-id master1
ssh-copy-id master2
ssh-copy-id $IPs 按照提示输入yes 和root密码
10 master1 和 master2 节点加入集群
10.1 复制密钥
从master3复制密钥及相关文件到master1
ssh root@master1 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master1:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master1:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master1:/etc/kubernetes/pki/etcd
从master3复制密钥及相关文件到master2
ssh root@master2 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@master2:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master2:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@master2:/etc/kubernetes/pki/etcd
10.2 master1 加入集群
kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
--discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
--control-plane
10.3 master2 加入集群
kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
--discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d \
--control-plane
检查状态
kubectl get node
kubectl get pods --all-namespaces
11 加入Kubernetes Node
在node1上执行
向集群添加新节点，执行在kubeadm init输出的kubeadm join命令：
kubeadm join k8s-vip:16443 --token fytj36.nxlv38msqco9t853 \
--discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d
检查状态
kubectl get node
kubectl get pods --all-namespaces
12 测试集群
在Kubernetes集群中创建一个pod，验证是否正常运行：
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
13 测试集群
在Kubernetes集群中创建一个pod，验证是否正常运行：
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
访问地址：http://NodeIP:Port
14 获取join命令参数
kubeadm token create --print-join-command
得到结果
[root@master3 ~]# kubeadm token create --print-join-command
W1015 17:26:12.916625 117107 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join k8s-vip:16443 --token vqqtvv.we08sbuxqjk63uk3 --discovery-token-ca-cert-hash sha256:f6c2a0bcf1bd27c1633e77469e211a4acade487c4aadf6b98b23d116aef5695d
有效时间
该 token 的有效时间为 2 个小时，2小时内，您可以使用此 token 初始化任意数量的 worker 节点。
15 添加node节点
# 只在 worker 节点执行
# 替换 x.x.x.x 为 master 节点的内网 IP
export MASTER_IP=192.168.109.150
# 替换 apiserver.demo 为初始化 master 节点时所使用的 APISERVER_NAME
export APISERVER_NAME=k8s-vip
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
获取join命令参数
kubeadm token create --print-join-command
执行得到的命令
检查初始化结果
在 master 节点上执行
# 只在 master 节点执行
kubectl get nodes -o wide
16 移除node节点
WARNING
正常情况下，您无需移除 worker 节点
在准备移除的 worker 节点上执行
kubeadm reset
在第一个 master 节点 demo-master-a-1 上执行
kubectl delete node node1
将 node1 替换为要移除的 worker 节点的名字
worker 节点的名字可以通过在第一个 master 节点 node1 上执行 kubectl get nodes 命令获得
17 helm安装
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh
查看是否安装成功
$ helm version
17.1 helm常用命令
添加仓库
helm repo add stable http://mirror.azure.cn/kubernetes/charts/
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
刷新
helm repo update
查看列表
helm repo list
删除仓库
helm repo remove aliyun
查询chart
helm search repo prometheus
helm install stable/prometheus
helm list
helm status prometheus
把参数行成文件
helm show values stable/prometheus > prometheus.yaml
安装
helm install prometheus --namespace monitor -f prometheus.yaml stable/prometheus
18 测试pod
# 运行完就删除掉
kubectl run busybox --rm -it --image=busybox /bin/sh
根据service名称访问pod
[root@ken ~]# kubectl run busybox --rm -it --image=busybox /bin/sh
/ # wget gateway-junban-gateway.jb-dev:15000
wget: bad address 'httpd2-svc:8080'
/ # wget gateway-junban-gateway.jb-dev:15000/swagger-ui.html
Connecting to gateway-junban-gateway.jb-dev:15000 (10.1.52.115:15000)
saving to 'swagger-ui.html'
swagger-ui.html 100% |************| 3318 0:00:00 ETA
'swagger-ui.html' saved
访问格式 [service名称].[命名空间名称]:[端口]
命名空间是default 可以不用写
因为属于不同的 namespace，必须使用 gateway-junban-gateway.jb-dev 才能访问到。
1145
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
