一、多Master集群部署
1、拷贝文件
从Master01上拷贝相关文件:
服务文件:
修改配置文件kube-apiserver:10.11.97.191为master02的IP
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.11.97.191:2379,https://10.11.97.192:2379,https://10.11.97.71:2379 \
--bind-address=10.11.97.191 \
--secure-port=6443 \
--advertise-address=10.11.97.191 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
2、启动服务
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
服务启动成功。
3、apiserver高可用配置
通过Nginx进行4层负载均衡:
stream {
log_format main "$remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.11.97.191:6443;
server 10.11.97.192:6443;
}
server {
listen 0.0.0.0:6443;
proxy_pass k8s-apiserver;
}
}
这样,就使用Nginx的https://10.11.97.191:6443替代了apiserver的地址了。
修改Node节点连接apiserver的地址:
bootstrap.kubeconfig
kubelet.kubeconfig
kube-proxy.kubeconfig
修改地方:apiserverIP统一修改为
server: https://10.11.97.191:6443
重启Node节点的服务。
4、LB高可用
使用keepalived对LB进行高可用配置。
keepalived的Master节点:
IP地址:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
xuequn@kingsoft.com
}
notification_email_from zabbix@kingsoft.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx_web {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_NGINX_WEB {
state BACKUP
interface eth0
virtual_router_id 60
priority 190
advert_int 1
authentication {
auth_type PASS
auth_pass ZabbixServer
}
track_script {
chk_nginx_web
}
virtual_ipaddress {
10.11.97.187
}
}
keepalived的Slave节点:
IP地址:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
xuequn@kingsoft.com
}
notification_email_from zabbix@kingsoft.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx_web {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_NGINX_WEB {
state MASTER
interface eth0
virtual_router_id 60
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass ZabbixServer
}
track_script {
#chk_nginx_web
}
virtual_ipaddress {
10.11.97.187
}
}
健康检查脚本:
cat /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
启动keepalived集群后,vip绑定在master上:
至此,k8s集群多Master高可用,LB高可用、Etcd高可用都已完成,其中任何一个节点挂掉都不会影响正常业务。