一 环境部署
主机名 | IP地址 | 部署的服务 |
---|---|---|
master01 | 20.0.0.101 | apiserver、scheduler、controller-manager、etcd |
master02 | 20.0.0.104 | apiserver、scheduler、controller-manager、etcd |
node01 | 20.0.0.102 | kubelet、kube-proxy、docker、flannel、etcd |
node02 | 20.0.0.103 | kubelet、kube-proxy、docker、flannel、etcd |
lb01 | 20.0.0.105 | nginx、keepalived |
lb02 | 20.0.0.106 | nginx、keepalived |
VIP | 20.0.0.100 |
二 实验操作步骤
在单节点实验基础上操作(详情见博客)
2.1 master02部署
##在master01上操作##
1. 复制kubernetes目录到master02
[root@localhost k8s]# scp -r /opt/kubernetes/ root@20.0.0.104:/opt/
The authenticity of host '20.0.0.104 (20.0.0.104)' can't be established.
ECDSA key fingerprint is SHA256:EM5og8Q/El6tYfMXxgXbd5ccgh2knP0HkFALiBNwd30.
ECDSA key fingerprint is MD5:4d:aa:26:2b:3c:c2:bd:12:92:1e:57:60:6d:72:26:1f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.104' (ECDSA) to the list of known hosts.
root@20.0.0.104's password:
token.csv 100% 84 140.6KB/s 00:00
kube-apiserver 100% 914 889.0KB/s 00:00
kube-scheduler 100% 85 118.4KB/s 00:00
kube-controller-manager 100% 483 849.6KB/s 00:00
kube-apiserver 100% 184MB 126.8MB/s 00:01
kubectl 100% 55MB 114.9MB/s 00:00
kube-controller-manager 100% 155MB 135.0MB/s 00:01
kube-scheduler 100% 55MB 135.0MB/s 00:00
ca-key.pem 100% 1675 1.2MB/s 00:00
ca.pem 100% 1359 624.2KB/s 00:00
server-key.pem 100% 1675 1.9MB/s 00:00
server.pem 100% 1643 1.8MB/s 00:00
2. 复制master01中的三个组件启动脚本kube-apiserver.service kube-controller-maneger.service kube-scheduler.service
[root@localhost k8s]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@20.0.0.104:/usr/lib/systemd/system
root@20.0.0.104's password:
kube-apiserver.service 100% 282 245.9KB/s 00:00
kube-controller-manager.service 100% 317 286.7KB/s 00:00
kube-scheduler.service 100% 281 378.9KB/s 00:00
##在master02上操作##
1. 查看复制的文件
[root@localhost ~]# ls /opt/kubernetes/bin/
kube-apiserver kube-controller-manager kubectl kube-scheduler
[root@localhost ~]# ls /opt/kubernetes/cfg/
kube-apiserver kube-controller-manager kube-scheduler token.csv
[root@localhost ~]# ls /opt/kubernetes/ssl/
ca-key.pem ca.pem server-key.pem server.pem
[root@localhost ~]# ls /usr/lib/systemd/system | grep "kube"
kube-apiserver.service
kube-controller-manager.service
kube-scheduler.service
2. 修改配置文件
[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# vim kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://20.0.0.101:2379,https://20.0.0.102:2379,https://20.0.0.103:2379 \
--bind-address=20.0.0.104 \ ##改为本地IP
--secure-port=6443 \
--advertise-address=20.0.0.104 \ ####改为本地IP
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
##在master01上操作##
1. 拷贝master01上已有的etcd证书给master02使用
//特别注意:master02一定要有etcd证书,否则apiserver服务无法启动
[root@localhost k8s]# scp -r /opt/etcd/ root@20.0.0.104:/opt/
root@20.0.0.104's password:
etcd 100% 488 297.5KB/s 00:00
etcd 100% 18MB 101.2MB/s 00:00
etcdctl 100% 15MB 119.9MB/s 00:00
ca-key.pem 100% 1679 1.5MB/s 00:00
ca.pem 100% 1265 607.2KB/s 00:00
server-key.pem 100% 1675 1.9MB/s 00:00
server.pem 100% 1338 643.5KB/s 00:00
##在master02上操作##
1. 启动master三个组件服务
[root@localhost cfg]# systemctl start kube-apiserver.service
[root@localhost cfg]# systemctl enable kube-apiserver.service
[root@localhost cfg]# systemctl start kube-controller-manager.service
[root@localhost cfg]# systemctl enable kube-controller-manager.service
[root@localhost cfg]# systemctl start kube-scheduler.service
[root@localhost cfg]# systemctl enable kube-scheduler.service
[root@localhost cfg]# systemctl status kube-apiserver.service
[root@localhost cfg]# systemctl status kube-controller-manager.service
[root@localhost cfg]# systemctl status kube-scheduler.service
2. 增加环境变量
[root@localhost cfg]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/ ##末尾添加
[root@localhost cfg]# source /etc/profile
3. 查看nodes节点状态信息
[root@localhost cfg]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
20.0.0.102 Ready <none> 55m v1.12.3
20.0.0.103 Ready <none> 48m v1.12.3
2.2 nginx负载均衡集群部署
##在lb01,lb02上操作##
以下以lb01为例
1. 设置主机名,关闭防火墙,核心防护
[root@localhost ~]# hostnamectl set-hostname lb01
[root@localhost ~]# su
[root@lb01 ~]# iptables -F
[root@lb01 ~]# setenforce 0
setenforce: SELinux is disabled
2. 把nginx.sh和keepalived.conf脚本拷贝到家目录
[root@lb01 ~]# rz -E
rz waiting to receive.
[root@lb01 ~]# ls
anaconda-ks.cfg keepalived.conf 公共 视频 文档 音乐
initial-setup-ks.cfg nginx.sh 模板 图片 下载 桌面
3. 编辑nginx线网源地址,下载安装nginx
[root@lb01 ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@lb01 ~]# yum -y install nginx
4. 添加四层转发,开启nginx服务
[root@lb01 ~]# vim /etc/nginx/nginx.conf
events {
worker_connections 1024;
}
stream { ##添加四层转发
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 20.0.0.101:6443;
server 20.0.0.104:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
[root@lb01 ~]# systemctl start nginx
5. 安装keepalived
[root@lb01 ~]# yum -y install keepalived
[root@lb01 ~]# cp keepalived.conf /etc/keepalived/keepalived.conf
cp:是否覆盖"/etc/keepalived/keepalived.conf"? yes
[root@lb01 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx { ##check_nginx名称可以修改
script "/etc/nginx/check_nginx.sh"
vrrp_instance VI_1 {
state MASTER
interface ens33 ##改为ens33
virtual_router_id 51 ##VRRP 路由 ID实例,每个实例是唯一的
priority 100 ##优先级,备服务器设置 90
advert_int 1 ##指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
20.0.0.100/24 ##vip地址
}
track_script {
check_nginx ##与上面的名称一致
}
}
-----------------------lb02配置-------------------
! Configuration File for keepalived
global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/nginx/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
20.0.0.100/24
}
track_script {
check_nginx
}
}
6. 编辑check_nginx.sh脚本
[root@lb01 ~]# vim /etc/nginx/check_nginx.sh
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
[root@lb01 ~]# chmod +x /etc/nginx/check_nginx.sh
[root@lb01 ~]# systemctl start keepalived.service
7. 查看lb01,lb02地址信息
[root@lb01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:bf:f4:82 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.105/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet 20.0.0.100/24 scope global secondary ens33 ##vip地址
valid_lft forever preferred_lft forever
inet6 fe80::324f:a4dd:6fe:7a09/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
--------------------------lb02----------------------------------
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::bff5:d805:be6c:19e4/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
8. 验证地址漂移
##在lb01上操作##
1. 杀死nginx进程
[root@lb01 ~]# pkill nginx
##在lb02上操作##
1. 查看vip地址漂移
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet 20.0.0.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::bff5:d805:be6c:19e4/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
9. 恢复lb01为master
##在lb01上操作##
1. 先启动nginx,再启动keepalived
[root@lb01 ~]# systemctl start nginx
[root@lb01 ~]# systemctl start keepalived.service
2. 查看vip地址漂移
[root@lb01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:bf:f4:82 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.105/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet 20.0.0.100/24 scope global secondary ens33 ##恢复成功
valid_lft forever preferred_lft forever
inet6 fe80::324f:a4dd:6fe:7a09/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:7b:a1:c1 brd ff:ff:ff:ff:ff:ff
##在lb02上操作##
1. 查看vip是否漂移
[root@lb02 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:83:e9:37 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.106/24 brd 20.0.0.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::bff5:d805:be6c:19e4/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:c1:8b:92 brd ff:ff:ff:ff:ff:ff
10. 修改node节点配置文件,统一VIP(bootstrap.kubeconfig,kubelet.kubeconfig)
##以下以node01为例
1. 修改为VIP地址
[root@node01 ~]# cd /opt/kubernetes/cfg/
[root@node01 cfg]# ls
bootstrap.kubeconfig kubelet kubelet.kubeconfig kube-proxy.kubeconfig
flanneld kubelet.config kube-proxy
[root@node01 cfg]# vim bootstrap.kubeconfig
[root@node01 cfg]# vim kubelet.kubeconfig
[root@node01 cfg]# vim kube-proxy.kubeconfig
##统统修改为VIP
server: https://20.0.0.100:6443
2. 自检
[root@node01 cfg]# grep 100 *
bootstrap.kubeconfig: server: https://20.0.0.100:6443
kubelet.kubeconfig: server: https://20.0.0.100:6443
kube-proxy.kubeconfig: server: https://20.0.0.100:6443
3. 在lb01上查看nginx的k8s日志
使用lb虚拟机自带的火狐浏览器访问20.0.0.106-----------welcome nginx!
[root@lb01 ~]# tail /var/log/nginx/k8s-access.log
20.0.0.1 20.0.0.101:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.104:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.101:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
20.0.0.1 20.0.0.104:6443 - [24/Nov/2020:17:39:14 +0800] 200 170
##在master01上操作##
1. 测试创建pod
[root@master01 ~]# kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created
2. 查看状态
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-jj54s 0/1 ContainerCreating 0 24s ##正在创建中
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-jj54s 1/1 Running 0 79s ##创建完成,运行中
##注意日志问题
[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-jj54s)
解决方法:
[root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s ##成功查看日志
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
3. 查看网络
[root@master01 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-jj54s 1/1 Running 0 5m58s 172.17.86.2 20.0.0.102 <none>
4. 在对应网段的node节点上操作直接访问
打开20.0.0.102上的火狐浏览器:172.17.86.2------welcome nginx!
5. 回到master01上查看日志
[root@master01 ~]# kubectl logs nginx-dbddb74b8-jj54s
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.86.1 - - [24/Nov/2020:10:36:49 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0" "-"