k8s多节点部署

前提

多节点在单节点的基础上进行
资源规划

名称主机名IP地址
ks8-master01master20.0.0.10/24
ks8-master02master0220.0.0.40/24
node01node0120.0.0.20/24
node02node0220.0.0.30/24
nginx-masternginx-master20.0.0.40/24
nginx-backupnginx-backup20.0.0.50/24
VIP地址20.0.0.100/24

master02部署

1、将master01的配置信息输出到master02上

[root@master kubeconfig]# scp -r /opt/kubernetes/ root@20.0.0.40:/opt/
The authenticity of host '20.0.0.40 (20.0.0.40)' can't be established.
ECDSA key fingerprint is SHA256:3YC4ehaBnUbIovfc6ha74KIHn4KPTl4hXHvXgZZX4J0.
ECDSA key fingerprint is MD5:f9:36:e7:e0:99:70:ba:c0:ba:46:85:58:6f:60:2c:70.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.40' (ECDSA) to the list of known hosts.
root@20.0.0.40's password: 
token.csv                                                            100%   84   102.3KB/s   00:00    
kube-apiserver                                                       100%  909     1.3MB/s   00:00    
kube-scheduler                                                       100%   94   161.6KB/s   00:00    
kube-controller-manager                                              100%  483   842.0KB/s   00:00    
kube-apiserver                                                       100%  184MB  93.6MB/s   00:01    
kubectl                                                              100%   55MB  94.6MB/s   00:00    
kube-controller-manager                                              100%  155MB  97.4MB/s   00:01    
kube-scheduler                                                       100%   55MB  97.3MB/s   00:00    
admin-key.pem                                                        100% 1679     1.9MB/s   00:00    
admin.pem                                                            100% 1399     1.9MB/s   00:00    
ca-key.pem                                                           100% 1675     3.7MB/s   00:00    
ca.pem                                                               100% 1359     3.0MB/s   00:00    
kube-proxy-key.pem                                                   100% 1679     3.9MB/s   00:00    
kube-proxy.pem                                                       100% 1403     3.2MB/s   00:00    
server-key.pem                                                       100% 1675     3.7MB/s   00:00    
server.pem                                                           100% 1643     3.9MB/s   00:00    

2、master01复制启动文件到master02中

[root@master kubeconfig]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@20.0.0.40:/usr/lib/systemd/system/
root@20.0.0.40's password: 
kube-apiserver.service                                               100%  282   509.4KB/s   00:00    
kube-controller-manager.service                                      100%  317   706.6KB/s   00:00    
kube-scheduler.service                                               100%  281   616.3KB/s   00:00    

3、master02修改kube-apiserver IP地址指向自己

[root@master02 ~]# vim /opt/kubernetes/cfg/kube-apiserver
--bind-address=20.0.0.40 \		#修改为自己主机IP地址

--advertise-address=20.0.0.40 \		#修改为自己主机IP地址

4、将master01的etcd证书复制到master02上面

[root@master kubeconfig]# scp -r /opt/etcd/ root@20.0.0.40:/opt/
root@20.0.0.40's password: 
etcd                                                                 100%  481    88.9KB/s   00:00    
etcd                                                                 100%   18MB  88.8MB/s   00:00    
etcdctl                                                              100%   15MB  92.6MB/s   00:00    
ca-key.pem                                                           100% 1679   899.3KB/s   00:00    
ca.pem                                                               100% 1265   490.2KB/s   00:00    
server-key.pem                                                       100% 1679     2.2MB/s   00:00    
server.pem                                                           100% 1338     1.2MB/s   00:00    

5、启动apiserver、scheduler、controller-manager

[root@master02 ~]# systemctl start kube-apiserver.service 
[root@master02 ~]# systemctl enable kube-apiserver.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.

root@master02 ~]# systemctl start kube-scheduler.service 
[root@master02 ~]# systemctl enable kube-scheduler.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.

[root@master02 ~]# systemctl start kube-controller-manager.service 
[root@master02 ~]# systemctl enable kube-controller-manager.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

6、添加环境变量

[root@master02 ~]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/	#增加

[root@master02 ~]# source /etc/profile	#加载环境变量

7、查看群集状态

[root@master02 ~]# kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
20.0.0.20   Ready    <none>   64m   v1.12.3
20.0.0.30   Ready    <none>   14m   v1.12.3

前端负载nginx搭建

nginx-master配置
1、设置nginx源

[root@nginx-master ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0

2、安装nginx

[root@nginx-master ~]# yum -y install nginx

3、修改nginx配置文件代理后端master

[root@nginx-master ~]# vim /etc/nginx/nginx.conf
events {
    worker_connections  1024;
}
增加下面内容
stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 20.0.0.10:6443;
        server 20.0.0.40:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }

4、检查语法

[root@nginx-master ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

5、启动nginx

[root@nginx-master ~]# systemctl start nginx.service 
[root@nginx-master ~]# systemctl enable nginx.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.

6、安装keepalived

[root@nginx-master ~]# yum -y install keepalived

7、修改keepalived配置文件

[root@nginx-master ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/etc/nginx/nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
    priority 100    # 优先级,备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间,默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        20.0.0.100/24
    }
    track_script {
        check_nginx
    }
}

8、创建nginx健康检查脚本

[root@nginx-master ~]# vim /etc/nginx/nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

9、给脚本加执行权限

[root@nginx-master ~]# chmod +x /etc/nginx/nginx.sh

10、启动keepalived

[root@nginx-master ~]# systemctl start keepalived.service
[root@nginx-master ~]# systemctl enable keepalived.service

nginx-backup配置
1、设置nginx源

[root@nginx-backup ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0

2、安装nginx

[root@nginx-backup ~]# yum -y install nginx

3、修改nginx配置文件代理后端master

[root@nginx-master ~]# vim /etc/nginx/nginx.conf
events {
    worker_connections  1024;
}
增加下面内容
stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 20.0.0.10:6443;
        server 20.0.0.40:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }

4、检查语法

[root@nginx-master ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

5、启动nginx

[root@nginx-master ~]# systemctl start nginx.service 
[root@nginx-master ~]# systemctl enable nginx.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.

6、安装keepalived

[root@nginx-backup ~]# yum -y install keepalived

7、修改keepalived配置文件

[root@nginx-backup ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/etc/nginx/nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
    priority 90    # 优先级,备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间,默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        20.0.0.100/24
    }
	track_script {
        check_nginx
    }
}

8、创建nginx健康检查脚本

[root@nginx-backup ~]# vim /etc/nginx/nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

9、给脚本加执行权限

[root@nginx-backup ~]# chmod +x /etc/nginx/nginx.sh

10、启动keepalived

[root@nginx-backup ~]# systemctl start keepalived.service
[root@nginx-backup ~]# systemctl enable keepalived.service

测试

测试漂移地址

1、查看IP地址
[root@nginx-master ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:fe:a2:12 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.50/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.100/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::edac:d705:9b30:5197/64 scope link 
       valid_lft forever preferred_lft forever

[root@nginx-backup ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e9:43:b6 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.60/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d283:86ee:ba5e:dcf8/64 scope link 
       valid_lft forever preferred_lft forever
	   
2、关闭主节点的nginx看VIP地址是否漂移
[root@nginx-master ~]# pkill nginx
[root@nginx-master ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:fe:a2:12 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.50/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::edac:d705:9b30:5197/64 scope link 
       valid_lft forever preferred_lft forever

[root@nginx-backup ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e9:43:b6 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.60/24 brd 20.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.100/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d283:86ee:ba5e:dcf8/64 scope link 
       valid_lft forever preferred_lft forever
	   
#主节点nginx恢复正常后VIP地址会自动漂移回来,先启动nginx在启动keepalived

node IP指向VIP地址

所有node节点都进行如下设置
1、修改地址为VIP地址

vim /opt/kubernetes/cfg/bootstrap.kubeconfig
server: https://20.0.0.100:6443		#修改为VIP地址

vim /opt/kubernetes/cfg/kubelet.kubeconfig
server: https://20.0.0.100:6443		#修改为VIP地址

vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
server: https://20.0.0.100:6443		#修改为VIP地址

2、重启

systemctl restart kubelet.service
systemctl restart kube-proxy.service

3、查看是否修改成功

 grep 100 *
 
bootstrap.kubeconfig:    server: https://20.0.0.100:6443
kubelet.kubeconfig:    server: https://20.0.0.100:6443
kube-proxy.kubeconfig:    server: https://20.0.0.100:6443

测试方法负载

[root@master k8s-cert]# kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
20.0.0.20   Ready    <none>   5h5m    v1.12.3
20.0.0.30   Ready    <none>   4h15m   v1.12.3

[root@nginx-backup ~]# tail -f /var/log/nginx/k8s-access.log 
20.0.0.30 20.0.0.10:6443 - [21/Jan/2021:15:25:32 +0800] 200 1115
20.0.0.30 20.0.0.40:6443 - [21/Jan/2021:15:25:32 +0800] 200 1115
20.0.0.20 20.0.0.10:6443 - [21/Jan/2021:15:33:55 +0800] 200 1115
20.0.0.20 20.0.0.40:6443 - [21/Jan/2021:15:33:55 +0800] 200 1114

1、创建pod测试

[root@master k8s-cert]# kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created

2、查看状态

[root@master k8s-cert]# kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
nginx-dbddb74b8-mdfnc   0/1     ContainerCreating   0          22s

[root@master k8s-cert]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-mdfnc   1/1     Running   0          50s

3、查看pod日志

[root@master k8s-cert]# kubectl logs nginx-dbddb74b8-mdfnc
#默认匿名用户没有权限
[root@master k8s-cert]# kubectl logs nginx-dbddb74b8-mdfnc
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-mdfnc)
#设置权限
[root@master k8s-cert]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
#查看pod创建在哪个节点创建
[root@master k8s-cert]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE
nginx-dbddb74b8-mdfnc   1/1     Running   0          4m15s   172.17.75.3   20.0.0.30   <none>

去node2上查看
[root@node2 ~]# docker ps -a
CONTAINER ID   IMAGE                                                                 COMMAND                  CREATED         STATUS         PORTS     NAMES
0040331bf8c1   nginx                                                                 "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes             k8s_nginx_nginx-dbddb74b8-mdfnc_default_c9b8648e-5bbb-11eb-b8b6-000c29d29307_0
66f042492ba0   registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_nginx-dbddb74b8-mdfnc_default_c9b8648e-5bbb-11eb-b8b6-000c29d29307_0
43a46bd8868f   centos:7
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值