K8s 多节点部署

最新推荐文章于 2024-10-06 23:48:52 发布

Ora.

最新推荐文章于 2024-10-06 23:48:52 发布

阅读量377

点赞数

分类专栏： Kubernetes 文章标签： nginx docker linux centos kubernetes

本文链接：https://blog.csdn.net/Ora_G/article/details/108944938

版权

Kubernetes 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

基于上次的单节点部署开始制作，master02的IP地址实际上在上一篇有所提及，欢迎查看上一篇博客

文章目录

来吧！展示！！

来吧！展示！！

master复制文件给master02

复制配置文件以及启动脚本

[root@localhost ~]# scp -r /opt/kubernetes/ root@20.0.0.6:/opt/
[root@localhost ~]# cd /usr/lib/systemd/system
[root@localhost system]# scp kube-apiserver.service kube-controller-manager.service kube-scheduler.service root@20.0.0.6:/usr/lib/systemd/system/
root@20.0.0.6's password: 
kube-apiserver.service                 100%  282   258.3KB/s   00:00    
kube-controller-manager.service        100%  317   343.0KB/s   00:00    
kube-scheduler.service                 100%  281   376.5KB/s   00:00

拷贝etcd的文件给master02

[root@localhost /]# scp -r /opt/etcd/ root@20.0.0.6:/opt/
root@20.0.0.6's password:

mater02进行修改

修改配置文件kube-apiserver中的IP地址

[root@localhost /]# vim /opt/kubernetes/cfg/kube-apiserver
--bind-address=20.0.0.6 \	##改这行
--secure-port=6443 \
--advertise-address=20.0.0.6 \	##还有这行
启动三个服务
[root@localhost ~]# systemctl start kube-apiserver.service 
[root@localhost ~]# systemctl start kube-controller-manager.service
[root@localhost ~]# systemctl start kube-scheduler.service 

增加环境变量
[root@localhost ~]# vim /etc/profile
##增加在末行
export PATH=$PATH:/opt/kubernetes/bin/
[root@localhost ~]# source /etc/profile
[root@localhost ~]# kubectl get node
NAME       STATUS   ROLES    AGE     VERSION
20.0.0.4   Ready    <none>   6d10h   v1.12.3
20.0.0.5   Ready    <none>   6d10h   v1.12.3

Nginx代理设置

两台ngxin，在上一篇博客里也有写道
在这里插入图片描述
两台nginx服务器操作一致

[root@7CentOS ~]# systemctl stop firewalld
[root@7CentOS ~]# setenforce 0
[root@7CentOS ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@7CentOS ~]# yum -y install nginx
[root@7CentOS ~]# vim /etc/nginx/nginx.conf
events {
    worker_connections  1024;
}
##在下面添加
stream {
      log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';	 ##日期格式
    access_log  /var/log/nginx/k8s-access.log  main;	##日志记载位置
    
    upstream k8s-apiserver {	##负载均衡IP地址
        server 20.0.0.3:6443;
        server 20.0.0.6:6443;
    }
    server {    
                listen 6443;      ##nginx服务的端口为6443
                proxy_pass k8s-apiserver;
    }
    }
[root@7CentOS ~]# systemctl start nginx
[root@7CentOS ~]# netstat -ntap |grep 6443
tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      13900/nginx: master

Keepalived部署

事先上传了一个keepalived.conf
keepalived master

[root@7CentOS /]# yum -y install keepalived
[root@7CentOS ~]# cp keepalived.conf /etc/keepalived/keepalived.conf 
cp：是否覆盖"/etc/keepalived/keepalived.conf"？ yes
[root@7CentOS ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/usr/local/nginx/check_nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 100    # 优先级，备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        20.0.0.150/24
    } 
    track_script {
        check_nginx
    }
}
[root@7CentOS ~]# vim /etc/nginx/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@7CentOS ~]# chmod +x /etc/nginx/check_nginx.sh
[root@7CentOS ~]# systemctl start keepalived
[root@7CentOS ~]# ip a
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:93:2c:0b brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.7/24 brd 20.0.0.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.150/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe93:2c0b/64 scope link 
       valid_lft forever preferred_lft forever

slave部署

[root@localhost nginx]# yum -y install keepalived

master复制文件给slave
[root@7CentOS ~]# scp -r /etc/keepalived/ root@20.0.0.8:/etc/
root@20.0.0.8's password: 
keepalived.conf                        100%  778     1.0MB/s   00:00 
[root@localhost nginx]# vim /etc/keepalived/keepalived.conf		##进行修改
vrrp_instance VI_1 {
    state BACKUP	##这行
    interface ens33
    virtual_router_id 51
    priority 90		##这行

[root@localhost nginx]# vim /etc/nginx/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@localhost nginx]# systemctl start keepalived
注意：slave的ip a s不应该有浮动ip

验证浮动IP
杀掉master的nginx

[root@7CentOS ~]# pkill nginx

slave 查看

[root@localhost nginx]# ip a
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f1:9c:a7 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.8/24 brd 20.0.0.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 20.0.0.150/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::68ac:e5ec:8230:c3b8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

master启动nginx，再启动keepalived

[root@7CentOS ~]# systemctl start nginx
[root@7CentOS ~]# systemctl start keepalived

slave再查看

[root@localhost nginx]# ip a
##浮动IP消失
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f1:9c:a7 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.8/24 brd 20.0.0.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::68ac:e5ec:8230:c3b8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

node节点修改配置文件

[root@localhost ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
    server: https://20.0.0.150:6443
[root@localhost ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig 
    server: https://20.0.0.150:6443
[root@localhost ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig 
    server: https://20.0.0.150:6443
[root@localhost ~]# systemctl restart kubelet.service 
[root@localhost ~]# systemctl restart kube-proxy.service 
[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# grep 150 *
bootstrap.kubeconfig:    server: https://20.0.0.150:6443
kubelet.kubeconfig:    server: https://20.0.0.150:6443
kube-proxy.kubeconfig:    server: https://20.0.0.150:6443

在nginx1上检查日志

[root@7CentOS ~]# tail /var/log/nginx/k8s-access.log 
20.0.0.4 20.0.0.3:6443 - [07/Oct/2020:00:39:06 +0800] 200 1115
20.0.0.4 20.0.0.6:6443 - [07/Oct/2020:00:39:06 +0800] 200 1114
20.0.0.5 20.0.0.3:6443 - [07/Oct/2020:00:43:27 +0800] 200 1113
20.0.0.5 20.0.0.3:6443 - [07/Oct/2020:00:43:27 +0800] 200 1115

创建pod测试

[root@localhost /]# kubectl run nginx --image=nginx
[root@localhost /]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-zxpk4   1/1     Running   0          2m43s

node1上能看到，node2上则没有
在这里插入图片描述
此时访问日志会报错

[root@localhost /]# kubectl logs nginx-dbddb74b8-zxpk4
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-zxpk4)

使用下面的授权，再查看就成功了

[root@localhost /]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
[root@localhost /]# kubectl logs nginx-dbddb74b8-zxpk4
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

查看网页信息

查看pod的地址
[root@localhost /]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE
nginx-dbddb74b8-zxpk4   1/1     Running   0          7m12s   172.17.13.2   20.0.0.4   <none>

去node1查看网页

[root@localhost cfg]# curl 172.17.13.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

此时在master上也能看到日志信息了

[root@localhost /]# kubectl logs nginx-dbddb74b8-zxpk4
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.13.1 - - [06/Oct/2020:16:53:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"