k8s二进制方式多节点部署

最新推荐文章于 2023-05-04 14:57:34 发布

长恋离亭

最新推荐文章于 2023-05-04 14:57:34 发布

阅读量293

点赞数

分类专栏： K8s 文章标签： docker kubernetes centos linux

本文链接：https://blog.csdn.net/weixin_47153988/article/details/108880976

版权

K8s 专栏收录该内容

13 篇文章 1 订阅

订阅专栏

文章目录

一、拓扑图和环境介绍
二、多节点部署

一、拓扑图和环境介绍

在这里插入图片描述
各主机ip地址分配

主机名	IP地址	部署的服务
master	192.168.179.121/24	apiserver、scheduler、controller-manager、etcd
master02	192.168.179.124/24	apiserver、scheduler、controller-manager
node01	192.168.179.122/24	kubelet、kube-proxy、docker、flannel、etcd
node02	192.168.179.123/24	kubelet、kube-proxy、docker、flannel、etcd
nginx01	192.168.179.125/24	nginx、keepalived
nginx02	192.168.179.126/24	nginx、keepalived
VIP	192.168.179.100/24

二、多节点部署

2.1 单节点部署

单节点部署可以看我的上一篇博客单节点部署k8s集群https://blog.csdn.net/weixin_47153988/article/details/108864050

2.2 master02部署

将master上的kubernetes配置文件、启动脚本、etcd证书复制到master02节点

[root@master ~]# scp -r /opt/kubernetes/ root@192.168.179.124:/opt/
[root@master ~]# scp -r /opt/etcd/ root@192.168.179.124:/opt/
[root@master ~]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.179.124:/usr/lib/systemd/system/

关闭防火墙，关闭核心防护，关闭网络管理功能

[root@master2 ~]# setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config
[root@master2 ~]# systemctl stop NetworkManager && systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

master2上修改apiserver配置文件中的ip地址

[root@master2 ~]# cd /opt/kubernetes/cfg/
[root@master2 cfg]# vim kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.179.121:2379,https://192.168.179.122:2379,https://192.168.179.123:2379 \
--bind-address=192.168.179.124 \
--secure-port=6443 \
--advertise-address=192.168.179.124 \

安装tree命令查看一下证书是否齐全

[root@master2 cfg]# cd ~
[root@master2 ~]# yum install tree -y
[root@master2 ~]# tree /opt/etcd
/opt/etcd
├── bin
│   ├── etcd
│   └── etcdctl
├── cfg
│   └── etcd
└── ssl
    ├── ca-key.pem
    ├── ca.pem
    ├── server-key.pem
    └── server.pem
3 directories, 7 files

启动三个组件

[root@master2 ~]# systemctl start kube-apiserver.service
[root@master2 ~]# systemctl status kube-apiserver.service
[root@master2 ~]# systemctl enable kube-apiserver.service 
[root@master2 ~]# systemctl status kube-apiserver.service
[root@master2 ~]# systemctl enable kube-controller-manager.service
[root@master2 ~]# systemctl start kube-scheduler.service 
[root@master2 ~]# systemctl status kube-scheduler.service
[root@master2 ~]# systemctl enable kube-scheduler.service

添加环境变量并查看节点状态

[root@master2 ~]# echo export PATH=$PATH:/opt/kubernetes/bin >> /etc/profile
[root@master2 ~]# source /etc/profile
[root@master2 ~]# kubectl get node
NAME              STATUS   ROLES    AGE   VERSION
192.168.179.122   Ready    <none>   8h    v1.12.3
192.168.179.123   Ready    <none>   8h    v1.12.3

2.3 nginx负载均衡部署

两台nginx的操作是一样的，我这里展示的nginx01的操作

主机开局优化：关闭防火墙和核心防护，编辑nginx yum源

[root@nginx01 ~]# setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config	
[root@nginx01 ~]# vi /etc/yum.repos.d/nginx.repo 	
[nginx]
name=nginx.repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
enabled=1
gpgcheck=0
[root@nginx01 ~]# yum clean all
[root@nginx01 ~]# yum makecache
[root@nginx01 ~]# yum -y install nginx

编辑nginx主配置文件，进行负载均衡的配置

[root@nginx01 ~]# vim /etc/nginx/nginx.conf
 13 stream {
 14 
 15      log_format  main  '$remote_addr $upstream_addr - [$time_loc    al] $status $upstream_bytes_sent';
 16       access_log  /var/log/nginx/k8s-access.log  main;  '指定日志目录'
 17 
 18       upstream k8s-apiserver {
 19   #此处为master的ip地址和端口
 20           server 192.168.179.121:6443;
 21   #此处为master2的ip地址和端口
 22           server 192.168.179.124:6443;
 23       }
 24       server {
 25                   listen 6443;
 26                   proxy_pass k8s-apiserver;
 27       }
 28       }

开启nginx服务

[root@nginx01 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successf
[root@nginx01 ~]# systemctl start nginx
[root@nginx01 ~]# systemctl status nginx
[root@nginx01 ~]# netstat -ntap |grep nginx
tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      11357/nginx: master 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      11357/nginx: maste

2.4 keepalived热备部署

两台nginx一样的操作，稍有不同处我会进行备注

安装keepalived服务

[root@nginx01 ~]# yum install keepalived -y
[root@nginx01 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"  'keepalived服务检查脚本的位置'
}

vrrp_instance VI_1 {
    state MASTER           'nginx02设置为BACKUP'
    interface ens33
    virtual_router_id 51   '这里的id两台nginx必须一样'
    priority 100            '优先级，nginx02可设置为90'
    advert_int 1            '指定VRRP 心跳包通告间隔时间，默认1秒 '
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.179.100/24    'VIP地址'
    }
    track_script {
        check_nginx
    }
}

创建监控脚本，启动keepalived服务，查看VIP地址

[root@nginx01 ~]# mkdir -p /usr/local/nginx/sbin/
[root@nginx01 ~]# vim /usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@nginx01 ~]# chmod +x /usr/local/nginx/sbin/check_nginx.sh
[root@nginx01 ~]# systemctl start keepalived.service 
[root@nginx01 ~]# systemctl status keepalived.service 

[root@nginx01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:16:80:af brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.125/24 brd 192.168.179.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.179.100/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::1301:89f0:4405:2aad/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::769:c122:2af2:c353/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::a78b:a0d4:7edd:f899/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff

[root@nginx02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:a1:c1:a2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.126/24 brd 192.168.179.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::1301:89f0:4405:2aad/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::769:c122:2af2:c353/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::a78b:a0d4:7edd:f899/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff

对漂移地址进行验证，在node01中关闭nginx服务，已查看不到VIP，此时VIP已飘到了node02中

[root@nginx01 ~]# pkill nginx
[root@nginx01 ~]# systemctl status keepalived.service 
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

[root@nginx02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:a1:c1:a2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.126/24 brd 192.168.179.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.179.100/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::1301:89f0:4405:2aad/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::769:c122:2af2:c353/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::a78b:a0d4:7edd:f899/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:48:05:53 brd ff:ff:ff:ff:ff:ff

将node01中服务恢复后，地址会飘回来

[root@nginx01 ~]# systemctl start nginx
[root@nginx01 ~]# systemctl start keepalived

2.5 修改node的VIP以及pod的创建

两台node节点都要修改VIP，我这里只展示了node01的操作

[root@node01 ~]# cd /opt/kubernetes/cfg/
[root@node01 cfg]# vim bootstrap.kubeconfig 
server: https://192.168.179.100:6443     '将地址修改为VIP地址'
[root@node01 cfg]# vim kubelet.kubeconfig 
server: https://192.168.179.100:6443     '将地址修改为VIP地址'
[root@node01 cfg]# vim kube-proxy.kubeconfig 
server: https://192.168.179.100:6443      '将地址修改为VIP地址'
[root@node01 cfg]# grep 100 *
bootstrap.kubeconfig:    server: https://192.168.179.100:6443
kubelet.kubeconfig:    server: https://192.168.179.100:6443
kube-proxy.kubeconfig:    server: https://192.168.179.100:6443
[root@node01 cfg]# systemctl restart kubelet.service 
[root@node01 cfg]# systemctl restart kube-proxy.service

此时查看k8s的日志，可以看到访问信息，是两台master轮流访问的，大大缓解了master的压力

[root@nginx01 nginx]# tail /var/log/nginx/k8s-access.log 
192.168.179.123 192.168.179.124:6443 - [29/Sep/2020:19:37:52 +0800] 200 1121
192.168.179.123 192.168.179.121:6443 - [29/Sep/2020:19:37:52 +0800] 200 1121

在master上创建pod

[root@master ~]# kubectl run nginx --image=nginx  '创建nginx的pod资源'
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx
[root@master ~]# kubectl get pods     '此时查看正在创建中'
NAME                    READY   STATUS              RESTARTS   AGE
nginx-dbddb74b8-cnkd6   0/1     ContainerCreating   0          49s
[root@master ~]# kubectl get pods     '创建完毕了'
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-cnkd6   1/1     Running   0          4m43s

此时查看pod日志是没有权限的

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-cnkd6)

指定集群中的匿名用户有管理员权限

[root@master ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created

此时可以查看pod日志，只能看到一些启动日志

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

查看pod的IP网络信息，在对应的节点进行访问

[root@master ~]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE              NOMINATED NODE
nginx-dbddb74b8-cnkd6   1/1     Running   0          10m   172.17.71.3   192.168.179.123   <none>
[root@node02 cfg]# curl 172.17.71.3

此时再次查看pod的日志信息，可以看到访问信息了

[root@master ~]# kubectl logs nginx-dbddb74b8-cnkd6
172.17.71.1 - - [29/Sep/2020:12:04:29 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
[root@master2 ~]# kubectl logs nginx-dbddb74b8-cnkd6
172.17.71.1 - - [29/Sep/2020:12:04:29 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"

长恋离亭

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
k8s二进制方式多节点部署

文章目录一、拓扑图和环境介绍二、多节点部署2.1 单节点部署2.2 master02部署2.3 nginx负载均衡部署2.4 keepalived热备部署2.5 修改node的VIP以及pod的创建一、拓扑图和环境介绍各主机ip地址分配主机名IP地址部署的服务master192.168.179.121/24apiserver、scheduler、controller-manager、etcdmaster02192.168.179.124/24apiserver、s
复制链接

扫一扫

专栏目录