k8s多master节点使用二进制部署实操（实现高可用）

最新推荐文章于 2024-05-02 10:16:08 发布

BIGmustang

最新推荐文章于 2024-05-02 10:16:08 发布

阅读量1.7k

点赞数 1

分类专栏： kubernetes(k8s)集群文章标签： docker kubernetes 运维 nginx 服务器

本文链接：https://blog.csdn.net/BIGmustang/article/details/108896561

版权

kubernetes(k8s)集群专栏收录该内容

19 篇文章 2 订阅

订阅专栏

文章目录

一：k8s二进制方式多节点部署
二. 环境规划
- 1. K8S多节点结构拓扑图
- 2. 设备地址规划
三. 具体配置

一：k8s二进制方式多节点部署

要先部署单节点集群，可查阅博客 https://blog.csdn.net/BIGmustang/article/details/108874111

二. 环境规划

1. K8S多节点结构拓扑图

在这里插入图片描述

2. 设备地址规划

在这里插入图片描述

三. 具体配置

1. 首先按照之前的单节点证书配置，进行地址规划

cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.100.3",  //master1
      "192.168.100.11",  //lb -backup
      "192.168.100.12",  //vip
      "192.168.100.7",  //lb -master
      "192.168.195.8",  //master2
      "kubernetes",
      "kubernetes.default",

2 . 部署master2

永久关闭manager 功能

[root@master2 ~]# systemctl stop NetworkManager && systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

永久关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

永久关闭核心防护

setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config 

master节点操作，将master节点的kubernetes配置文件和启动脚本复制到master2节点

------------------------------master02部署-------------------------------------

在master01上操作复制kubernetes目录到master02

[root@master ~]# scp -r /opt/kubernetes/ root@192.168.100.8:/opt
root@192.168.100.8's password:
token.csv                                                        100%   84    63.3KB/s   00:00
kube-apiserver                                                   100%  929   685.7KB/s   00:00
kube-scheduler                                                   100%   94   132.9KB/s   00:00
kube-controller-manager                                          100%  483   853.9KB/s   00:00
kube-apiserver                                                   100%  184MB 141.8MB/s   00:01
kubectl                                                          100%   55MB 131.0MB/s   00:00
kube-controller-manager                                          100%  155MB 133.9MB/s   00:01
kube-scheduler                                                   100%   55MB 124.7MB/s   00:00
ca-key.pem                                                       100% 1675     1.9MB/s   00:00
ca.pem                                                           100% 1359     1.6MB/s   00:00
server-key.pem                                                   100% 1675     1.9MB/s   00:00
server.pem                                                       100% 1627     1.8MB/s   00:00
[root@master ~]#

复制master中的三个组件启动脚本

kube-apiserver.service    kube-controller-manager.service    kube-scheduler.service  

 scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.100.8:/usr/lib/systemd/system/
//修改配置文件kube-apiserver中的IP

复制master中的 ETCD证书

etcd 下的相关证书也需要拷贝过来，因为设备需要从2379外端口访问etcd 群集，访问时需要证书认证

//特别注意：master02一定要有etcd证书
//需要拷贝master01上已有的etcd证书给master02使用
scp -r /opt/etcd/ root@192.168.100.8:/opt/
//启动master02中的三个组件服务
systemctl start kube-apiserver.service 
systemctl start kube-controller-manager.service 
systemctl start kube-scheduler.service 
//增加环境变量
 vim /etc/profile
#末尾添加
export PATH=$PATH:/opt/kubernetes/bin/
source /etc/profile
 kubectl get node
systemctl restart kube-apiserver.service 
systemctl restart kube-controller-manager.service 
systemctl restart kube-scheduler.service 
systemctl restart kubelet.service

修改配置文件kube-apiserver中的IP为自身IP

[root@master2 cfg]# vim kube-apiserver
重启服务
[root@master2 cfg]# systemctl start kube-apiserver.service
[root@master2 cfg]# systemctl start kube-controller-manager.service
[root@master2 cfg]# systemctl start kube-scheduler.service

查看节点成功，单master部署成功

[root@master2 ssl]# kubectl get node
NAME            STATUS   ROLES    AGE     VERSION
192.168.100.5   Ready    <none>   2m56s   v1.12.3
192.168.100.6   Ready    <none>   3m7s    v1.12.3

3. 配置keepalived高可用，及nginx反向代理

环境部署，关闭核心防护，防火墙等

Lb-master       lb-backup     配置
关闭防火墙，核心防护

安装nginx服务（两台lb同样操作）

源码编译安装nginx （两台lb同样操作）
注意需要开启stream功能
–with-stream
–with-ipv6
编译时，加入此参数项
------支持ipv6
------支持四层转发以及负载均衡

或者也可以 yum 一键安装
vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@localhost ~]# yum install nginx -y

修改nginx配置文件,设置反向代理两个master（两台一样配置）

[root@lb-master opt]# cd /usr/local/nginx/conf/
[root@lb-master conf]# vim nginx.conf

events {
    worker_connections  1024;
}

stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {              //反向代理
        server 192.168.100.3:6443;     //两台master的IP地址
        server 192.168.100.8:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }

http {
    include       mime.types;
    default_type  application/octet-stream;

开启nginx服务

[root@lb-master conf]# systemctl start nginx
[root@lb-backup conf]# systemctl start nginx

安装高可用 keepalive （两台操作一样）

[root@lb-master conf]# yum install keepalived -y

修改配置文件 lb-master ，设置漂移地址等

[root@lb-master sbin]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   # 接收邮件地址 
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
    # 邮件发送地址 
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER     // 另外一台需要修改此参数NGINX_BACKUP
}
vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"
}
}

vrrp_instance VI_1 {
    state MASTER     //此台为主， 另外一台需要修改此参数为 BACKUP interface ens33
    virtual_router_id 51  //VRRP 路由 ID实例，每个实例是唯一的
    priority 100      //优先级， 另外一台需要修改此参数为90
    advert_int 1  # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.12/24   //定义的VIP漂移地址
    }
    track_script {
        check_nginx
    }
}

创建监控nginx 程序，当nginx程序意外关闭时，keepalived 也会触发关闭。

这里使用脚本操作
创建脚本：nginx2 并执行

 #!/bin/bash
mkdir /usr/local/nginx/sbin/ -p
cat <<'EOF' >/usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
    /etc/init.d/keepalived stop
  fi
EOF

chmod +x /usr/local/nginx/sbin/check_nginx.sh

执行

bash   nginx2

启动服务，测试

Systemctl start keepalived

测试keepalive漂移地址

在这里插入图片描述

结束nginx进程

Pkill  nginx

查看漂移情况
在这里插入图片描述

现在将lb-master 的 nginx 服务重启
Nginx 修复之后，地址会重新漂回master

[root@lb-master sbin]# systemctl start nginx
[root@lb-master sbin]# systemctl start keepalives

在这里插入图片描述

4. 开始修改node节点配置文件统一VIP（bootstrap.kubeconfig,kubelet.kubeconfig）

[root@node2 ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@node2 ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
[root@node2 ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig

统统修改为VIP地址

server: https://192.168.100.12:6443
重启服务
systemctl restart kubelet.service
systemctl restart kube-proxy.service

替换完成直接自检，在cfg 下操作，过滤文件中的包含100信息

root@node1 cfg]# grep 100 *
bootstrap.kubeconfig:    server: https://192.168.100.12:6443
flanneld:FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.100.3:2379,https://192.168.100.5:2379,https://192.168.100.6:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
kubelet:--hostname-override=192.168.100.5 \
kubelet.config:address: 192.168.100.5
kubelet.kubeconfig:    server: https://192.168.100.12:6443
kube-proxy:--hostname-override=192.168.100.5 \
kube-proxy.kubeconfig:    server: https://192.168.100.12:6443

检查节点状态，检查成功

[root@master2 cfg]# kubectl get node
NAME            STATUS   ROLES    AGE   VERSION
192.168.100.5   Ready    <none>   20h   v1.12.3
192.168.100.6   Ready    <none>   20h   v1.12.3

5. 查看K8S日志

[root@lb-master /]# tail /var/log/nginx/k8s-access.log

192.168.100.5 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1566
192.168.100.6 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119
192.168.100.5 192.168.100.3:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119
192.168.100.5 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119

6. 测试创建pod 资源

//在master01上操作
//测试创建pod 资源
[root@master cfg]# kubectl run nginx2 --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx2 created
[root@master cfg]# kubectl get pode
error: the server doesn't have a resource type "pode"
[root@master cfg]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-9jjsf    1/1     Running   0          83m
nginx2-cc5f746cb-252d8   1/1     Running   0          55s


ContainerCreating    代表--   //正在创建中

7. 查看master日志

//注意日志问题，第一次查看日志，发现报错

[root@master cfg]# kubectl logs nginx2-cc5f746cb-252d8
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx2-cc5f746cb-252d8)

因为默认是anonymous权限 ,需要绑定一个admin 权限来进行访问日志

[root@master cfg]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created

加完admin权限后再次访问,查看日志成功

[root@master cfg]# kubectl logs nginx2-cc5f746cb-252d8
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@master cfg]#

群集是同步的

在另外一台master 访问

[root@master2 cfg]# kubectl logs nginx2-cc5f746cb-252d8
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@master2 cfg]#

8. 查看pode 网络地址，在master 上操作

[root@master cfg]# kubectl get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE
nginx-dbddb74b8-9jjsf     1/1     Running   0          88m     172.17.14.2   192.168.100.5   <none>
nginx2-cc5f746cb-252d8    1/1     Running   0          5m29s   172.17.30.2   192.168.100.6   <none>
nginx3-674f7cffbd-gpm2w   1/1     Running   0          3m40s   172.17.14.3   192.168.100.5   <none>

在对应节点访问测试

[root@node1 cfg]# curl 172.17.14.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

访问就会产生日志，回到master上查看日志：

[root@master cfg]# kubectl logs nginx3-674f7cffbd-gpm2w
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.14.1 - - [01/Oct/2020:06:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
[root@master cfg]#

BIGmustang

关注

1
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
k8s多master节点使用二进制部署实操（实现高可用）

文章目录一：k8s二进制方式多节点部署二. 环境规划1. K8S多节点结构拓扑图2. 设备地址规划三. 具体配置1. 首先按照之前的单节点证书配置，进行地址规划2 . 部署master2永久关闭manager 功能永久关闭防火墙永久关闭核心防护在master01上操作复制kubernetes目录到master02复制master中的三个组件启动脚本复制master中的 ETCD证书修改配置文件kube-apiserver中的IP为自身IP查看节点成功，单master部署成功3. 配置
复制链接

扫一扫