文章目录
一:k8s二进制方式多节点部署
要先部署单节点集群,可查阅博客 https://blog.csdn.net/BIGmustang/article/details/108874111
二. 环境规划
1. K8S多节点结构拓扑图
2. 设备地址规划
三. 具体配置
1. 首先按照之前的单节点证书配置,进行地址规划
cat > server-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.100.3", //master1
"192.168.100.11", //lb -backup
"192.168.100.12", //vip
"192.168.100.7", //lb -master
"192.168.195.8", //master2
"kubernetes",
"kubernetes.default",
2 . 部署master2
永久关闭manager 功能
[root@master2 ~]# systemctl stop NetworkManager && systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.
永久关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
永久关闭核心防护
setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config
master节点操作,将master节点的kubernetes配置文件和启动脚本复制到master2节点
------------------------------master02部署-------------------------------------
在master01上操作复制kubernetes目录到master02
[root@master ~]# scp -r /opt/kubernetes/ root@192.168.100.8:/opt
root@192.168.100.8's password:
token.csv 100% 84 63.3KB/s 00:00
kube-apiserver 100% 929 685.7KB/s 00:00
kube-scheduler 100% 94 132.9KB/s 00:00
kube-controller-manager 100% 483 853.9KB/s 00:00
kube-apiserver 100% 184MB 141.8MB/s 00:01
kubectl 100% 55MB 131.0MB/s 00:00
kube-controller-manager 100% 155MB 133.9MB/s 00:01
kube-scheduler 100% 55MB 124.7MB/s 00:00
ca-key.pem 100% 1675 1.9MB/s 00:00
ca.pem 100% 1359 1.6MB/s 00:00
server-key.pem 100% 1675 1.9MB/s 00:00
server.pem 100% 1627 1.8MB/s 00:00
[root@master ~]#
复制master中的三个组件启动脚本
kube-apiserver.service kube-controller-manager.service kube-scheduler.service
scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.100.8:/usr/lib/systemd/system/
//修改配置文件kube-apiserver中的IP
复制master中的 ETCD证书
etcd 下的相关证书也需要拷贝过来,因为设备需要从2379外端口访问etcd 群集,访问时需要证书认证
//特别注意:master02一定要有etcd证书
//需要拷贝master01上已有的etcd证书给master02使用
scp -r /opt/etcd/ root@192.168.100.8:/opt/
//启动master02中的三个组件服务
systemctl start kube-apiserver.service
systemctl start kube-controller-manager.service
systemctl start kube-scheduler.service
//增加环境变量
vim /etc/profile
#末尾添加
export PATH=$PATH:/opt/kubernetes/bin/
source /etc/profile
kubectl get node
systemctl restart kube-apiserver.service
systemctl restart kube-controller-manager.service
systemctl restart kube-scheduler.service
systemctl restart kubelet.service
修改配置文件kube-apiserver中的IP为自身IP
[root@master2 cfg]# vim kube-apiserver
重启服务
[root@master2 cfg]# systemctl start kube-apiserver.service
[root@master2 cfg]# systemctl start kube-controller-manager.service
[root@master2 cfg]# systemctl start kube-scheduler.service
查看节点成功,单master部署成功
[root@master2 ssl]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.100.5 Ready <none> 2m56s v1.12.3
192.168.100.6 Ready <none> 3m7s v1.12.3
3. 配置keepalived高可用,及nginx反向代理
环境部署,关闭核心防护,防火墙等
Lb-master lb-backup 配置
关闭防火墙,核心防护
安装nginx服务(两台lb同样操作)
源码编译安装nginx (两台lb同样操作)
注意需要开启stream功能
–with-stream
–with-ipv6
编译时,加入此参数项
------支持ipv6
------支持四层转发以及负载均衡
或者也可以 yum 一键安装
vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@localhost ~]# yum install nginx -y
修改nginx配置文件,设置反向代理两个master(两台一样配置)
[root@lb-master opt]# cd /usr/local/nginx/conf/
[root@lb-master conf]# vim nginx.conf
events {
worker_connections 1024;
}
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver { //反向代理
server 192.168.100.3:6443; //两台master的IP地址
server 192.168.100.8:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
http {
include mime.types;
default_type application/octet-stream;
开启nginx服务
[root@lb-master conf]# systemctl start nginx
[root@lb-backup conf]# systemctl start nginx
安装高可用 keepalive (两台操作一样)
[root@lb-master conf]# yum install keepalived -y
修改配置文件 lb-master ,设置漂移地址等
[root@lb-master sbin]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER // 另外一台需要修改此参数NGINX_BACKUP
}
vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
}
}
vrrp_instance VI_1 {
state MASTER //此台为主, 另外一台需要修改此参数为 BACKUP interface ens33
virtual_router_id 51 //VRRP 路由 ID实例,每个实例是唯一的
priority 100 //优先级, 另外一台需要修改此参数为90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.100.12/24 //定义的VIP漂移地址
}
track_script {
check_nginx
}
}
创建监控nginx 程序,当nginx程序意外关闭时,keepalived 也会触发关闭。
这里使用脚本操作
创建脚本:nginx2 并执行
#!/bin/bash
mkdir /usr/local/nginx/sbin/ -p
cat <<'EOF' >/usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
/etc/init.d/keepalived stop
fi
EOF
chmod +x /usr/local/nginx/sbin/check_nginx.sh
执行
bash nginx2
启动服务,测试
Systemctl start keepalived
测试keepalive漂移地址
结束nginx进程
Pkill nginx
查看漂移情况
现在将lb-master 的 nginx 服务重启
Nginx 修复之后,地址会重新漂回master
[root@lb-master sbin]# systemctl start nginx
[root@lb-master sbin]# systemctl start keepalives
4. 开始修改node节点配置文件统一VIP(bootstrap.kubeconfig,kubelet.kubeconfig)
[root@node2 ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@node2 ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
[root@node2 ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
统统修改为VIP地址
server: https://192.168.100.12:6443
重启服务
systemctl restart kubelet.service
systemctl restart kube-proxy.service
替换完成直接自检,在cfg 下操作,过滤文件中的包含100信息
root@node1 cfg]# grep 100 *
bootstrap.kubeconfig: server: https://192.168.100.12:6443
flanneld:FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.100.3:2379,https://192.168.100.5:2379,https://192.168.100.6:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
kubelet:--hostname-override=192.168.100.5 \
kubelet.config:address: 192.168.100.5
kubelet.kubeconfig: server: https://192.168.100.12:6443
kube-proxy:--hostname-override=192.168.100.5 \
kube-proxy.kubeconfig: server: https://192.168.100.12:6443
检查节点状态,检查成功
[root@master2 cfg]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.100.5 Ready <none> 20h v1.12.3
192.168.100.6 Ready <none> 20h v1.12.3
5. 查看K8S日志
[root@lb-master /]# tail /var/log/nginx/k8s-access.log
192.168.100.5 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1566
192.168.100.6 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119
192.168.100.5 192.168.100.3:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119
192.168.100.5 192.168.100.8:6443 - [01/Oct/2020:13:47:34 +0800] 200 1119
6. 测试创建pod 资源
//在master01上操作
//测试创建pod 资源
[root@master cfg]# kubectl run nginx2 --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx2 created
[root@master cfg]# kubectl get pode
error: the server doesn't have a resource type "pode"
[root@master cfg]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-9jjsf 1/1 Running 0 83m
nginx2-cc5f746cb-252d8 1/1 Running 0 55s
ContainerCreating 代表-- //正在创建中
7. 查看master日志
//注意日志问题,第一次查看日志,发现报错
[root@master cfg]# kubectl logs nginx2-cc5f746cb-252d8
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx2-cc5f746cb-252d8)
因为默认是anonymous权限 ,需要绑定一个admin 权限来进行访问日志
[root@master cfg]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
加完admin权限后再次访问,查看日志成功
[root@master cfg]# kubectl logs nginx2-cc5f746cb-252d8
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@master cfg]#
群集是同步的
在另外一台master 访问
[root@master2 cfg]# kubectl logs nginx2-cc5f746cb-252d8
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@master2 cfg]#
8. 查看pode 网络地址,在master 上操作
[root@master cfg]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-9jjsf 1/1 Running 0 88m 172.17.14.2 192.168.100.5 <none>
nginx2-cc5f746cb-252d8 1/1 Running 0 5m29s 172.17.30.2 192.168.100.6 <none>
nginx3-674f7cffbd-gpm2w 1/1 Running 0 3m40s 172.17.14.3 192.168.100.5 <none>
在对应节点访问测试
[root@node1 cfg]# curl 172.17.14.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
访问就会产生日志,回到master上查看日志:
[root@master cfg]# kubectl logs nginx3-674f7cffbd-gpm2w
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.14.1 - - [01/Oct/2020:06:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
[root@master cfg]#