Kubernetes(k8s) 1.23.6版本基于Docker的集群安装部署

Bulut0907

已于 2022-07-03 15:48:35 修改

阅读量7k

点赞数 7

分类专栏： Kubernetes 文章标签： kubernetes docker 1.23.6 集群安装云原生

于 2022-05-24 10:36:37 首次发布

本文链接：https://blog.csdn.net/yy8623977/article/details/124685772

版权

Kubernetes 专栏收录该内容

24 篇文章 20 订阅

订阅专栏

1. 部署方式

有几下几种部署方式：

minikube：一个用于快速搭建单节点的kubernetes工具
kubeadm：一个用于快速搭建kubernetes集群的工具
二进制包：从官网上下载每个组件的二进制包，依次去安装

这里我们选用kubeadm方式进行安装

2. 集群规划

Kubernetes有一主多从或多主多从的集群部署方式，这里我们采用一主多从的方式

服务器名称	服务器IP	角色	CPU(最低要求)	内存(最低要求)
k8s-master	192.168.23.160	master	2核	2G
k8s-node1	192.168.23.161	node	2核	2G
k8s-node2	192.168.23.162	node	2核	2G

3. docker安装

这里需要安装与Kubernetes兼容的docker版本，参考链接：

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md
https://github.com/kubernetes/kubernetes/blob/v1.23.6/build/dependencies.yaml

containerd也需要和Docker兼容，参考链接：

https://docs.docker.com/engine/release-notes/
https://github.com/moby/moby/blob/v20.10.7/vendor.conf

所以这里Docker安装20.10.7版本，containerd安装1.4.6版本

Docker的安装参考centos7基于yum repository方式安装docker和卸载docker

4. 安装k8s集群

4.1 基础环境

1. 禁用selinux

临时禁用方法

[root@k8s-master ~]# setenforce 0
[root@k8s-master ~]# getenforce
Permissive
[root@k8s-master ~]#

永久禁用方法。需重启服务器

[root@k8s-master ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

2. 关闭swap

swap分区指的是虚拟内存分区，它的作用是在物理内存使用完之后，将磁盘空间虚拟成内存来使用。但是会对系统性能产生影响。所以这里需要关闭。如果不能关闭，则在需要修改集群的配置参数

临时关闭方法

[root@k8s-master ~]# swapoff -a  
[root@k8s-master ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           1819         286         632           9         900        1364
Swap:             0           0           0
[root@k8s-master ~]#

永久关闭方法。需重启服务器

[root@k8s-master ~]# sed -ri 's/.*swap.*/#&/' /etc/fstab

3. bridged网桥设置

为了让服务器的iptables能发现bridged traffic，需要添加网桥过滤和地址转发功能

新建modules-load.d/k8s.conf文件

[root@k8s-master ~]# cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
[root@k8s-master ~]#

新建sysctl.d/k8s.conf文件

[root@k8s-master ~]# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-master ~]#

加载配置文件

[root@k8s-master ~]# sysctl --system

加载br_netfilter网桥过滤模块，和加载网络虚拟化技术模块

[root@k8s-master ~]# modprobe br_netfilter
[root@k8s-master ~]# modprobe overlay

检验网桥过滤模块是否加载成功

[root@k8s-master ~]# lsmod | grep -e br_netfilter -e overlay
br_netfilter           22256  0 
bridge                151336  1 br_netfilter
overlay                91659  0 
[root@k8s-master ~]#

3.4 配置IPVS

service有基于iptables和基于ipvs两种代理模型。基于ipvs的性能要高一些。需要手动载入才能使用ipvs模块

安装ipset和ipvsadm

[root@k8s-master ~]# yum install ipset ipvsadm

新建脚本文件/etc/sysconfig/modules/ipvs.modules，内容如下

[root@k8s-master ~]# cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
 
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
[root@k8s-master ~]#

添加执行权限给脚本文件，然后执行脚本文件

[root@k8s-master ~]# chmod +x /etc/sysconfig/modules/ipvs.modules
[root@k8s-master ~]# /bin/bash /etc/sysconfig/modules/ipvs.modules
[root@k8s-master ~]#

检验模块是否加载成功

[root@k8s-master ~]# lsmod | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  0 
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4      15053  2 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_conntrack          139264  7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack
[root@k8s-master ~]#

4.2 安装kubelet、kubeadm、kubectl

添加yum源

[root@k8s-master ~]# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
   http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@k8s-master ~]#

安装，然后启动kubelet

[root@k8s-master ~]# yum install -y --setopt=obsoletes=0 kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6
[root@k8s-master ~]# systemctl enable kubelet --now 
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s-master ~]#

说明如下：

obsoletes等于1表示更新旧的rpm包的同时会删除旧包，0表示更新旧的rpm包不会删除旧包
kubelet启动后，可以用命令journalctl -f -u kubelet查看kubelet更详细的日志
kubelet默认使用systemd作为cgroup driver
启动后，kubelet现在每隔几秒就会重启，因为它陷入了一个等待kubeadm指令的死循环

4.3 下载各个机器需要的镜像

查看集群所需镜像的版本

[root@k8s-master ~]# kubeadm config images list
I0510 21:58:56.690111    9902 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.23
k8s.gcr.io/kube-apiserver:v1.23.6
k8s.gcr.io/kube-controller-manager:v1.23.6
k8s.gcr.io/kube-scheduler:v1.23.6
k8s.gcr.io/kube-proxy:v1.23.6
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6
[root@k8s-master ~]#

编辑镜像下载文件images.sh，然后执行。其中node节点只需要kube-proxy和pause

[root@k8s-master ~]# tee ./images.sh <<'EOF'
#!/bin/bash

images=(
kube-apiserver:v1.23.6
kube-controller-manager:v1.23.6
kube-scheduler:v1.23.6
kube-proxy:v1.23.6
pause:3.6
etcd:3.5.1-0
coredns:v1.8.6
)
for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done
EOF
[root@k8s-master ~]#
[root@k8s-master ~]# chmod +x ./images.sh && ./images.sh

4.4 初始化主节点(只在master节点执行)

[root@k8s-master ~]# kubeadm init \
--apiserver-advertise-address=192.168.23.160 \
--control-plane-endpoint=k8s-master \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.23.6 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.23.6
[preflight] Running pre-flight checks
......省略部分......
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join k8s-master:6443 --token 0qc9py.n6az0o2jy1tryg2b \
	--discovery-token-ca-cert-hash sha256:f049a62946af45c27d9a387468d598906fd68e6d918d925ce699cb4f2a32e111 \
	--control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master:6443 --token 0qc9py.n6az0o2jy1tryg2b \
	--discovery-token-ca-cert-hash sha256:f049a62946af45c27d9a387468d598906fd68e6d918d925ce699cb4f2a32e111 
[root@k8s-master ~]#

说明：

可以使用参数--v=6或--v=10等查看详细的日志
所有参数的网络不能重叠。比如192.168.2.x和192.168.3.x是重叠的
–pod-network-cidr：指定pod网络的IP地址范围。直接填写这个就可以了
–service-cidr：service VIP的IP地址范围。默认就10.96.0.0/12。直接填写这个就可以了
–apiserver-advertise-address：API Server监听的IP地址

另一种kubeadm init的方法

# 打印默认的配置信息
[root@k8s-master ~]# kubeadm config print init-defaults --component-configs KubeletConfiguration
# 通过默认的配置信息，进行编辑修改，其中serviceSubnet和podSubnet在同一层级。然后拉取镜像
[root@k8s-master ~]# kubeadm config images pull --config kubeadm-config.yaml
# 进行初始化
[root@k8s-master ~]# kubeadm init --config kubeadm-config.yaml

如果init失败，使用如下命令进行回退

[root@k8s-master ~]# kubeadm reset -f
[root@k8s-master ~]# 
[root@k8s-master ~]# rm -rf /etc/kubernetes
[root@k8s-master ~]# rm -rf /var/lib/etcd/
[root@k8s-master ~]# rm -rf $HOME/.kube

4.5 设置.kube/config(只在master执行)

[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl会读取该配置文件

4.6 安装网络插件flannel(只在master执行)

插件使用的是DaemonSet的控制器，会在每个节点都运行

根据github上的README.md当前说明，这个是支持Kubenetes1.17+的

如果因为镜像下载导致部署出错。可以先替换yaml文件内的image源为国内的镜像源

[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@k8s-master ~]#

会下载rancher/mirrored-flannelcni-flannel:v0.17.0和rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1这两个镜像

此时查看master的状态

[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS      AGE
kube-system   coredns-65c54cc984-lqcxl             1/1     Running   0             49m
kube-system   coredns-65c54cc984-q2n72             1/1     Running   0             49m
kube-system   etcd-k8s-master                      1/1     Running   2 (14m ago)   49m
kube-system   kube-apiserver-k8s-master            1/1     Running   2 (14m ago)   49m
kube-system   kube-controller-manager-k8s-master   1/1     Running   2 (14m ago)   49m
kube-system   kube-flannel-ds-6v9jg                1/1     Running   0             9m15s
kube-system   kube-proxy-6dz9x                     1/1     Running   2 (14m ago)   49m
kube-system   kube-scheduler-k8s-master            1/1     Running   2 (14m ago)   49m
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   49m   v1.23.6
[root@k8s-master ~]#

4.7 加入node节点(只在node执行)

由上面的kubeadm init成功后的结果得来的

[root@k8s-node1 ~]# kubeadm join k8s-master:6443 --token 0qc9py.n6az0o2jy1tryg2b \
	--discovery-token-ca-cert-hash sha256:f049a62946af45c27d9a387468d598906fd68e6d918d925ce699cb4f2a32e111
[preflight] Running pre-flight checks
......省略部分......
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-node1 ~]#

令牌有效期24小时，可以在master节点生成新令牌命令

[root@k8s-master ~]# kubeadm token create --print-join-command

然后在master通过命令watch -n 3 kubectl get pods -A和kubectl get nodes查看状态

4.7.1 node节点可以执行kubectl命令方法

在master节点上将$HOME/.kube复制到node节点的$HOME目录下

[root@k8s-master ~]# 
[root@k8s-master ~]# scp -r $HOME/.kube k8s-node1:$HOME
[root@k8s-master ~]#

5. 部署dashboard(只在master执行)

Kubernetes官方可视化界面：https://github.com/kubernetes/dashboard

5.1 部署

dashboard和kubernetes的版本对应关系，参考：https://github.com/kubernetes/dashboard/blob/v2.5.1/go.mod

[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@k8s-master ~]#

会下载kubernetesui/dashboard:v2.5.1、kubernetesui/metrics-scraper:v1.0.7两个镜像

在master通过命令watch -n 3 kubectl get pods -A查看状态

5.2 设置访问端口

将type: ClusterIP改为：type: NodePort

[root@k8s-master ~]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
service/kubernetes-dashboard edited
[root@k8s-master ~]#

查看端口命令

[root@k8s-master ~]# kubectl get svc -A | grep kubernetes-dashboard
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.96.114.89    <none>        8000/TCP                 2m52s
kubernetes-dashboard   kubernetes-dashboard        NodePort    10.96.180.218   <none>        443:32314/TCP            2m52s
[root@k8s-master ~]#

访问dashborad页面：https://k8s-node1:32314，如下所示
dashboard
这里需要登录令牌，通过下面的步骤获取

5.3 创建访问账号

创建资源文件，然后应用

[root@k8s-master ~]# tee ./dash.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f dash.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
[root@k8s-master ~]#

5.4 获取访问令牌

[root@k8s-master ~]# kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"
eyJhbGciOiJSUzI1NiIsImtpZCI6ImJYYjZIYUV5YXZ6QTZMUFV1UTVnZG9pb3ZlUGkwUjNRcVVqbW4waUJ3aTQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTZoeng3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI0ZTZjNTU1NC1kYThiLTRjYTgtYjMwYy1jY2UwZDIxNzVlNzYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.YZm6NOIR8Owh8FOu_z0XVfgNjlW2qfz5SaAYz3blzXy51-HZUdkKqQq7fc5zHaAnlVWlIHF35FTOrk-JKI89IlLvbNYiTLbUHOthWf075O4gMIB6siX863c9Ao0ZAujEnrXjyQGdpI3HgdjHBEFkNgTrzR5kRPNnpf36dNG4IZ0hNzyFLH2daJTri0bVRXZ40ZsqaH_0BPf_uVYdZzlVMxe_ZgDYVWgR9W0OYr1oV-OW3vFHBy9b_GhJZkruzN58QDj-Zg20KfYD5Kk5xBS5SMaYVyq7cHs0RagI-3SNFHWYVYYaSKYvLzZWjjwx1SopF9rBbBeEIjdLJkgMZ0RqeQ[root@k8s-master ~]#

令牌为：eyJhbGci…gMZ0RqeQ

将令牌复制到登录页面，进行登录即可

6. 安装nginx进行测试

部署

[root@k8s-master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@k8s-master ~]#

在master通过命令watch -n 3 kubectl get pods -A查看状态

暴露端口

[root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@k8s-master ~]#

查看端口

[root@k8s-master ~]# kubectl get pods,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-85b98978db-hldcq   1/1     Running   0          64s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        76m
service/nginx        NodePort    10.96.167.226   <none>        80:30523/TCP   15s
[root@k8s-master ~]#

访问nginx页面：http://k8s-node1:30523

nginx页面