参考:[1]使用部署工具安装 Kubernetes | Kubernetes
[2]linux安装部署k8s(kubernetes)和解决遇到的坑 - 简书
kubernetes之前在阿里云上直接使用过,这次尝试下与centos同源的almalinux系统,搭建本地测试的kubernetes集群。
【1】虚拟机安装
使用vmware创建四台虚拟机,在almalinux官网下载系统镜像,可选择完整版镜像,大概9个G多(AlmaLinux OS - Forever-Free Enterprise-Grade Operating System),硬件配置4核4GB内存40G硬盘,almalinux当前vmware没有直接支持,本文在安装时选择red hat enterprise linux 4 64位,almalinux版本为8.5对应red hat 4.x版本,可在官网查询。
注意在创建过程中,自定义硬件处理器设置至少两个(kubeadm支持需要),网络适配器选择桥接模式(方便四台虚拟机组网配置,也可参考其他组网方法)。
做好初始配置后即可继续安装虚拟机,系统安装时选择中文总是出现卡住的问题,选择英文安装就没有问题了,选择存储空间、创建用户、创建root密码、连接以太网络,等待安装完成后重启验证,依次完成四台虚拟机安装。
【2】配置网络
该环节主要任务是四台机器组成一个局域网,可以互相ping通并可以访问公网。
桥接模式将虚拟机的虚拟网络适配器与主机的物理网络适配器进行交接,虚拟机中的虚拟网络适配器可通过主机中的物理网络适配器直接访问到外部网络,比较适合组网的配置。首先在宿主机查看IP地址,本文中宿主机为win,在终端执行ipconfig即可查看,假设IP为192.168.111.133。
进入到虚拟机中配置,可使用图形界面配置网络,本文使用命令行配置,虚拟机IP地址按照修改宿主机IP的最后一位设置,保证在同一网段,网关、DNS等跟随宿主机配置。
[root@localhost]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
#网卡名称可能不同,在network-scripts目录下查看,修改以下字段
ONBOOT=yes
IPADDR=192.168.111.14
NETMASK=255.255.255.0
GATEWAY=192.168.111.254
DNS1=202.101.172.35
DNS2=202.101.172.47
PREFIX=24
BOOTPROTO=static
配置完成后,重启Network,查看虚拟机ip是否已经修改完成,并尝试ping主机、其他几台虚拟机、公网,均可连通,配置完成。
[root@localhost]# nmcli c reload
[root@localhost]# ifconfig
[root@localhost]# ping www.baidu.com
[root@localhost]# ping 192.168.111.133
【3】kubernetes基础配置
参考[2]进行本文的kubernetes配置,每台虚拟机都需要做的一部分配置。
设置主机名称,主机名称就是在k8s集群中的节点名称。
[root@localhost]# hostnamectl set-hostname master
[root@localhost]# hostname
禁用SELinux并关闭系统swap,具体原因查看参考[1]
[root@localhost]# vim /etc/sysconfig/selinux
SELINUX=disabled #永久禁用
[root@localhost]# swapoff -a
[root@localhost]# free -m #确认swap是否已关闭
[root@localhost]# vim /etc/sysctl.conf #永久调整swappiness参数
vm.swappiness=0
[root@localhost]# sysctl -p #使配置生效
配置iptables
[root@localhost]# cat <<EOF > /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
[root@localhost]# sysctl --system
安装docker
[root@localhost]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@localhost]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@localhost]# yum install -y docker-ce --allowerasing
[root@localhost]# systemctl start docker
[root@localhost]# systemctl enable docker
[root@localhost]# docker version
配置docker国内镜像加速,修改systemd作为驱动
[root@localhost]# vim /etc/docker/daemon.json
{
"registry-mirrors" : ["https://q5bf287q.mirror.aliyuncs.com", "https://registry.docker-cn.com","http://hub-mirror.c.163.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
[root@localhost]# systemctl daemon-reload
[root@localhost]# systemctl restart docker
配置k8s的yum源
[root@localhost]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
配置生效
[root@localhost]# yum clean all
[root@localhost]# yum makecache
#添加阿里源可能提示索引检查失败,加上参数--nogpgcheck
查看可用的kubernetes版本,本文选择1.23.0-0版本
[root@localhost]# yum list kubectl --showduplicates | sort -r
安装kubelet,kubeadm,kubectl
#可指定版本,也可删去后缀安装最新版本
[root@localhost]# yum install -y kubelet-1.23.0-0 kubeadm-1.23.0-0 kubectl-1.23.0-0
[root@localhost]# kubeadm version
[root@localhost]# kubelet --version
启动kubelet并设置开机启动
[root@localhost]# systemctl daemon-reload
[root@localhost]# systemctl start kubelet
[root@localhost]# systemctl enable kubelet #设置自启动
[root@localhost]# systemctl is-enabled kubelet
[root@localhost]# systemctl status kubelet #查看状态
[root@localhost]# journalctl -xefu kubelet #查看日志
【4】配置master节点
开放master节点防火墙端口
[root@localhost]# firewall-cmd --zone=public --add-port=2379/tcp --permanent
#客户端监听
[root@localhost]# firewall-cmd --zone=public --add-port=2380/tcp --permanent
#节点间内部通信
[root@localhost]# firewall-cmd --zone=public --add-port=6443/tcp --permanent
#kube-apiserver端口
[root@localhost]# systemctl restart firewalld.service
[root@localhost]# firewall-cmd --zone=public --list-ports
初始化k8s集群,注意指定master节点的IP和kubernetes版本
[root@localhost]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.111.2 --kubernetes-version=v1.23.0 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.1.0.0/16
初始化完成后,注意看窗口跳出的信息,记录下最后两行kubeadm join的信息,这两行是其他节点加入集群的执行命令。然后执行跳出信息的上面几行命令,添加集群配置。
[root@localhost]# mkdir -p $HOME/.kube
[root@localhost]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@localhost]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
此时,即可查看到k8s集群节点,可以看到,节点还处于NotReady状态。
[root@localhost]# kubectl get nodes
原因是cni容器网络模型没有准备好,安装flannel网络插件,需要在GitHub上curl(flannel/kube-flannel.yml at master · flannel-io/flannel · GitHub)下来,本文采用创建文件直接复制的方式,yml文件内容也贴了出来
[root@localhost]# mkdir flannel && cd flannel
[root@localhost]# vim kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni-plugin
image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
image: rancher/mirrored-flannelcni-flannel:v0.16.1
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: rancher/mirrored-flannelcni-flannel:v0.16.1
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
[root@localhost]# kubectl apply -f kube-flannel.yml
[root@localhost]# kubectl get nodes
查看节点,可以发现已经变成Ready状态了。
【5】节点配置
节点加入集群只需要配置join信息即可,即初始化集群输出的kubeadm join命令,复制输入即可
[root@localhost]# kubeadm join X.X.X.X:X --token X.X....
如果token过期或token忘记了,可以到master节点再次生成命令。
[root@master]# kubeadm token generate
2i13l6.t2ub08rgbbhcikut
[root@master]# kubeadm token create 2i13l6.t2ub08rgbbhcikut --print-join-command --ttl=0
kubeadm join x.x.x.x:x --token 2i13l6.t2ub08rgbbhcikut --discovery-token-ca-cert-hash sha256:c82973baff7a8c00d4cbaa83cf8d1cff3620a15f786878833edd98a8b8457703
此时执行kubectl命令会报错
[root@localhost]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?
kubectl命令需要使用kubernetes-admin来运行,解决方法如下,将主节点中的【/etc/kubernetes/admin.conf】文件拷贝到从节点相同目录下,然后配置环境变量并使其生效
[root@localhost]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@localhost]# source ~/.bash_profile
此时在从节点上即可使用kubectl命令,依次配置剩余虚拟机,开始kubernetes之旅