CentOS7搭建Kubernetes v1.20集群
一、环境准备
1.1基本环境
项目 | 软件及版本 |
---|---|
操作系统 | Mac Mojave 10.14.6 |
虚拟化软件 | VMware Fusion 10.0.0 |
CentOS镜像 | CentOS-7-x86_64-DVD-1908.iso |
Docker | 1.13.1 |
Kubernetes | 1.20.2 |
创建两个虚拟机,安装CentOS系统,虚拟机要求:
CPU:最低2c
内存:最低4G
1.2网络配置
先配置网络,然后通过secureCRT连接进行操作,vmware fusion直连的终端还是有很多不方便的地方。
两台虚拟机,分配配置IP地址为172.16.197.160
和172.16.197.161
,修改各自的网络配置文件
cd /etc/sysconfig/network-scripts/
vi ifcfg-ens33
172.16.197.160
的网络配置为:
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO="static" #修改为static,静态IP地址
IPADDR=172.16.197.160 #设置为合法范围内的某个IP地址
GATEWAY=172.16.197.2 #设置为前面记录的网关
NETMASK=255.255.255.0 #设置为前面记录的子网掩码
DNS1="10.13.7.64" #设置为前面记录的DNS地址
DNS2="10.13.7.63"
DNS3="172.20.10.1"
DNS4="192.168.1.1"
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=fe4bddc9-aa46-46ed-8c39-56dfa3961460
DEVICE=ens33
ONBOOT=yes
172.16.197.161
的网络配置为:
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO="static" #修改为static,静态IP地址
IPADDR=172.16.197.161 #设置为合法范围内的某个IP地址
GATEWAY=172.16.197.2 #设置为前面记录的网关
NETMASK=255.255.255.0 #设置为前面记录的子网掩码
DNS1="10.13.7.64" #设置为前面记录的DNS地址
DNS2="10.13.7.63"
DNS3="172.20.10.1"
DNS4="192.168.1.1"
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=fe4bddc9-aa46-46ed-8c39-56dfa3961460
DEVICE=ens33
ONBOOT=yes
新虚拟机查看网络配置时,发现不能使用ifconfig
命令,通过yum install net-tools
进行安装,又发现不能使用yum命令,报错信息为:could not resolve host: mirrorlist.centos.org
,参考《CentOS7的yum无法安装,报错could not resolve host: mirrorlist.centos.org》进行处理。
本机使用VMware Fusion搭建虚拟机,配置静态IP地址可以参考《Mac VMware Fusion CentOS7配置静态IP》
1.3yum源配置
需要增加国内kubernetes源,可以使用清华kubernetes源,也可以采用阿里云的kubernetes源
,下面介绍下使用阿里云的kubernetes源:
cat <<EOF > kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sudo mv kubernetes.repo /etc/yum.repos.d/
另外也可以为了安装相关组件提升速度,配置CentOS国内yum源,可以看我单独写的另外一篇文章:《CentOS7配置国内yum源》。
1.4修改机器名
修改机器名,否则在搭建集群时,所有机器都是localhost.localdomain,导致无法注册到master。
设置IP地址为172.16.197.160
的虚拟机机器名为k8s.master001
hostnamectl set-hostname k8s.master001
设置IP地址为172.16.197.161
的虚拟机机器名为k8s.node001
hostnamectl set-hostname k8s.node001
1.5修改安全策略
关闭防火墙
systemctl disable firewalld
关闭selinux,修改配置/etc/selinux/config
,将SELINUX设置为disabled
,然后重启机器;
vi /etc/selinux/config
修改完成后重启机器。
reboot now
1.6修改系统配置
关闭swap,修改/etc/fstab
,注释最后一行;
vi /etc/fstab
开启 bridge-nf,修改
/etc/sysctl.conf
,末尾添加如下配置:
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
然后重启机器,reboot now
,修改完毕。
二、安装Docker
2.1安装Docker
yum -y install docker
2.2启动Docker
启动docker,并设置开机启动:
systemctl enable docker && systemctl start docker
2.3配置国内镜像仓库
修改docker配置文件/etc/docker/daemon.json
,如果没有就新建一个,配置内容:
其中https://tqb1ibke.mirror.aliyuncs.com
为阿里云镜像加速站,注册一个阿里云账号就可以查看到分配给自己的加速站地址。
重启Docker服务
sudo systemctl daemon-reload
sudo systemctl restart docker
三、安装Kubernetes(Master)
3.1安装相关组件
yum -y install kubelet kubeadm kubectl kubernetes-cni
启动kubelet,并设置开机启动:
systemctl enable kubelet && systemctl start kubelet
3.2安装master节点
使用kubeadm工具安装kubernetes集群master节点,因为本案例打算使用Flannel作为集群的网络插件,所以在安装时,需要制定CIDR,添加参数pod-network-cidr
:
kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
kubeadm
默认会去国外的镜像仓库下载kubernetes的相关镜像,我们通过传参使用阿里云镜像仓库--image-repository registry.aliyuncs.com/google_containers
。
等待一段时间后,可以得到安装完成的界面:
[root@k8s manifests]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.20.2
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "k8s.master001" could not be reached
[WARNING Hostname]: hostname "k8s.master001": lookup k8s.master001 on 10.13.7.64:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s.master001 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.197.160]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s.master001 localhost] and IPs [172.16.197.160 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s.master001 localhost] and IPs [172.16.197.160 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.003945 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s.master001 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node k8s.master001 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 2c1vo6.s4owm5tqggl08way
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.197.160:6443 --token 2c1vo6.s4owm5tqggl08way \
--discovery-token-ca-cert-hash sha256:7ef0f00150be68e05e5b39001a09899f2c2cd3426b254a10f86c81e57aed9c8c
按照上述提示,顺次执行一下命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
如果是root用户,可以只执行如下命令:
export KUBECONFIG=/etc/kubernetes/admin.conf
最后,需要牢记安装最后的kubeadm join命令,在节点加入到集群时,会用到该命令:
kubeadm join 172.16.197.160:6443 --token 2c1vo6.s4owm5tqggl08way --discovery-token-ca-cert-hash sha256:7ef0f00150be68e05e5b39001a09899f2c2cd3426b254a10f86c81e57aed9c8c
token有效期默认是24小时,如果忘记了,可以使用如下命令再生成一个:
kubeadm token create --print-join-command
也可以通过kubeadm token list
查看token
,通过openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
查看discovery-token-ca-cert-hash
kubeadm token list
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
可以看到token
和discovery-token-ca-cert-hash
的值。
3.3环境变量
上节中的export命令重启后需要重新执行,所以我们可以把它写入到.bash_profile中,具体如下:
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
kubectl命令需要使用kubernetes-admin来运行,所以在不执行以上命令时,使用kubectl命令,会出现如下错误:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
3.4检查状态
执行kubectl get cs
查看组件及状态,发现scheduler
和controller-manager
的状态为Unhealthy
,一般是因为kube-scheduler和kube-controller-manager组件配置默认禁用了非安全端口,修改配置文件中的port=0配置,配置文件路径:
/etc/kubernetes/manifests/kube-scheduler.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
分别找到两个文件的- --port=0
,将其注释掉:
然后执行systemctl restart kubelet
重启kubelet,查看状态,已经正常:
3.5安装Flannel网络插件
通过执行官网命令安装Flannel插件:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
此时发现无法进行安装,报错:Unable to connect to the server: read tcp 172.16.197.160:52028->151.101.76.133:443: read: connection reset by peer
,说明因为网络问题无法访问到yml文件。
我已经将该文件kube-flannel.yml下载下来,供大家使用。
将该文件上传到服务器后,在当前文件目录执行命令进行安装:
kubectl apply -f kube-flannel.yml
可以看到输出日志:
[root@k8s ~]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/flannel created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
通过命令kubectl get pods -o wide --all-namespaces
查看当前pod:
[root@k8s ~]# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-7f89b7bc75-5c6zv 0/1 Pending 0 50m <none> <none> <none> <none>
kube-system coredns-7f89b7bc75-lbnxs 0/1 Pending 0 50m <none> <none> <none> <none>
kube-system etcd-k8s.master001 1/1 Running 0 50m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-apiserver-k8s.master001 1/1 Running 0 50m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-controller-manager-k8s.master001 1/1 Running 0 37m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-flannel-ds-amd64-qhbzl 0/1 Init:0/1 0 80s 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-proxy-5856c 1/1 Running 0 50m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-scheduler-k8s.master001 1/1 Running 0 36m 172.16.197.160 k8s.master001 <none> <none>
发现pod kube-flannel-ds-amd64-qhbzl
一直为Init:0/1
状态,通过执行命令查看pod启动情况可以发现问题,
kubectl describe pods kube-flannel-ds-amd64-qhbzl -n kube-system
发现一直停留在Pulling image "quay.io/coreos/flannel:v0.12.0-amd64"
,说明又是因为网络原因,无法下载镜像,可以从其他地方下载,然后重新打tag,也可以从其他地方导出,然后重新load到当前机器,此处就不具体说了。
解决问题后,可以看到pod正常启动的日志:
然后重新查看pods:
[root@k8s ~]# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-7f89b7bc75-5c6zv 1/1 Running 0 57m 10.244.0.2 k8s.master001 <none> <none>
kube-system coredns-7f89b7bc75-lbnxs 1/1 Running 0 57m 10.244.0.3 k8s.master001 <none> <none>
kube-system etcd-k8s.master001 1/1 Running 0 57m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-apiserver-k8s.master001 1/1 Running 0 57m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-controller-manager-k8s.master001 1/1 Running 0 43m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-flannel-ds-amd64-qhbzl 1/1 Running 0 7m40s 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-proxy-5856c 1/1 Running 0 57m 172.16.197.160 k8s.master001 <none> <none>
kube-system kube-scheduler-k8s.master001 1/1 Running 0 43m 172.16.197.160 k8s.master001 <none> <none>
kube-flannel-ds-amd64-qhbzl
正常启动后,coredns
即可自动启动。
四、安装Kubernetes(Node)
4.1初始化环境
按照如上步骤,配置节点环境:
网络配置静态IP(1.2)
修改机器名(1.4)
修改安全策略(1.5)
修改系统配置(1.6)
安装docker(二)
安装kubeadm、kubelet、kubectl以及kubernetes-cni(3.1)
拷贝master节点的admin.conf
到相同位置,文件位置:/etc/kubernetes/admin.conf
,
然后在node节点执行:
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
4.2加入集群
在node节点执行kubeadm join
命令:
kubeadm join 172.16.197.160:6443 --token 2c1vo6.s4owm5tqggl08way --discovery-token-ca-cert-hash sha256:7ef0f00150be68e05e5b39001a09899f2c2cd3426b254a10f86c81e57aed9c8c
4.3环境检查
执行命令kubectl get nodes
查看node状态,发现node节点为NotReady
状态:
执行kubectl get pods -o wide --all-namespaces
,查看pods,发现pod kube-flannel-ds-amd64-84ql2
为Init:0/1
状态:
执行kubectl describe pods kube-flannel-ds-amd64-84ql2 -n kube-system
查看pod启动日志,发现无法获取镜像:
照旧通过其他手段获取到镜像后,查看pods状态,发现pod kube-flannel-ds-amd64-84ql2
的状态变为Init:ImagePullBackOff
,重启kubelet服务:
systemctl restart kubelet
重新查看pods状态,全部正常Running:
查看node状态,全部为Ready状态:
至此,Kubernetes集群搭建完成。