节点信息:
主机名 | IP | 角色 |
k8s-master | 10.10.0.10 | master节点 |
k8s-node01 | 10.10.0.11 | 集群worke节点 |
k8s-node02 | 10.10.0.12 | 集群worke节点 |
一、初始化系统环境
系统初始化环境配置,需要在三台服务器都执行,以下命令请看对应的命令参数,在不同服务器执行。
1)主机名
分别在三台机器执行对应命令,设置主机名
[root@centos01 ~]# hostnamectl set-hostname k8s-master [root@centos02 ~]# hostnamectl set-hostname k8s-node01 [root@centos03 ~]# hostnamectl set-hostname k8s-node02
2)关闭防火墙、selinux
[root@centos01 ~]# systemctl disable firewalld [root@centos01 ~]# systemctl stop firewalld [root@centos01 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
3)设置内核
[root@centos01 ~]# vim /etc/sysctl.d/k8s.conf
## 添加如下内容
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 ## 执行命令生效 [root@centos01 ~]# modprobe br_netfilter [root@centos01 ~]# sysctl -p /etc/sysctl.d/k8s.conf
4)免密钥
在master节点进行对2台node节点进行免密钥登陆
[root@k8s-master ~]# ssh-keygen [root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.11
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.12
二、部署kubernetes集群
1)配置yum源
yum源三台机器都需要配置,这里我们以master主机为例,node节点也按照此yum配置即可
[root@k8s-master ~]# yum -y install wget [root@k8s-master ~]# cd /etc/yum.repos.d ## 配置docker-ce源 [root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ## 配置kubernetes源 [root@k8s-master yum.repos.d]# vim /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes Repo baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg enabled=1 ## 下载校验文件 [root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg [root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg ## 导入校验文件 [root@k8s-master ~]# rpm --import rpm-package-key.gpg [root@k8s-master ~]# rpm --import yum-key.gpg
[root@k8s-master ~]# yum clean all && yum makecache fast
2)安装docker、kubelet、kubeadm等工具
[root@k8s-master ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 kubectl-1.14.1 docker-ce -y
## node节点无需配置kubctl组件
[root@k8s-node01 ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
[root@k8s-node02 ~]# yum install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
3)配置docker kubelet
2台node节点也需要修改对应配置
## 设置kubelet启动时忽略swap报错 [root@k8s-master ~]# vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false"
## 设置开机自启动
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable docker && systemctl restart docker
[root@k8s-master ~]# systemctl enable kubelet && systemctl restart kubelet
4)初始化master节点
kubernetes从1.13版本开始,可以指定镜像仓库进行集群初始化操作,所以我们直接指定阿里云镜像仓库进行集群初始化,这样无需再关心国内网络环境是否可以下载到对应的官方原始pod镜像
[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# kubeadm init --apiserver-advertise-address=10.10.0.10 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.14.1 \ --pod-network-cidr=10.244.0.0/16
参数解释:
- apiserver-advertise-address:apiserver地址及master节点地址
- image-repository:镜像仓库地址
- pod-network-cidr:pod网络
以上初始化,需要等待一段时间,因为需要下载对应组件镜像,master节点初始化完成后,可看到以下提示信息:
........ [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \ --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
注意:
token是node节点加入时需要用到的信息,需要记录下来,tocken值24小时后失效,若果超过24小时你再进行node节点加入集群,需要重新生成tocken。
5)配置集群环境变量
用户想要使用kubectl操作集群,则需要配置kubectl环境变量,这些命令也是上面kubeadm init后输出的内容
[root@k8s-master ~]# mkdir -p $HOME/.kube [root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config
## 查看集群信息
[root@k8s-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
## 查看node就绪状态
### 由于还没有安装网络插件,以及node节点未加入集群,所以只显示一个master节点信息
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 7m33s v1.14.1
三、部署网络插件
集群的运行依赖于网络,k8s本身并不支持网络,需要额外部署对应的网络插件,才可实现集群的个组件网络通信。我们这里采用flannel作为集群网络插件。
项目地址:https://github.com/coreos/flannel
## 在线部署 [root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ## 或者把清单配置文件下载本地再执行清单文件应用 [root@k8s-master ~]# mkdir /opt/k8s/flannel [root@k8s-master ~]# cd /opt/k8s/flannel [root@k8s-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@k8s-master ~]# kubectl apply -f kube-flannel.yml
四、node节点加入集群
1)加入集群
node节点加入集群操作基本一致,这里以其中一台为例。kubeadm join命令为master节点初始化成功后显示的命令,上文已经提到过。
[root@k8s-node01 ~]# swapoff -a
[root@k8s-node01 ~]# kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \ --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "k8s-node01" could not be reached
[WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
注意:
node节点初始化加入集群,会看到对应的初始化成功信息,初始化时由于需要下载对应pod镜像,需要等待一段时间,node才会成功加入。
2)查看节点就绪状态
[root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready master 139m v1.14.1 k8s-node01 Ready <none> 2m8s v1.14.1 k8s-node02 Ready <none> 64s v1.14.1
3)集群删除
以上就是kubeadm部署集群过程,过程其实并不复杂,如果在集群部署过程中,出现问题,想要删除重新初始化集群,使用以下命令:
kubeadm reset ifconfig cni0 down && ip link delete cni0 ifconfig flannel.1 down && ip link delete flannel.1 rm -rf /var/lib/cni/
特别说明:
1)关于token失效
上面说过,master集群初始化后,token24小时后就会失效,如果到了token失效时间,node再加入集群,需要重新生产token:
## 查看token状态
### TTL值 就是token生于时间 [root@k8s-master ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 5ti5kd.o32bm9lofv6zej94 21h 2019-05-22T11:16:31+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token ## 重新生产token [root@k8s-master ~]# kubeadm token create hb0mhv.ckb79uumxh06br8e ## 获取--discovery-token-ca-cert-hash值 [root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'
cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
## 加入集群命令
[root@k8s-master ~]# kubeadm join 10.10.0.10:6443 --token hb0mhv.ckb79uumxh06br8e \
--discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f