使用 kubeadm 搭建 v1.15.3 版本 Kubernetes 集群

使用 kubeadm 搭建 v1.15.3 版本 Kubernetes 集群

原创： 阳明 k8s技术圈 6天前

前面文章和课程中我们都是使用的 Kubeadm 搭建的 Kubernetes 集群，但是版本比较低了（1.10.0版本），近期有不少反馈让更新下版本，本文将通过 Kubeadm 来搭建最新版本的 Kubernetes 1.15.3 集群，其实和以前搭建的方式方法基本一致，我们这里准备使用 calico 网络插件以及 ipvs 模式的 kube-proxy。

 https://unsplash.com/photos/nfxD5dWy1wk

环境准备

3个节点，都是 Centos 7.6 系统，内核版本：3.10.0-957.12.2.el7.x86_64，在每个节点上添加 hosts 信息：

1. $ cat /etc/hosts

2. 10.151.30.11 ydzs-master

3. 10.151.30.22 ydzs-node1

4. 10.151.30.23 ydzs-node2

禁用防火墙：

1. $ systemctl stop firewalld

2. $ systemctl disable firewalld

禁用SELINUX：

1. $ setenforce 0

2. $ cat /etc/selinux/config

3. SELINUX=disabled

创建 /etc/sysctl.d/k8s.conf文件，添加如下内容：

1. net.bridge.bridge-nf-call-ip6tables = 1

2. net.bridge.bridge-nf-call-iptables = 1

3. net.ipv4.ip_forward = 1

执行如下命令使修改生效：

1. $ modprobe br_netfilter

2. $ sysctl -p /etc/sysctl.d/k8s.conf

安装 ipvs

1. $ cat > /etc/sysconfig/modules/ipvs.modules <<EOF

2. #!/bin/bash

3. modprobe -- ip_vs

4. modprobe -- ip_vs_rr

5. modprobe -- ip_vs_wrr

6. modprobe -- ip_vs_sh

7. modprobe -- nf_conntrack_ipv4

8. EOF

9. $ chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的 /etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。使用 lsmod|grep-e ip_vs-e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

接下来还需要确保各个节点上已经安装了 ipset 软件包：

1. $ yum install ipset

为了便于查看 ipvs 的代理规则，最好安装一下管理工具 ipvsadm：

1. $ yum install ipvsadm

同步服务器时间

1. $ yum install chrony -y

2. $ systemctl enable chronyd

3. $ systemctl start chronyd

4. $ chronyc sources

5. 210 Number of sources = 4

6. MS Name/IP address Stratum Poll Reach LastRx Last sample

7. ===============================================================================

8. ^+ sv1.ggsrv.de 2 6 17 32 -823us[-1128us] +/- 98ms

9. ^- montreal.ca.logiplex.net 2 6 17 32 -17ms[ -17ms] +/- 179ms

10. ^- ntp6.flashdance.cx 2 6 17 32 -32ms[ -32ms] +/- 161ms

11. ^* 119.28.183.184 2 6 33 32 +661us[ +357us] +/- 38ms

12. $ date

13. Tue Aug 27 09:28:41 CST 2019

关闭 swap 分区：

1. $ swapoff -a

修改 /etc/fstab文件，注释掉 SWAP 的自动挂载，使用 free-m确认 swap 已经关闭。swappiness 参数调整，修改 /etc/sysctl.d/k8s.conf添加下面一行：

1. vm.swappiness=0

执行 sysctl-p/etc/sysctl.d/k8s.conf使修改生效。

接下来可以安装 Docker

1. $ yum install -y yum-utils \

2. device-mapper-persistent-data \

3. lvm2

4. $ yum-config-manager \

5. --add-repo \

6. https://download.docker.com/linux/centos/docker-ce.repo

7. $ yum list docker-ce --showduplicates | sort -r

8. * updates: mirrors.tuna.tsinghua.edu.cn

9. Loading mirror speeds from cached hostfile

10. Loaded plugins: fastestmirror, langpacks

11. Installed Packages

12. * extras: mirrors.tuna.tsinghua.edu.cn

13. * epel: mirrors.yun-idc.com

14. docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable

15. docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable

16. docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable

17. ......

18. docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable

19. docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable

20. ......

21. * base: mirror.lzu.edu.cn

22. Available Packages

可以选择安装一个版本，比如我们这里安装最新版本：

1. $ yum install docker-ce-19.03.1-3.el7

配置 Docker 镜像加速器

1. $ vi /etc/docker/daemon.json

2. {

3. "exec-opts": ["native.cgroupdriver=systemd"],

4. "registry-mirrors" : [

5. "https://ot2k4d59.mirror.aliyuncs.com/"

6. ]

7. }

启动 Docker

1. $ systemctl start docker

2. $ systemctl enable docker

在确保 Docker 安装完成后，上面的相关环境配置也完成了，现在我们就可以来安装 Kubeadm 了，我们这里是通过指定yum 源的方式来进行安装的：

1. cat <<EOF > /etc/yum.repos.d/kubernetes.repo

2. [kubernetes]

3. name=Kubernetes

4. baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

5. enabled=1

6. gpgcheck=1

7. repo_gpgcheck=1

8. gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg

9. https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

10. EOF

当然了，上面的 yum 源是需要科学上网的，如果不能科学上网的话，我们可以使用阿里云的源进行安装：

1. cat <<EOF > /etc/yum.repos.d/kubernetes.repo

2. [kubernetes]

3. name=Kubernetes

4. baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

5. enabled=1

6. gpgcheck=0

7. repo_gpgcheck=0

8. gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

9. http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

10. EOF

然后安装 kubeadm、kubelet、kubectl：

1. $ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

2. $ kubeadm version

3. kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:11:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

可以看到我们这里安装的是 v1.15.3 版本，然后将 kubelet 设置成开机启动：

1. $ systemctl enable kubelet.service

到这里为止上面所有的操作都需要在所有节点执行配置。

初始化集群

然后接下来在 master 节点配置 kubeadm 初始化文件，可以通过如下命令导出默认的初始化配置：

1. $ kubeadm config print init-defaults > kubeadm.yaml

然后根据我们自己的需求修改配置，比如修改 imageRepository 的值，kube-proxy 的模式为 ipvs，另外需要注意的是我们这里是准备安装 calico 网络插件的，需要将 networking.podSubnet 设置为 192.168.0.0/16：

1. apiVersion: kubeadm.k8s.io/v1beta2

2. bootstrapTokens:

3. - groups:

4. - system:bootstrappers:kubeadm:default-node-token

5. token: abcdef.0123456789abcdef

6. ttl: 24h0m0s

7. usages:

8. - signing

9. - authentication

10. kind: InitConfiguration

11. localAPIEndpoint:

12. advertiseAddress: 10.151.30.11 # apiserver 节点内网IP

13. bindPort: 6443

14. nodeRegistration:

15. criSocket: /var/run/dockershim.sock

16. name: ydzs-master

17. taints:

18. - effect: NoSchedule

19. key: node-role.kubernetes.io/master

20. ---

21. apiServer:

22. timeoutForControlPlane: 4m0s

23. apiVersion: kubeadm.k8s.io/v1beta2

24. certificatesDir: /etc/kubernetes/pki

25. clusterName: kubernetes

26. controllerManager: {}

27. dns:

28. type: CoreDNS # dns类型

29. etcd:

30. local:

31. dataDir: /var/lib/etcd

32. imageRepository: gcr.azk8s.cn/google_containers

33. kind: ClusterConfiguration

34. kubernetesVersion: v1.15.3 # k8s版本

35. networking:

36. dnsDomain: cluster.local

37. podSubnet: 192.168.0.0/16

38. serviceSubnet: 10.96.0.0/12

39. scheduler: {}

40. ---

41. apiVersion: kubeproxy.config.k8s.io/v1alpha1

42. kind: KubeProxyConfiguration

43. mode: ipvs # kube-proxy 模式

然后使用上面的配置文件进行初始化：

1. $ kubeadm init --config kubeadm.yaml

2. [init] Using Kubernetes version: v1.15.3

3. [preflight] Running pre-flight checks

4. [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.1. Latest validated version: 18.09

5. [preflight] Pulling images required for setting up a Kubernetes cluster

6. [preflight] This might take a minute or two, depending on the speed of your internet connection

7. [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

8. [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

9. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

10. [kubelet-start] Activating the kubelet service

11. [certs] Using certificateDir folder "/etc/kubernetes/pki"

12. [certs] Generating "etcd/ca" certificate and key

13. [certs] Generating "etcd/server" certificate and key

14. [certs] etcd/server serving cert is signed for DNS names [ydzs-master localhost] and IPs [10.151.30.11 127.0.0.1 ::1]

15. [certs] Generating "etcd/peer" certificate and key

16. [certs] etcd/peer serving cert is signed for DNS names [ydzs-master localhost] and IPs [10.151.30.11 127.0.0.1 ::1]

17. [certs] Generating "apiserver-etcd-client" certificate and key

18. [certs] Generating "etcd/healthcheck-client" certificate and key

19. [certs] Generating "ca" certificate and key

20. [certs] Generating "apiserver" certificate and key

21. [certs] apiserver serving cert is signed for DNS names [ydzs-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.151.30.11]

22. [certs] Generating "apiserver-kubelet-client" certificate and key

23. [certs] Generating "front-proxy-ca" certificate and key

24. [certs] Generating "front-proxy-client" certificate and key

25. [certs] Generating "sa" key and public key

26. [kubeconfig] Using kubeconfig folder "/etc/kubernetes"

27. [kubeconfig] Writing "admin.conf" kubeconfig file

28. [kubeconfig] Writing "kubelet.conf" kubeconfig file

29. [kubeconfig] Writing "controller-manager.conf" kubeconfig file

30. [kubeconfig] Writing "scheduler.conf" kubeconfig file

31. [control-plane] Using manifest folder "/etc/kubernetes/manifests"

32. [control-plane] Creating static Pod manifest for "kube-apiserver"

33. [control-plane] Creating static Pod manifest for "kube-controller-manager"

34. [control-plane] Creating static Pod manifest for "kube-scheduler"

35. [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

36. [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

37. [kubelet-check] Initial timeout of 40s passed.

38. [apiclient] All control plane components are healthy after 42.012149 seconds

39. [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

40. [kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster

41. [upload-certs] Skipping phase. Please see --upload-certs

42. [mark-control-plane] Marking the node ydzs-master as control-plane by adding the label "node-role.kubernetes.io/master=''"

43. [mark-control-plane] Marking the node ydzs-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

44. [bootstrap-token] Using token: abcdef.0123456789abcdef

45. [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

46. [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

47. [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

48. [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

49. [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

50. [addons] Applied essential addon: CoreDNS

51. [addons] Applied essential addon: kube-proxy

52.  

53. Your Kubernetes control-plane has initialized successfully!

54.  

55. To start using your cluster, you need to run the following as a regular user:

56.  

57. mkdir -p $HOME/.kube

58. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

59. sudo chown $(id -u):$(id -g) $HOME/.kube/config

60.  

61. You should now deploy a pod network to the cluster.

62. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

63. https://kubernetes.io/docs/concepts/cluster-administration/addons/

64.  

65. Then you can join any number of worker nodes by running the following on each as root:

66.  

67. kubeadm join 10.151.30.11:6443 --token abcdef.0123456789abcdef \

68. --discovery-token-ca-cert-hash sha256:deb5158b39948a4592ff48512047ea6e45b288c248872724a28f15008962178b

可以看到最新验证的 docker 版本是18.09，虽然是一个 warning，所以最好还是安装18.09版本的 docker。

拷贝 kubeconfig 文件

1. $ mkdir -p $HOME/.kube

2. $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

3. $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

添加节点

 

1. $ kubeadm join 10.151.30.11:6443 --token abcdef.0123456789abcdef \

2. --discovery-token-ca-cert-hash sha256:deb5158b39948a4592ff48512047ea6e45b288c248872724a28f15008962178b

3. [preflight] Reading configuration from the cluster...

4. [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

5. [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace

6. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

7. [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

8. [kubelet-start] Activating the kubelet service

9. [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

10.  

11. This node has joined the cluster:

12. * Certificate signing request was sent to apiserver and a response was received.

13. * The Kubelet was informed of the new secure connection details.

14.  

15. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

如果忘记了上面的 join 命令可以使用命令 kubeadm token create--print-join-command重新获取。

执行成功后运行 get nodes 命令：

1. $ kubectl get nodes

2. NAME STATUS ROLES AGE VERSION

3. ydzs-master NotReady master 39m v1.15.3

4. ydzs-node1 NotReady <none> 106s v1.15.3

可以看到是 NotReady 状态，这是因为还没有安装网络插件，接下来安装网络插件，可以在文档 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ 中选择我们自己的网络插件，这里我们安装 calio:

1. $ wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

2. # 因为有节点是多网卡，所以需要在资源清单文件中指定内网网卡

3. $ vi calico.yaml

4. ......

5. spec:

6. containers:

7. - env:

8. - name: DATASTORE_TYPE

9. value: kubernetes

10. - name: IP_AUTODETECTION_METHOD # DaemonSet中添加该环境变量

11. value: interface=eth0 # 指定内网网卡

12. - name: WAIT_FOR_DATASTORE

13. value: "true"

14. ......

15. $ kubectl apply -f calico.yaml # 安装calico网络插件

隔一会儿查看 Pod 运行状态：

1. $ kubectl get pods -n kube-system

2. NAME READY STATUS RESTARTS AGE

3. calico-kube-controllers-65b8787765-svztx 1/1 Running 0 82s

4. calico-node-gmng9 1/1 Running 0 82s

5. calico-node-t695p 1/1 Running 0 82s

6. coredns-cf8fb6d7f-jsq5h 1/1 Running 0 42m

7. coredns-cf8fb6d7f-vxz4c 1/1 Running 0 42m

8. etcd-ydzs-master 1/1 Running 0 41m

9. kube-apiserver-ydzs-master 1/1 Running 0 41m

10. kube-controller-manager-ydzs-master 1/1 Running 0 41m

11. kube-proxy-4z4vf 1/1 Running 0 42m

12. kube-proxy-qk57t 1/1 Running 0 5m11s

13. kube-scheduler-ydzs-master 1/1 Running 0 41m

网络插件运行成功了，node 状态也正常了：

1. kubectl get nodes

2. NAME STATUS ROLES AGE VERSION

3. ydzs-master Ready master 3h25m v1.15.3

4. ydzs-node1 Ready <none> 168m v1.15.3

用同样的方法添加另外一个节点即可。

安装 Dashboard

1. $ wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

2. $ vi kubernetes-dashboard.yaml

3. # 修改镜像名称

4. ......

5. containers:

6. - args:

7. - --auto-generate-certificates

8. image: gcr.azk8s.cn/google_containers/kubernetes-dashboard-amd64:v1.10.1

9. imagePullPolicy: IfNotPresent

10. ......

11. # 修改Service为NodePort类型

12. ......

13. selector:

14. k8s-app: kubernetes-dashboard

15. type: NodePort

16. ......

直接创建：

1. $ kubectl apply -f kubernetes-dashboard.yaml

2. $ kubectl get pods -n kube-system -l k8s-app=kubernetes-dashboard

3. NAME READY STATUS RESTARTS AGE

4. kubernetes-dashboard-fcfb4cbc-t462n 1/1 Running 0 50m

5. $ kubectl get svc -n kube-system -l k8s-app=kubernetes-dashboard

6. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

7. kubernetes-dashboard NodePort 10.110.172.49 <none> 443:32497/TCP 55m

然后可以通过上面的 32497 端口去访问 Dashboard，要记住使用 https，Chrome不生效可以使用Firefox测试：

然后创建一个具有全局所有权限的用户来登录Dashboard：(admin.yaml)

1. kind: ClusterRoleBinding

2. apiVersion: rbac.authorization.k8s.io/v1beta1

3. metadata:

4. name: admin

5. annotations:

6. rbac.authorization.kubernetes.io/autoupdate: "true"

7. roleRef:

8. kind: ClusterRole

9. name: cluster-admin

10. apiGroup: rbac.authorization.k8s.io

11. subjects:

12. - kind: ServiceAccount

13. name: admin

14. namespace: kube-system

15.  

16. ---

17. apiVersion: v1

18. kind: ServiceAccount

19. metadata:

20. name: admin

21. namespace: kube-system

22. labels:

23. kubernetes.io/cluster-service: "true"

24. addonmanager.kubernetes.io/mode: Reconcile

直接创建：

1. $ kubectl apply -f admin.yaml

2. $ kubectl get secret -n kube-system|grep admin-token

3. admin-token-d5jsg kubernetes.io/service-account-token 3 1d

4. $ kubectl get secret admin-token-d5jsg -o jsonpath={.data.token} -n kube-system |base64 -d# 会生成一串很长的base64后的字符串

然后用上面的base64解码后的字符串作为token登录Dashboard即可：

最终我们就完成了。

---

 

k8s进阶课程推荐：打造独当一面的 Kubernetes 运维、开发工程师

k8s进阶课程：https://youdianzhishi.com/course/6n8xd6/

golang实战课程：https://youdianzhishi.com/course/67kv5m/

扫描下面的二维码(或微信搜索 k8s技术圈)关注我们的微信公众帐号，在微信公众帐号中回复 加群 即可加入到我们的 kubernetes 讨论群里面共同学习。