使用kubeadm在CentOS 7上安装Kubernetes 1.8

1. 系统配置

1.1 关闭防火墙

1. systemctl stop firewalld
2. systemctl disable firewalld

1.2 禁用SELinux

1. setenforce 0

编辑文件/etc/selinux/config，将SELINUX修改为disabled，如下：

1. SELINUX=disabled

1.3 关闭系统Swap

Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。方法一,通过kubelet的启动参数–fail-swap-on=false更改这个限制。方法二,关闭系统的Swap。

1. swapoff -a

修改/etc/fstab文件，注释掉SWAP的自动挂载，使用free -m确认swap已经关闭。

2. 安装Docker

注： 所有节点均需执行该步骤。

2.1 下载Docker安装包

• 下载地址：https://download.docker.com/linux/centos/7/x86_64/stable/Packages/

• 下载安装包：

1. mkdir ~/k8s
2. cd k8s
3. wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
4. wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm

2.2 安装Docker

1. cd k8s
2. yum install ./docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
3. yum install ./docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm
4. systemctl enable docker
5. systemctl start docker

2.3 配置Docker

• 开启iptables filter表的FORWARD链 
  编辑/lib/systemd/system/docker.service，在ExecStart=..上面加入如下内容：

1. ExecStartPost=/usr/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT

如下：

1. ......
2. ExecStartPost=/usr/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
3. ExecStart=/usr/bin/dockerd
4. ......

• 配置Cgroup Driver 
  创建文件/etc/docker/daemon.json，添加如下内容：

1. {
2. "exec-opts": ["native.cgroupdriver=systemd"]
3. }

• 重启Docker服务

1. systemctl daemon-reload && systemctl restart docker && systemctl status docker

3. 安装Kubernetes

3.1 安装kubeadm、kubectl、kubelet

• 配置软件源

1. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
2. [kubernetes]
3. name=Kubernetes
4. baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
5. enabled=1
6. gpgcheck=1
7. repo_gpgcheck=1
8. gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
9. EOF

• 解决路由异常

1. cat <<EOF > /etc/sysctl.d/k8s.conf
2. net.bridge.bridge-nf-call-ip6tables = 1
3. net.bridge.bridge-nf-call-iptables = 1
4. EOF
5. sysctl --system

• 调整swappiness参数 
  修改/etc/sysctl.d/k8s.conf添加下面一行：

1. vm.swappiness=0

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

• 安装kubeadm、kubectl、kubelet 
  ① 查看可用版本

1. yum list --showduplicates | grep 'kubeadm\|kubectl\|kubelet'

② 安装指定版本

1. yum install kubeadm-1.8.1 kubectl-1.8.1 kubelet-1.8.1
2. systemctl enable kubelet
3. systemctl start kubelet

3.2 使用kubeadm init初始化集群

注：该小节仅在Master节点上执行

• 初始化Master节点

1. kubeadm init --kubernetes-version=v1.8.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=master.k8s.samwong.im

• 配置普通用户使用kubectl访问集群

1. mkdir -p $HOME/.kube
2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

• 查看集群状态

1. [root@master ~]# kubectl get cs
2. NAME STATUS MESSAGE ERROR
3. scheduler Healthy ok
4. controller-manager Healthy ok
5. etcd-0 Healthy {"health": "true"}

• 初始化失败清理命令

1. kubeadm reset
2. ifconfig cni0 down
3. ip link delete cni0
4. ifconfig flannel.1 down
5. ip link delete flannel.1
6. rm -rf /var/lib/cni/

3.3 安装Pod Network

注：该小节仅在Master节点上执行

• 安装Flannel

1. [root@master ~]# cd ~/k8s
2. [root@master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
3. [root@master ~]# kubectl apply -f kube-flannel.yml
4. clusterrole "flannel" created
5. clusterrolebinding "flannel" created
6. serviceaccount "flannel" created
7. configmap "kube-flannel-cfg" created
8. daemonset "kube-flannel-ds" created

• 指定网卡 
  如果有多个网卡，需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=。

1. ......
2. apiVersion: extensions/v1beta1
3. kind: DaemonSet
4. metadata:
5. name: kube-flannel-ds
6. ......
7. containers:
8. - name: kube-flannel
9. image: quay.io/coreos/flannel:v0.9.0-amd64
10. command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ]
11. ......

• 查询Pod状态

1. kubectl get pod --all-namespaces -o wide

3.4 Master节点参与工作负载

使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，可使用如下命令使Master节点参与工作负载。

1. kubectl taint nodes node1 node-role.kubernetes.io/master-

3.5 向Kubernetes集群添加Node

• 查看master的token

1. kubeadm token list | grep authentication,signing | awk '{print $1}'

• 查看discovery-token-ca-cert-hash

1. openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

• 添加节点到Kubernetes集群

1. kubeadm join --token=a20844.654ef6410d60d465 --discovery-token-ca-cert-hash sha256:0c2dbe69a2721870a59171c6b5158bd1c04bc27665535ebf295c918a96de0bb1 master.k8s.samwong.im:6443

• 查看集群中的节点

1. [root@master ~]# kubectl get nodes
2. NAME STATUS ROLES AGE VERSION
3. master.k8s.samwong.im Ready master 1d v1.8.1

3.6 从Kubernetes集群中移除节点

• Master节点操作

1. kubectl drain master.k8s.samwong.im --delete-local-data --force --ignore-daemonsets
2. kubectl delete node master.k8s.samwong.im

• Node节点操作

1. kubeadm reset
2. ifconfig cni0 down
3. ip link delete cni0
4. ifconfig flannel.1 down
5. ip link delete flannel.1
6. rm -rf /var/lib/cni/

• 查看集群节点

1. kubectl get nodes

3.7 部署Dashboard插件

• 下载Dashboard插件配置文件

1. cd ~/k8s
2. wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

• 修改Dashboard Service 
  编辑kubernetes-dashboard.yaml文件，在Dashboard Service中添加type: NodePort，暴露Dashboard服务。

1. # ------------------- Dashboard Service ------------------- #
3. kind: Service
4. apiVersion: v1
5. metadata:
6. labels:
7. k8s-app: kubernetes-dashboard
8. name: kubernetes-dashboard
9. namespace: kube-system
10. spec:
11. type: NodePort
12. ports:
13. - port: 443
14. targetPort: 8443
15. selector:
16. k8s-app: kubernetes-dashboard

• 安装Dashboard插件

1. kubectl create -f kubernetes-dashboard.yaml

• 授予Dashboard账户集群管理权限 
  创建一个kubernetes-dashboard-admin的ServiceAccount并授予集群admin的权限，创建kubernetes-dashboard-admin.rbac.yaml。

1. ---
2. apiVersion: v1
3. kind: ServiceAccount
4. metadata:
5. labels:
6. k8s-app: kubernetes-dashboard
7. name: kubernetes-dashboard-admin
8. namespace: kube-system
10. ---
11. apiVersion: rbac.authorization.k8s.io/v1beta1
12. kind: ClusterRoleBinding
13. metadata:
14. name: kubernetes-dashboard-admin
15. labels:
16. k8s-app: kubernetes-dashboard
17. roleRef:
18. apiGroup: rbac.authorization.k8s.io
19. kind: ClusterRole
20. name: cluster-admin
21. subjects:
22. - kind: ServiceAccount
23. name: kubernetes-dashboard-admin
24. namespace: kube-system

执行命令：

1. [root@master ~]# kubectl create -f kubernetes-dashboard-admin.rbac.yaml
2. serviceaccount "kubernetes-dashboard-admin" created
3. clusterrolebinding "kubernetes-dashboard-admin" created

• 查看kubernete-dashboard-admin的token

1. [root@master ~]# kubectl -n kube-system get secret | grep kubernetes-dashboard-admin
2. kubernetes-dashboard-admin-token-jxq7l kubernetes.io/service-account-token 3 22h
3. [root@master ~]# kubectl describe -n kube-system secret/kubernetes-dashboard-admin-token-jxq7l
4. Name: kubernetes-dashboard-admin-token-jxq7l
5. Namespace: kube-system
6. Labels: <none>
7. Annotations: kubernetes.io/service-account.name=kubernetes-dashboard-admin
8. kubernetes.io/service-account.uid=686ee8e9-ce63-11e7-b3d5-080027d38be0
10. Type: kubernetes.io/service-account-token
12. Data
13. ====
14. namespace: 11 bytes
15. token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1qeHE3bCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY4NmVlOGU5LWNlNjMtMTFlNy1iM2Q1LTA4MDAyN2QzOGJlMCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.Ua92im86o585ZPBfsOpuQgUh7zxgZ2p1EfGNhr99gAGLi2c3ss-2wOu0n9un9LFn44uVR7BCPIkRjSpTnlTHb_stRhHbrECfwNiXCoIxA-1TQmcznQ4k1l0P-sQge7YIIjvjBgNvZ5lkBNpsVanvdk97hI_kXpytkjrgIqI-d92Lw2D4xAvHGf1YQVowLJR_VnZp7E-STyTunJuQ9hy4HU0dmvbRXBRXQ1R6TcF-FTe-801qUjYqhporWtCaiO9KFEnkcYFJlIt8aZRSL30vzzpYnOvB_100_DdmW-53fLWIGYL8XFnlEWdU1tkADt3LFogPvBP4i9WwDn81AwKg_Q
16. ca.crt: 1025 bytes

• 查看Dashboard服务端口

1. [root@master k8s]# kubectl get svc -n kube-system
2. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
3. kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 1d
4. kubernetes-dashboard NodePort 10.102.209.161 <none> 443:32513/TCP 21h

• 访问Dashboard 
  打开浏览器输入https://192.168.56.2:32513，如下： 
  

3.8 部署heapster插件

安装Heapster为集群添加使用统计和监控功能，为Dashboard添加仪表盘。

1. mkdir -p ~/k8s/heapster
2. cd ~/k8s/heapster
3. wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
4. wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
5. wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
6. wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
7. kubectl create -f ./

4. 遇到的问题

4.1 使用代理科学上网

• 没有代理 
  可申请AWS免费账户，创建EC2实例，搭建Shadowsocks服务器。

• 配置代理客户端 
  参考链接：https://www.zybuluo.com/ncepuwanghui/note/954160

• 配置Docker代理 
  ① 创建docker服务配置文件

1. mkdir -p /etc/systemd/system/docker.service.d

② 编辑vi /etc/systemd/system/docker.service.d/http-proxy.conf，添加如下内容：

1. [Service]
2. Environment="HTTP_PROXY=http://master.k8s.samwong.im:8118" "NO_PROXY=localhost,*.samwong.im,192.168.0.0/16,127.0.0.1,10.244.0.0/16"

③ 编辑/etc/systemd/system/docker.service.d/https-proxy.conf，添加如下内容：

1. [Service]
2. Environment="HTTPS_PROXY=https://master.k8s.samwong.im:8118" "NO_PROXY=localhost,*.samwong.im,192.168.0.0/16,127.0.0.1,10.244.0.0/16"

④ 重启Docker服务

1. systemctl daemon-reload && systemctl restart docker

⑤ 查看是否配置成功

1. [root@master k8s]# systemctl show --property=Environment docker | more
2. Environment=HTTP_PROXY=http://master.k8s.samwong.im:8118 NO_PROXY=localhost,*.samwong.im,192.168.0.0/16,127.0.0.1,10.244.0.0/16 HTTPS_PROXY=https://master.k8
3. s.samwong.im:8118

• 配置yum代理 
  ① 编辑/etc/yum.conf文件，追加如下内容：

1. proxy=http://master.k8s.samwong.im:8118

② 更新yum缓存

1. yum makecache

• 配置wget代理 
  编辑/etc/wgetrc文件，追加如下内容：

1. ftp_proxy=http://master.k8s.samwong.im:8118
2. http_proxy=http://master.k8s.samwong.im:8118
3. https_proxy=http://master.k8s.samwong.im:8118

• 配置全局代理 
  如需上网，可编辑/etc/profile文件，追加如下内容：

1. PROXY_HOST=master.k8s.samwong.im
2. export all_proxy=http://$PROXY_HOST:8118
3. export ftp_proxy=http://$PROXY_HOST:8118
4. export http_proxy=http://$PROXY_HOST:8118
5. export https_proxy=http://$PROXY_HOST:8118
6. export no_proxy=localhost,*.samwong.im,192.168.0.0/16.,127.0.0.1,10.10.0.0/16

注： 部署Kubernetes时需禁用全局代理，会导致访问内部服务失败。

4.2 下载软件包和镜像

• 下载kubeadm、kubectl、kubelet

1. wget https://storage.googleapis.com/kubernetes-release/release/v1.8.1/bin/linux/amd64/kubeadm
2. wget https://storage.googleapis.com/kubernetes-release/release/v1.8.1/bin/linux/amd64/kubectl
3. wget https://storage.googleapis.com/kubernetes-release/release/v1.8.1/bin/linux/amd64/kubelet

参考链接：https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl

4.3 推送本地镜像到镜像仓库

• 上传镜像

1. docker login -u xxxxxx@163.com -p xxxxxx hub.c.163.com
2. docker tag gcr.io/google_containers/kube-apiserver-amd64:v1.8.1 hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1
3. docker push hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1
4. docker rmi hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1
5. docker logout hub.c.163.com

• 下载镜像

1. docker pull hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1
2. docker tag hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1 gcr.io/google_containers/kube-apiserver-amd64:v1.8.1
3. docker rmi hub.c.163.com/xxxxxx/kube-apiserver-amd64:v1.8.1
4. docker logout hub.c.163.com

• 更新镜像

1. docker update --restart=no $(docker ps -q) && docker stop $(docker ps -q) && docker rm $(docker ps -q)

4.4 kubeadm init错误

• 错误描述

1. {
2. "kind": "Status",
3. "apiVersion": "v1",
4. "metadata": {
6. },
7. "status": "Failure",
8. "message": "nodes is forbidden: User \"system:anonymous\" cannot list nodes at the cluster scope",
9. "reason": "Forbidden",
10. "details": {
11. "kind": "nodes"
12. },
13. "code": 403
14. }

• 问题原因 
  该节点在/etc/profile中配置了全局代理，kubectl访问kube-apiserver也通过代理转发请求，导致证书不对，连接拒绝。

• 解决方法 
  取消全局代理，只配置Docker代理、yum代理、wget代理。 
  参考4.1。

4.5 向Kubernetes集群添加Node失败

• 问题描述 
  在Node上使用kubeadm join命令向kubernetes集群添加节点时提示Failed，如下：

1. kubeadm join --token=a20844.654ef6410d60d465 --discovery-token-ca-cert-hash sha256:0c2dbe69a2721870a59171c6b5158bd1c04bc27665535ebf295c918a96de0bb1 master.k8s.samwong.im:6443
2. [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
3. [preflight] Running pre-flight checks
4. [discovery] Trying to connect to API Server "master.k8s.samwong.im:6443"
5. [discovery] Created cluster-info discovery client, requesting info from "https://master.k8s.samwong.im:6443"
6. [discovery] Failed to request cluster info, will try again: [Get https://master.k8s.samwong.im:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: EOF]

• 问题原因 
  token失效被删除。在Master上查看token，结果为空。

1. kubeadm token list

• 解决方法 
  重新生成token，默认token有效期为24小时，生成token时通过指定--ttl 0可设置token永久有效。

1. [root@master ~]# kubeadm token create --ttl 0
2. 3a536a.5d22075f49cc5fb8
3. [root@master ~]# kubeadm token list
4. TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
5. 3a536a.5d22075f49cc5fb8 <forever> <never> authentication,signing <none> system:bootstrappers:kubeadm:default-node-token

5. 参考链接：

• 使用kubeadm安装Kubernetes 1.8版本
• Docker HTTP/HTTPS 代理设置
• CentOS 7安装配置Shadowsocks客户端
• 推送本地镜像到网易云镜像仓库