国内环境Kubernetes v1.12.3的安装与配置

@[TOC]国内环境Kubernetes v1.12.3的安装与配置
0、环境
主机名 IP
master1 172.16.105.21
master2 172.16.105.22
master3 172.16.105.23
node1 172.16.105.21
node2 172.16.105.22
node2 172.16.105.23
ansible-client 172.16.105.21
1、安装ansible和依赖
在172.16.105.21安装ansible

升级内环版本（用于cephfs文件系统）：
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum -y --enablerepo=elrepo-kernel install kernel-ml-devel kernel-ml

grep menuentry /boot/grub2/grub.cfg
grub2-set-default ‘CentOS Linux (4.20.3-1.el7.elrepo.x86_64) 7 (Core)’
modprobe nf_conntrack

安装 python 及 epel

yum install -y epel-release python-pip python34 python34-pip

安装 ansible

yum install -y ansible
pip install netaddr
pip install --upgrade jinja2
2、建立公私钥，分发各服务器
在ansible-clinet机器生成免密密钥对

ssh-keygen -t rsa -P ‘’
将生成的公钥(id_rsa.pub)传到其他节点，这样ansible-client可以免密登陆其他机器

cat id_rsa.pub >> ~/.ssh/authorized_keys

3、下载源码
git clone https://github.com/kubernetes-sigs/kubespray
git checkout release-2.8

4.安装指定docker
yum list docker-ce --showduplicates | sort -r #要安装特定版本的Docker CE，请列出repo中的可用版本
yum install docker-ce- #安装特定版本
yum install docker-ce-18.06.1.ce-3.el7

yum install kubectl-1.12.5-0
yum install kubelet-1.12.5
yum install kubeadm-1.12.5-0
5.生成cluster配置文件

Copy inventory/sample as inventory/mycluster

cp -rfp inventory/sample inventory/mycluster

Update Ansible inventory file with inventory builder

declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}

6.替换镜像
find . -name ‘.yml’ | xargs -n1 -I{} sed -i ‘s/gcr.io/google-containers//mirrorgooglecontainers//’ {}
find . -name '.yml’ | xargs -n1 -I{} sed -i ‘s/k8s.gcr.io//mirrorgooglecontainers//’ {}

7.运行命令
ansible-playbook -i inventory/mycluster/hosts.ini --become --become-user=root cluster.yml -vvv

8.启用addons
编辑inventory/testcluster/group_vars/k8s-cluster/addons.yml，将需要的addon设为true，并配置对应的参数。比如我这里启用了dashboard、heml、local_volume_provisioner、cephfs_provisioner、ingress_nginx和cert_manager：

dashboard_enabled: true
helm_enabled: true
registry_enabled: false
metrics_server_enabled: true
local_volume_provisioner_enabled: true
cephfs_provisioner_enabled: true
cephfs_provisioner_namespace: "cephfs-provisioner"
cephfs_provisioner_cluster: ceph
cephfs_provisioner_monitors: "10.32.3.70:6789,10.32.3.71:6789,10.32.3.72:6789"
cephfs_provisioner_admin_id: k8s
cephfs_provisioner_secret: AQBCk+tbHeLjORAAHiUMFIeu8f76JWBWlCWfbg==
cephfs_provisioner_storage_class: cephfs
cephfs_provisioner_reclaim_policy: Delete
cephfs_provisioner_claim_root: /k8s_volumes
cephfs_provisioner_deterministic_names: true
ingress_nginx_enabled: true
cert_manager_enabled: true

7、安装失败如何清理
rm -rf /etc/kubernetes/
rm -rf /etc/ssl/etcd/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd
rm -rf /usr/local/bin/kubectl
rm -rf /etc/systemd/system/calico-node.service
rm -rf /etc/systemd/system/kubelet.service
systemctl stop etcd.service
systemctl disable etcd.service
systemctl stop calico-node.service
systemctl disable calico-node.service
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)
systemctl restart docker

访问dashboard
创建dashboard管理用户

kubectl create serviceaccount dashboard-admin -n kube-system

绑定用户为集群管理用户

kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

获取tocken
kubectl describe secret -n kube-system dashboard-admin-token-

cephfs:
git clone https://github.com/kubernetes-incubator/external-storage.git
cd external-storage/ceph/cephfs/deploy/rbac/
vi clusterrole.yaml
加入配置项：

• apiGroups: [""]
  resources: [“secrets”]
  verbs: [“get”, “create”, “delete”]