CentOS7 Kubernetes 1.14.1安装、启动、验证及踩坑

Kubernetes v1.14.1安装、启动、验证

安装、启动流程

修改主机名（node同样道理）

vi /etc/hostname
k8s-master
reboot

配置主机和ip的对应关系（master和node都要配）

vi /etc/hosts
192.168.80.133 k8s-master
192.168.80.134 k8s-node1

关闭防火墙（master和node都要配）

sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
sudo firewall-cmd --state

在主机上禁用SELinux，使容器可以读取主机文件（master和node都要配）

sudo setenforce 0

创建k8s网桥,，添加如下配置（master和node都要配）

sudo vi /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

sudo modprobe br_netfilter
sudo sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

kube-proxy开启ipvs的前置条件（master和node都要配）

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

关闭swap，否则后续add node会有问题（master和node都要配）

swapoff -a

配置阿里yum镜像库（master和node都要配）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装相关工具（node只需要安装docker kubelet）

yum install -y docker kubelet kubeadm kubectl kubernetes-cni

将docker和kubelet设置开机自启（master和node都要配）

systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet

配置gcr.io镜像库加速器，去掉最后的"}“前的”,"，否则可能下载不了镜像，并重启docker

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

systemctl restart docker

查看kubeadm版本需要的镜像列表，返回如下（目前该阿里云镜像yum安装的kubernetes是v1.14.1，后续依赖版本号调整以下步骤）（master上）

kubeadm config images list

k8s.gcr.io/kube-apiserver:v1.14.1
k8s.gcr.io/kube-controller-manager:v1.14.1
k8s.gcr.io/kube-scheduler:v1.14.1
k8s.gcr.io/kube-proxy:v1.14.1
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

下载所需镜像

docker pull mirrorgooglecontainers/kube-apiserver:v1.14.1
docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.1
docker pull mirrorgooglecontainers/kube-scheduler:v1.14.1
docker pull mirrorgooglecontainers/kube-proxy:v1.14.1
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.10
docker pull coredns/coredns:1.3.1

将镜像打tag，因为要改成kubeadm默认的名字才能识别到

docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.14.1 k8s.gcr.io/kube-proxy:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.14.1 k8s.gcr.io/kube-scheduler:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.14.1 k8s.gcr.io/kube-apiserver:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.14.1 k8s.gcr.io/kube-controller-manager:v1.14.1
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.10  k8s.gcr.io/etcd:3.3.10
docker tag docker.io/mirrorgooglecontainers/pause:3.1  k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.3.1  k8s.gcr.io/coredns:1.3.1

初始化，–pod-network-cidr=10.244.0.0/16要和后续的flannel中的Network：要一致，–apiserver-advertise-address是master的ip（master上）

kubeadm init \
   --kubernetes-version=1.14.1 \
   --pod-network-cidr=10.244.0.0/16 \
   --apiserver-advertise-address=192.168.80.133

加入node，下面一串是init成功master上的一串（node上），也可以通过一下方式获取sha256

kubeadm join 192.168.80.133:6443 --token 5st1rn.kw5lzwcq0t8d0na9 \
    --discovery-token-ca-cert-hash sha256:574037bb6e61bcb7e70c10e026f0ba39afec52ba3b3d22ff5f9a0f3224b5e797
    
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

查看k8s所有节点状态（master）

kubectl get pod --all-namespaces -o wide

安装flannel

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

查看所有资源都变为RUNNING

[外链图片转存失败(img-6WWCtTPs-1566567818005)(C:\Users\chenzhen15\Desktop\k8sSuccess.png)]

安装kubernetes-dashboard

docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.0
docker tag mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.0 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0

wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml

或者

wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml

sed -i 's#k8s.gcr.io#gcrxio#g' kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml

开放kubernetes-dashboard service

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  # 在Service中使用NodePort类型，开放30443端口，k8s默认端口的范围是30000-32767
  type: NodePort
  ports:
  - port: 6443
    targetPort: 8443
    nodePort: 30443 
  selector:
    k8s-app: kubernetes-dashboard

添加一个用户，名字为administrator

kubectl create serviceaccount administrator -n kube-system

绑定为集群的管理员

kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:administrator

kubectl get secret --all-namespaces
#NAME                  TYPE                                  DATA   AGE
#kube-system       administrator-token-5qcdv                        kubernetes.io/service-account-token   3      2m56s

获取administrator的token（可用于令牌登陆）

kubectl describe secret -n kube-system administrator-token-5qcdv

生成kubeconfig文件（config登陆）

DASH_TOCKEN=$(kubectl get secret -n kube-system administrator-token-5qcdv -o jsonpath={.data.token}|base64 -d)

kubectl config set-cluster kubernetes --server=192.168.80.133:6443 --kubeconfig=/root/administrator.conf

kubectl config set-credentials administrator --token=$DASH_TOCKEN --kubeconfig=/root/administrator.conf

kubectl config set-context administrator@kubernetes --cluster=kubernetes --user=administrator --kubeconfig=/root/administrator.conf

kubectl config use-context administrator@kubernetes --kubeconfig=/root/administrator.conf

登陆方式

#1令牌登陆，按照一下指令将查询到的token直接复制到页面即可登陆
kubectl describe secret -n kube-system admin

#2使用kubeconfig文件登陆
DASH_TOCKEN=$(kubectl get secret -n kube-system admin-token-l7kpn -o jsonpath={.data.token}|base64 -d)

kubectl config set-cluster kubernetes --server=192.168.0.25:6443 --kubeconfig=/root/dashbord-admin.conf

kubectl config set-credentials dashboard-admin --token=$DASH_TOCKEN --kubeconfig=/root/dashbord-admin.conf

kubectl config set-context dashboard-admin@kubernetes --cluster=kubernetes --user=dashboard-admin --kubeconfig=/root/dashbord-admin.conf

kubectl config user-context dashboard-admin@kubernets --kubeconfig=/root/dashbord-admin.conf

删除kubernetes-dashboard

kubectl delete deployment kubernetes-dashboard --namespace=kube-system 
kubectl delete service kubernetes-dashboard  --namespace=kube-system 
kubectl delete role kubernetes-dashboard-minimal --namespace=kube-system 
kubectl delete rolebinding kubernetes-dashboard-minimal --namespace=kube-system
kubectl delete sa kubernetes-dashboard --namespace=kube-system 
kubectl delete secret kubernetes-dashboard-certs --namespace=kube-system
kubectl delete secret kubernetes-dashboard-key-holder --namespace=kube-system

验证

测试dns是否正常，使用命令

kubectl run curl --image=radial/busyboxplus:curl -i --tty

输出如下：

If you don't see a command prompt, try pressing enter.
[ root@curl-66bdcf564-wbpfc:/ ]$ nslookup kubernetes.default

进入后执行

nslookup kubernetes.default

确认解析正常,输出如下:

[ root@curl-66bdcf564-wbpfc:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

退出

exit;

---

踩坑及解决方法

问题1：daemonsets.extensions is forbidden: User “system:node:xxxx” cannot create daemonsets.extensions in the namespace “kube-system”

执行

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

或者

export KUBECONFIG=/etc/kubernetes/admin.conf

在或者将配置直接写入环境变量，否则只对当前session有效

问题2：sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory

执行

sudo modprobe br_netfilter

问题3：Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)

执行（普通用户权限）

export KUBECONFIG=/etc/kubernetes/kubelet.conf

或（管理员权限）

export KUBECONFIG=/etc/kubernetes/admin.conf