centos7 Kubeadm安装配置K8S 及Dashboard外部服务

环境：
Kubernetes Master节点：192.168.0.47
Kubernetes node1节点：192.168.0.33
Kubernetes node2节点：192.168.0.37
Docker Harbor私有库节点：192.168.0.59 （过程省略，见 https://blog.csdn.net/oToyix/article/details/117738036）

一、centos7 Kubeadm安装配置K8S

1、所有节点，HOST及防火墙设置

#添加hosts解析；
cat >/etc/hosts<<EOF
127.0.0.1 localhost localhost.localdomain
192.168.0.47 master1
192.168.0.33 node1
192.168.0.37 node2
EOF
#临时关闭selinux和防火墙；
sed -i '/SELINUX/s/enforcing/disabled/g'  /etc/sysconfig/selinux
setenforce  0
systemctl   stop     firewalld.service
systemctl   disable   firewalld.service
#同步节点时间；
yum install ntpdate -y
ntpdate  pool.ntp.org
#修改对应节点主机名；
hostname `cat /etc/hosts|grep $(ifconfig|grep broadcast|awk '{print $2}'|grep 192)|awk '{print $2}'`;su
#关闭swapoff；
swapoff -a

2、所有节点，内核优化

cat > /etc/modules-load.d/ipvs.conf <<EOF
# Load IPVS at boot
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF
systemctl enable --now systemd-modules-load.service
#确认内核模块加载成功
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
#安装ipset、ipvsadm
yum install -y ipset ipvsadm
#配置内核参数;
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

3、所有节点，安装docker-ce及docker-compose

执行下面脚本

if [ $id -ne 0 ];then
	echo "username is not root,please use root,now exit"
	exit
fi

cd /usr/local/
echo "download docker-19.03.0.tgz, wait......"
wget -c http://mirrors.163.com/docker-ce/linux/static/stable/x86_64/docker-20.10.7.tgz
tar -xf docker-20.10.7.tgz
useradd -s /sbin/nologin docker -M

echo "set docker path"
cat>>/etc/profile<<EOF
export PATH=\$PATH:/usr/local/docker/
EOF

mkdir /etc/docker

echo "daemon.json"
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"]
}
EOF

cd docker
echo "cp -rf * /usr/bin/"
cp -rf * /usr/bin/
echo "show version"
docker --version

echo "download docker-compose"
wget -c https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
chmod a+x /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

echo "docker.sevice in systemd"
cat > /etc/systemd/system/docker.service <<EOF
[Unit]
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

echo "reload daemon"
systemctl daemon-reload
echo "restart docker"
systemctl restart docker
docker ps

4、所有节点-添加yum源

cat>>/etc/yum.repos.d/kubernetes.repo<<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
EOF

5、所有节点安装 kubeadm kubelet kubectl

#安装Kubeadm；
yum install -y kubeadm-1.20.4 kubelet-1.20.4 kubectl-1.20.4
rpm -qa|grep kube
kubeadm-1.20.4-0.x86_64
kubectl-1.20.4-0.x86_64
kubelet-1.20.4-0.x86_64
kubernetes-cni-0.8.7-0.x86_64
#启动kubelet服务
systemctl enable kubelet.service
systemctl start kubelet.service

6、master1节点 执行kubeadm init初始化安装Master相关软件；

[root@master1 ~]# kubeadm init --control-plane-endpoint=192.168.0.47:6443 \
  --image-repository registry.aliyuncs.com/google_containers   \
  --kubernetes-version v1.20.4   \
  --service-cidr=10.10.0.0/16   \
  --pod-network-cidr=10.244.0.0/16  \
  --upload-certs

[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1] and IPs [10.10.0.1 192.168.0.47]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master1] and IPs [192.168.0.47 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master1] and IPs [192.168.0.47 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 67.003009 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
419fb3fbdadc2ee186a8b575f474496d95d4ca4a369cb1289c60fd3d4d96ac17
[mark-control-plane] Marking the node master1 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: y6tzxw.8ltzsp8pekc3fzbg
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.0.47:6443 --token y6tzxw.8ltzsp8pekc3fzbg \
    --discovery-token-ca-cert-hash sha256:c8d21a9b8cef9e8385e89c5b4439ae4e26454293db3bf8c1ddc4167cec04f743 \
    --control-plane --certificate-key 419fb3fbdadc2ee186a8b575f474496d95d4ca4a369cb1289c60fd3d4d96ac17

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.47:6443 --token y6tzxw.8ltzsp8pekc3fzbg \
    --discovery-token-ca-cert-hash sha256:c8d21a9b8cef9e8385e89c5b4439ae4e26454293db3bf8c1ddc4167cec04f743 

7、master1节点 拷贝admin配置文件

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

8、kubectl get nodes

[root@master1 ~]# kubectl get nodes
NAME      STATUS     ROLES                  AGE     VERSION
master1   NotReady   control-plane,master   2m54s   v1.20.4

9、node1、node2节点添加

[root@node1 ~]# kubeadm join 192.168.0.47:6443 --token y6tzxw.8ltzsp8pekc3fzbg  \
      --discovery-token-ca-cert-hash sha256:c8d21a9b8cef9e8385e89c5b4439ae4e26454293db3bf8c1ddc4167cec04f743
      
[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 19.03
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

kubectl get nodes

[root@master1 ~]# kubectl get nodes
NAME      STATUS     ROLES                  AGE     VERSION
master1   NotReady   control-plane,master   6m14s   v1.20.4
node1     NotReady   <none>                 3s      v1.20.4
node2     NotReady   <none>                 7s      v1.20.4

二、K8S节点网络配置

Kubernetes整个集群所有服务器（Master、Minions）配置Flanneld，操作方法和指令如下：
参考：https://github.com/containernetworking/cni
必须安装pod网络插件，以便pod之间可以相互通信，必须在任何应用程序之前部署网络，CoreDNS不会在安装网络插件之前启动。

1、安装Flanneld网络插件；

Fanneld定义POD的网段为: 10.244.0.0/16，POD容器的IP地址会自动分配10.244开头的网段IP。安装Flanneld网络插件指令如下：

#下载Fanneld插件YML文件；
yum install wget -y
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
#提前下载Flanneld组建所需镜像；
for i in $(cat kube-flannel.yml |grep image|awk -F: '{print $2":"$3}'|uniq );do docker pull $i ;done
#应用YML文件；
kubectl apply -f kube-flannel.yml
#查看Flanneld网络组建是否部署成功；
kubectl -n kube-system get pods|grep -aiE flannel

Kube-flannel.yaml配置文件代码如：
[root@master1 src]# cat kube-flannel.yml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

[root@master1 src]# kubectl -n kube-system get pods
NAME                              READY   STATUS    RESTARTS   AGE
coredns-7f89b7bc75-28khh          1/1     Running   0          45h
coredns-7f89b7bc75-klm29          1/1     Running   0          45h
etcd-master1                      1/1     Running   0          45h
kube-apiserver-master1            1/1     Running   0          45h
kube-controller-manager-master1   1/1     Running   3          45h
kube-flannel-ds-7n2vd             1/1     Running   0          25h
kube-flannel-ds-d4mhm             1/1     Running   0          25h
kube-flannel-ds-tfzqk             1/1     Running   0          25h
kube-proxy-bqtq7                  1/1     Running   0          25h
kube-proxy-kqv2s                  1/1     Running   0          25h
kube-proxy-ss2qx                  1/1     Running   0          25h
kube-scheduler-master1            1/1     Running   2          45h

至此，Kubernetes 的 Master 节点就部署完成了。如果只需要一个单节点的 Kubernetes，现在你就可以使用了。

三、K8S开启IPVS模块

修改kube-proxy的configmap，在config.conf中找到mode参数，改为mode: "ipvs"然后保存：
kubectl -n kube-system get cm kube-proxy -o yaml | sed ‘s/mode: “”/mode: “ipvs”/g’ | kubectl replace -f -
#或者手动修改

kubectl -n kube-system edit cm kube-proxy
kubectl -n kube-system get cm kube-proxy -o yaml | grep mode
    mode: "ipvs"
#重启kube-proxy pod        
kubectl -n kube-system delete pods -l k8s-app=kube-proxy
#确认ipvs模式开启成功
kubectl -n kube-system logs -f -l k8s-app=kube-proxy | grep -iE ipvs
ipvsadm -ln

日志中打印出Using ipvs Proxier，说明ipvs模式已经开启。

四、Dashboard

Kubernetes实现的最重要的工作是对Docker容器集群统一的管理和调度，通常使用命令行来操作Kubernetes集群及各个节点，命令行操作非常不方便，如果使用UI界面来可视化操作，会更加方便的管理和维护。如下为配置kubernetes dashboard完整过程：

1、下载Dashboard配置文件；

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml

\cp recommended.yaml recommended.yaml.bak

2、修改文件recommended.yaml的39行内容，

#因为默认情况下，service的类型是cluster IP，需更改为NodePort的方式，便于访问，也可映射到指定的端口。 两行

 32 kind: Service
 33 apiVersion: v1
 34 metadata:
 35   labels:
 36     k8s-app: kubernetes-dashboard
 37   name: kubernetes-dashboard
 38   namespace: kubernetes-dashboard
 39 spec:
 40   type: NodePort     改了此行
 41   ports:
 42     - port: 443
 43       targetPort: 8443
 44       nodePort: 31001     加了此行
 45   selector:
 46     k8s-app: kubernetes-dashboard
 47 
 48 ---
 49 
 50 apiVersion: v1
 51 kind: Secret
 52 metadata:

3、修改文件recommended.yaml的198行内容，

#因为默认情况下Dashboard为英文显示，可以设置为中文。
- name: ACCEPT_LANGUAGE
value: zh

172 kind: Deployment
173 apiVersion: apps/v1
174 metadata:
175   labels:
176     k8s-app: kubernetes-dashboard
177   name: kubernetes-dashboard
178   namespace: kubernetes-dashboard
179 spec:
180   replicas: 1
181   revisionHistoryLimit: 10
182   selector:
183     matchLabels:
184       k8s-app: kubernetes-dashboard
185   template:
186     metadata:
187       labels:
188         k8s-app: kubernetes-dashboard
189     spec:
190       containers:
191         - name: kubernetes-dashboard
192           image: kubernetesui/dashboard:v2.0.0-rc5
193           imagePullPolicy: Always
194           ports:
195             - containerPort: 8443
196               protocol: TCP
197           env:
198             - name: ACCEPT_LANGUAGE				加了此行及下行
199               value: zh						加了此行
200           args:

4、创建Dashboard服务，指令操作如下：

kubectl apply -f recommended.yaml

5、查看Dashboard运行状态；

kubectl get pod -n kubernetes-dashboard
kubectl get svc -n kubernetes-dashboard

6、基于Token的方式访问，设置和绑定Dashboard权限，命令如下；

#创建Dashboard的管理用户；
kubectl create serviceaccount dashboard-admin -n kube-system
#将创建的dashboard用户绑定为管理用户；
kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
#获取刚刚创建的用户对应的Token名称；
kubectl get secrets -n kube-system | grep dashboard
#查看Token的详细信息;
kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system | grep dashboard |awk '{print $1}')

[root@master1 src]# kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system | grep dashboard |awk '{print $1}')
Name:         dashboard-admin-token-q2skz
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 1c4d03e1-23b6-44d4-aacf-3291b4a39077

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1066 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlY3M1pQMVphU0FGRzNHQUx4b3NiLWdjZHh0am1NYVZWZlNPcHYyM0FrUjgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tcTJza3oiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMWM0ZDAzZTEtMjNiNi00NGQ0LWFhY2YtMzI5MWI0YTM5MDc3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.jCwGC1NwzwSONlnuEhHZWrvVBaDxGH2V3rzaZ22XhdLFj2EeKP5fdlYm1BLyW-Jel2BjDx9tlB11hb6RrpzlMLDKjXsIwZj5B7pNQAMMJ9-9qCej9-bKxGD0irCzm_KJWmnT1xXIz2N-iQv95f687L40eQ8GYA5GGNO758a-QgxBIVa215ZC7Nw06l6S128twC3kAxYxsJKcD8mketbY3VZW_9hVkj9SkHumGaHkIdve85Z4fmdkCjzWAetBqrtAOOS6s2jFgTCiY6sgdZLhPcT32EWwFy-OmXgPAUIHdRRZb3Nv7nGrme-h80rmJahQQue5X3rRMl12ctmaankv6Q

7、通过浏览器访问Dashboard WEB，https://192.168.0.47:31001/，如图所示，输入Token登录即可。

五、测试 -创建一个外部服务

1、创建一个nginx服务

在概况中点击右上角的 + 号 -> ‘从表单中创建’ 点击Deploy

2、测试

由上图 得到 集群ip为：10.10.160.114 内部端口为8888，外部端口为 32324

[root@master1 src]# curl 10.10.160.114:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>

注：
1、新节点加入群集命令

kubeadm token create --print-join-command

2、查看token命令

 kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system | grep dashboard |awk '{print $1}')

3、机器重启后，恢复命令

swapoff -a ;service docker restart;service kubelet restart;docker ps -aq|xargs docker restart

4、报401的话，在界面的dev中编辑

spec:
  volumes:
    - name: kubernetes-dashboard-certs
      secret:
        secretName: kubernetes-dashboard-certs
        defaultMode: 420
    - name: tmp-volume
      emptyDir: {}
  containers:
    - name: kubernetes-dashboard
      image: 'kubernetesui/dashboard:v2.0.0-rc5'
      args:
        - '--auto-generate-certificates'
        - '--namespace=kubernetes-dashboard'
        - '--token-ttl=68400'     增加了此行
      ports:
        - containerPort: 8443
          protocol: TCP
      env:
        - name: ACCEPT_LANGUAGE
          value: zh

5、k8s NFS动态PV中，1.20及之后的版本需要在apiserver中加入一行
vim /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.101
    - --allow-privileged=true
    - --feature-gates=RemoveSelfLink=false  增加了此行
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction

6、修改默认端口限制
The range of valid ports is 30000-32767
vim /etc/kubernetes/manifests/kube-apiserver.yaml 增加一行即可，自定义端口

    - --service-node-port-range=10-32767

7、去除污点

kubectl taint nodes --all node-role.kubernetes.io/master-

--------------------------------END