K8S部署(kubeadm方式)

系统环境

软件	版本
操作系统	CentOS7.8_x64 （mini）
Docker	19-ce
Kubernetes	1.18

节点组件

角色	IP	组件
k8s-master	192.168.104.200
k8s-node1	192.168.104.201
k8s-node2	192.168.104.202

基础环境配置–所有节点

hosts配置

cat >> /etc/hosts << EOF
192.168.104.200 k8s-master
192.168.104.201 k8s-node1
192.168.104.202 k8s-node2
EOF

##　主机名修改

例如master

cat > /etc/hostname << EOF
k8s-master
EOF

系统配置文件sysctl设置

## 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
## 生效
sysctl --system

## 关闭缓存,配置/etc/fstab,永久关闭
swapoff -a

## 时间同步
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ntpdate time.windows.com

## 创建目录结构
mkdir -pv /opt/k8s/{bin,cfg,ssl,logs}

##配置环境变量(根据节点情况)
echo 'export PATH=$PATH:/opt/k8s/bin/' >> /etc/profile
source /etc/profile

节点安装Docker

卸载本机docker

yum remove docker

安装依赖软件

yum install yum-utils device-mapper-persistent-data lvm2

添加docker yum源

阿里镜像源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Docker官方镜像源
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

安装docker

yum list docker-ce --showduplicates | sort -r
yum install docker-ce docker-ce-cli containerd.io

配置阿里云镜像加速

vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"]
}
systemctl restart docker
## 使用docker info查看生效情况

启动docker

## 查看docker版本
docker -v
## 启动docker
systemctl start docker
#  配置docker开机启动
systemctl enable docker

kubeadm安装集群

组件安装(所有节点)

配置yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装组件

yum install  kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0 -y
systemctl enable kubelet

注意版本号,要与后面指定K8S版本号对应

创建集群(Master节点)

kubeadm init \
  --apiserver-advertise-address=192.168.104.200 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.18.0 \
  --service-cidr=10.10.0.0/16 \
  --pod-network-cidr=10.244.0.0/16

由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。

使用kubectl工具：

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes

安装flannel网络

kubectl create -f kube-flannel.yaml

##　Node节点加入集群

kubeadm join 192.168.104.200:6443 --token 12n909.ptwiysemj7834ybh \
    --discovery-token-ca-cert-hash sha256:12ce7522eaac8956410f31aa1c3e489fe0dc2ba978cbfd0a4db7f3a706f1d207

安装flannel网络

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

确保能够访问到quay.io这个registery。

如果Pod镜像下载失败，可以改成这个镜像地址：lizhenliang/flannel:v0.11.0-amd64

安装flannel后,节点依旧为NotReady状态,排查是节点无法加载cni插件,报错如下

no networks found in /etc/cni/net.d

master执行:

scp  /etc/cni/net.d/10-flannel.conflist root@k8s-node1:/etc/cni/net.d/10-flannel.conflist
scp  /etc/cni/net.d/10-flannel.conflist root@k8s-node2:/etc/cni/net.d/10-flannel.conflist

正常情况下不会有上面的报错,办证所有安装组件版本和k8s版本一直,node先加入集群,再安装flannel,基本不会出错

默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，操作如下：

# kubeadm token create
# kubeadm token list
# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
63bca849e0e01691ae14eab449570284f0c3ddeea590f8da988c07fe2729e924

# kubeadm join 192.168.31.61:6443 --token nuja6n.o3jrhsffiqs9swnu --discovery-token-ca-cert-hash sha256:63bca849e0e01691ae14eab449570284f0c3ddeea590f8da988c07fe2729e924

部署Dashboard

下载地址

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

文件内容

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.0.4
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: kubernetes-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-metrics-scraper
  name: kubernetes-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: kubernetes-metrics-scraper
    spec:
      containers:
        - name: kubernetes-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.0
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

版本可以去dockerhub上去查

https://hub.docker.com/r/kubernetesui/dashboard/tags

默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部(kubernetes-dashboard部分), 如下：

vi recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

创建dashboard-admin帐号

cat >> recommended.yaml << EOF
---
# ------------------- dashboard-admin ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
EOF

如果不使用上面的方法授权帐号,也可以使用命令

// 先创建一个帐号

kubectl create serviceaccount dashboard-admin-01 -n kubernetes-dashboard

// 给账号授权角色

kubectl create clusterrolebinding dashboard-admin-01 --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin-01

// 获取角色帐号TOKEN令牌

kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk ‘/dashboard-admin-01/{print $1}’)

相关查询命令:

kubectl -n kubernetes-dashboard get/describe serviceaccount/clusterrolebinding/secret dashboard-admin-01

serviceaccount: 创建帐号

clusterrolebinding: 绑定角色

secret: token相关

部署kubernetes-dashboard

## 部署
kubectl apply -f recommended.yaml

## 查看pod分配节点信息
kubectl get all -n kubernetes-dashboard -o wide

## 查看svc信息
kubectl get -n kubernetes-dashboard svc
NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.0.0.96    <none>        8000/TCP        2m16s
kubernetes-dashboard        NodePort    10.0.0.142   <none>        443:30001/TCP   2m16s

获取TOKEN令牌

kubectl describe secrets -n kubernetes-dashboard dashboard-admin

Name:         dashboard-admin-token-wn89j
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: d8bca1b6-7c35-424c-baa0-e7a70ffb2ffc

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InoyUmRYcjZQSTEwdUJiQV9sUW83VEZ0UVFlbWxzN3JxRXFFUmhhdElpTncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4td244OWoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZDhiY2ExYjYtN2MzNS00MjRjLWJhYTAtZTdhNzBmZmIyZmZjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.gKomAYQjXNz1is6RpeNUhtKBUgNUuotHI87O6YHYvp4xxeCuQimUG-9PY1MKqX-Myc1QOn_h1lTCj5lnOd9nzJsHsbC8DZKlif41lS2qEFtegukBegBkhAtBXMso1F6YwHH4LvWoOXpZMjZx_nbNFaoy38iTH8yJ6ErXCdtKPD3yoptmrTd3tpzjflrQkCNSMryAxgO_4E4DuUCbVYY8qMw_aP5v1hxL-rEX_lB9vTiO3nt1_WIekWTNT57clpxxhIosjAroZSmG9knON0PkqdsL_0JriBgFPmuYKZdc5wOXa-2wKVRuPgIBImo7wXvJ0euMiHjX6hsiufE2dFTtDw

浏览器访问节点https://IP:30001,使用上面生成的token,即可登录kubernetes-dashboard

创建一个示例

kubectl create deployment nginx --image=nginx
kubectl expose deployment/nginx --port=80 --type=NodePort --name=service-nginx

常见问题处理

浏览器不能访问

• 二进制 部署

  注意你部署Dashboard的命名空间（之前部署默认是kube-system，新版是kubernetes-dashboard）

# 1、 删除默认的secret，用自签证书创建新的secret
kubectl delete secret kubernetes-dashboard-certs -n kubernetes-dashboard
kubectl create secret generic kubernetes-dashboard-certs \
--from-file=/opt/k8s/ssl/server-key.pem --from-file=/opt/k8s/ssl/server.pem -n kubernetes-dashboard


# 2、修改 dashboard.yaml 文件，在args下面增加证书两行
       args:
         # PLATFORM-SPECIFIC ARGS HERE
         - --auto-generate-certificates
         - --tls-key-file=server-key.pem
         - --tls-cert-file=server.pem

kubectl apply -f kubernetes-dashboard.yaml

• kubeadm 部署

  注意你部署Dashboard的命名空间（之前部署默认是kube-system，新版是kubernetes-dashboard）

# 1、删除默认的secret，用自签证书创建新的secret
kubectl delete secret kubernetes-dashboard-certs -n kubernetes-dashboard
kubectl create secret generic kubernetes-dashboard-certs \
--from-file=/etc/kubernetes/pki/apiserver.key --from-file=/etc/kubernetes/pki/apiserver.crt -n kube-system


# 2、修改 dashboard的yaml 文件，在args下面增加证书两行
       args:
         # PLATFORM-SPECIFIC ARGS HERE
         - --auto-generate-certificates
         - --tls-key-file=apiserver.key
         - --tls-cert-file=apiserver.crt

kubectl apply -f kubernetes-dashboard.yaml

Token过期

## 创建token
kubeadm token create
## 查看token
kubeadm token list
## 获取token
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt |openssl rsa -pubin -outform der 2>/dev/null |openssl dgst -sha256 -hex |sed 's/^.* //'

12ce7522eaac8956410f31aa1c3e489fe0dc2ba978cbfd0a4db7f3a706f1d207

kubeadm join 192.168.104.200:6443 --token 12n909.ptwiysemj7834ybh \
    --discovery-token-ca-cert-hash sha256:12ce7522eaac8956410f31aa1c3e489fe0dc2ba978cbfd0a4db7f3a706f1d207

环境清理

kubeadm reset

在master上执行就是清除master上的,在node上执行就是清除node的