Kubernetes实战 | minikube部署kubernetes集群

前言

本文使用minkube部署本地单机版kubernetes集群，一般可用做开发或测试环境。
部署基于一台可以访问互联网的linux服务器，我这里是部署在笔记本搭建的linux虚机上。本文大部分步骤都是整合官网或参考其他部署文章而来，是一个亲身实践成功部署的流程，有需要的同学可以参考下。

环境准备

需要一台至少2核2g内存，硬盘20G的linux服务器，并安装有docker容器，能连互联网。
本文linux环境如下

前置环境	规格
linux版本	CentOS Linux release 7.3.1611 (Core)
cpu	2核
内存	2g
硬盘	20g
容器	docker 19.03.11

基础软件安装

1. 下载阿里云的yum源repo文件

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

2. 安装基本软件包

yum install wget net-tools ntp git ‐y

3. 同步系统时间

ntpdate 0.asia.pool.ntp.org

4. 配置Docker, K8S的阿里云yum源

cat >>/etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5. 桥接流量

确保 br_netfilter 模块被加载。
通过运行 lsmod | grep br_netfilter 来确认br_netfilter已经加载。若要加载该模块，可执行 sudo modprobe br_netfilter。
将桥接的IPv4流量传递到iptables
为了让你的 Linux 节点上的 iptables 能够正确地查看桥接流量，你需要确保在你的 sysctl 配置中将 net.bridge.bridge-nf-call-iptables 设置为 1。执行：

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

写入配置：

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

6. 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

7. 关闭Selinux

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

8. 关闭swap

swapoff -a
yes | cp /etc/fstab /etc/fstab_bak

vi /etc/fstab #注释下面这一行

#/dev/mapper/centos-swap swap                    swap    defaults        0 0

minikube 部署集群

前置设置

使用 systemd 作为 cgroup 的驱动

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://registry.cn-hangzhou.aliyuncs.com"]
}
EOF

# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

设置主机名

本文设置成master

hostnamectl set-hostname master
hostname

开始部署

1. 下载安装Minikube

在Minikube官网上，根据选择的系统及版本，官网会生成下载安装命令

# 下包
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
# 安装
sudo install minikube-linux-amd64 /usr/local/bin/minikube

本文版本v1.26.0
也可通过下载指定版本tar包然后运行安装步骤

安装完成后可通过minikube version命令查看 minikube 的版本

2. 启动集群

minikube不建议使用root账号启动，可以建立非root账号，或者选择使用如下命令启动

minikube start --image-mirror-country=cn --driver=none

3. 查看日志

这里先统一列举几个查看状态、日志的常用命令

# 查看节点状态
kubectl get node 
# 查看pod状态
kubectl get pod -A
# 查看指定命名空间pod,比如系统命名空间kube-system
kubectl get po -n kube-system
# 查看pod日志
kubectl logs --tail 100 -f   pod名称 -n 命名空间
kubectl describe pod pod名称 -n 命名空间
# 查看日志
systemctl status kubelet
journalctl -xeu kubelet
# 查看节点日志，比如下面是master节点日志
kubectl describe  node master -n kube-system | grep -A 20 "Conditions"
# 重启kubelet
systemctl restart kubelet

4. 问题解决

问题一 X Exiting due to GUEST_MISSING_CONNTRACK: Sorry, Kubernetes 1.20.2 requires conntrack to be installed in root’s path

安装conntrack

yum install conntrack

然后再次启动

minikube start --image-mirror-country=cn --driver=none

问题二 Failed to enable unit: Unit file cri-docker.socket does not exist

需要安装cri-dockerd

下载

拷贝cri-dockerd文件到服务器并安装

rz 

tar -xvf cri-dockerd-0.2.3.amd64.tgz

cp cri-dockerd/cri-dockerd /usr/bin/

chmod +x /usr/bin/cri-dockerd

# 确认已安装版本
cri-dockerd --version

配置启动文件

创建cri-docker.service配置文件

vi /lib/systemd/system/cri-docker.service

输入以下内容

[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

创建cri-docker.socket配置文件

vi /lib/systemd/system/cri-docker.socket

输入以下内容

[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

启动CRI-docker

systemctl daemon-reload

systemctl start docker.service

systemctl enable cri-docker

systemctl status docker.service

再次启动

minikube start --image-mirror-country=cn --driver=none

问题三 Exiting due to RUNTIME_ENABLE: Temporary Error: sudo crictl version: exit status 1

需要安装crictl，官网步骤

VERSION="v1.24.1"

wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz

sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin

rm -f crictl-$VERSION-linux-amd64.tar.gz

再次启动

minikube start --image-mirror-country=cn --driver=none

问题四 sudo: crictl: command not found

未配置环境变量

vi /etc/sudoers

找到"secure_path"一行，当你执行 sudo 命令时，"secure_path"中包含的路径将被当做默认 PATH 变量使用。没有可用直接添加下面这行到文件末尾

Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

再次启动

minikube start --image-mirror-country=cn --driver=none

启动成功。日志大概如下，下面这种图是二次重启成功，和第一次可能有小的差别

问题五 节点状态notReady

minikube内部是可以使用kubectl的，但是如果想直接操作可以参照官网如下命令设置

alias kubectl="minikube kubectl --"

或者

ln -s $(which minikube) /usr/local/bin/kubectl

安装flannel插件
创建flannel.yaml文件

vi flannel.yaml

输入下面内容

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: registry.cn-beijing.aliyuncs.com/qingfeng666/flannel:v0.13.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: registry.cn-beijing.aliyuncs.com/qingfeng666/flannel:v0.13.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

安装

kubectl apply -f flannel.yaml

查看是否成功创建flannel网络

# 查看命令
ifconfig |grep flan
# 返回如下结果表示成功
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450

重启

systemctl restart kubelet

问题六 network plugin is not ready: cni config uninitialized

查看集群状态仍然是notReady,且有pod为pending状态。

kubectl get pod -A 

查看详细日志

kubectl describe  node master04 -n kube-system | grep -A 20 "Conditions"

发现是 network plugin is not ready: cni config uninitialized
需要安装cni-plugins
下载

sudo mkdir -p /opt/cni/bin
cd /opt/cni/bin
# 上传cni-plugins-linux-amd64-v0.8.6.tgz包
rz
#然后将其解压即可。
tar -xzvf cni-plugins-linux-amd64-v0.8.6.tgz

查看集群状态

kubectl get pod -A 

5. 参考文章

https://www.jianshu.com/p/2eb952ffe89b
https://blog.csdn.net/hello_1995/article/details/109222650
https://blog.csdn.net/qq_34988341/article/details/106305567?utm_medium=distribute.pc_feed_404.none-task-blog-2_defaultBlogCommendFromBaidu_{Rate-2-106305567-blog-null.pc_404_mixedpudn&depth_1-utm_source=distribute.pc_feed_404.none-task-blog-2}default_{BlogCommendFromBaidu}Rate-2-106305567-blog-null.pc_404_mixedpud