almalinux部署kubernetes生产环境（vmware虚拟机环境）

参考：[1]使用部署工具安装 Kubernetes | Kubernetes

           [2]linux安装部署k8s(kubernetes)和解决遇到的坑 - 简书

kubernetes之前在阿里云上直接使用过，这次尝试下与centos同源的almalinux系统，搭建本地测试的kubernetes集群。

【1】虚拟机安装

使用vmware创建四台虚拟机，在almalinux官网下载系统镜像，可选择完整版镜像，大概9个G多(AlmaLinux OS - Forever-Free Enterprise-Grade Operating System)，硬件配置4核4GB内存40G硬盘，almalinux当前vmware没有直接支持，本文在安装时选择red hat enterprise linux 4 64位，almalinux版本为8.5对应red hat 4.x版本，可在官网查询。

注意在创建过程中，自定义硬件处理器设置至少两个（kubeadm支持需要），网络适配器选择桥接模式（方便四台虚拟机组网配置，也可参考其他组网方法）。 

做好初始配置后即可继续安装虚拟机，系统安装时选择中文总是出现卡住的问题，选择英文安装就没有问题了，选择存储空间、创建用户、创建root密码、连接以太网络，等待安装完成后重启验证，依次完成四台虚拟机安装。

【2】配置网络

该环节主要任务是四台机器组成一个局域网，可以互相ping通并可以访问公网。

桥接模式将虚拟机的虚拟网络适配器与主机的物理网络适配器进行交接，虚拟机中的虚拟网络适配器可通过主机中的物理网络适配器直接访问到外部网络，比较适合组网的配置。首先在宿主机查看IP地址，本文中宿主机为win，在终端执行ipconfig即可查看，假设IP为192.168.111.133。

进入到虚拟机中配置，可使用图形界面配置网络，本文使用命令行配置，虚拟机IP地址按照修改宿主机IP的最后一位设置，保证在同一网段，网关、DNS等跟随宿主机配置。

[root@localhost]# vim  /etc/sysconfig/network-scripts/ifcfg-ens33

#网卡名称可能不同，在network-scripts目录下查看，修改以下字段
ONBOOT=yes
IPADDR=192.168.111.14
NETMASK=255.255.255.0
GATEWAY=192.168.111.254
DNS1=202.101.172.35
DNS2=202.101.172.47
PREFIX=24
BOOTPROTO=static

配置完成后，重启Network，查看虚拟机ip是否已经修改完成，并尝试ping主机、其他几台虚拟机、公网，均可连通，配置完成。

[root@localhost]# nmcli c reload
[root@localhost]# ifconfig
[root@localhost]# ping www.baidu.com
[root@localhost]# ping 192.168.111.133

【3】kubernetes基础配置

参考[2]进行本文的kubernetes配置，每台虚拟机都需要做的一部分配置。

设置主机名称，主机名称就是在k8s集群中的节点名称。

[root@localhost]# hostnamectl set-hostname master
[root@localhost]# hostname

禁用SELinux并关闭系统swap，具体原因查看参考[1]

[root@localhost]# vim /etc/sysconfig/selinux
SELINUX=disabled   #永久禁用
[root@localhost]# swapoff -a
[root@localhost]# free -m    #确认swap是否已关闭
[root@localhost]# vim /etc/sysctl.conf   #永久调整swappiness参数
vm.swappiness=0
[root@localhost]# sysctl -p  #使配置生效

配置iptables

[root@localhost]# cat <<EOF > /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
[root@localhost]# sysctl --system

安装docker

[root@localhost]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@localhost]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@localhost]# yum install -y docker-ce --allowerasing
[root@localhost]# systemctl start docker
[root@localhost]# systemctl enable docker
[root@localhost]# docker version

配置docker国内镜像加速，修改systemd作为驱动

[root@localhost]# vim /etc/docker/daemon.json
{
 "registry-mirrors" : ["https://q5bf287q.mirror.aliyuncs.com", "https://registry.docker-cn.com","http://hub-mirror.c.163.com"],
 "exec-opts": ["native.cgroupdriver=systemd"]
}
[root@localhost]# systemctl daemon-reload
[root@localhost]# systemctl restart docker

配置k8s的yum源

[root@localhost]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

配置生效

[root@localhost]# yum clean all
[root@localhost]# yum makecache
#添加阿里源可能提示索引检查失败，加上参数--nogpgcheck

查看可用的kubernetes版本，本文选择1.23.0-0版本

[root@localhost]# yum list kubectl --showduplicates | sort -r

安装kubelet，kubeadm，kubectl

#可指定版本，也可删去后缀安装最新版本
[root@localhost]# yum install -y kubelet-1.23.0-0 kubeadm-1.23.0-0 kubectl-1.23.0-0
[root@localhost]# kubeadm version
[root@localhost]# kubelet --version

启动kubelet并设置开机启动

[root@localhost]# systemctl daemon-reload
[root@localhost]# systemctl start kubelet
[root@localhost]# systemctl enable kubelet   #设置自启动
[root@localhost]# systemctl is-enabled kubelet
[root@localhost]# systemctl status kubelet   #查看状态
[root@localhost]# journalctl -xefu kubelet   #查看日志

【4】配置master节点

开放master节点防火墙端口

[root@localhost]# firewall-cmd --zone=public --add-port=2379/tcp --permanent
#客户端监听
[root@localhost]# firewall-cmd --zone=public --add-port=2380/tcp --permanent
#节点间内部通信
[root@localhost]# firewall-cmd --zone=public --add-port=6443/tcp --permanent
#kube-apiserver端口
[root@localhost]# systemctl restart firewalld.service
[root@localhost]# firewall-cmd --zone=public --list-ports

初始化k8s集群,注意指定master节点的IP和kubernetes版本

[root@localhost]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.111.2 --kubernetes-version=v1.23.0 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.1.0.0/16

初始化完成后，注意看窗口跳出的信息，记录下最后两行kubeadm join的信息，这两行是其他节点加入集群的执行命令。然后执行跳出信息的上面几行命令,添加集群配置。

 [root@localhost]# mkdir -p $HOME/.kube
 [root@localhost]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 [root@localhost]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时，即可查看到k8s集群节点,可以看到，节点还处于NotReady状态。

[root@localhost]# kubectl get nodes

原因是cni容器网络模型没有准备好，安装flannel网络插件，需要在GitHub上curl(flannel/kube-flannel.yml at master · flannel-io/flannel · GitHub)下来，本文采用创建文件直接复制的方式，yml文件内容也贴了出来

[root@localhost]# mkdir flannel && cd flannel
[root@localhost]# vim kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: rancher/mirrored-flannelcni-flannel:v0.16.1
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: rancher/mirrored-flannelcni-flannel:v0.16.1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
[root@localhost]# kubectl apply -f kube-flannel.yml
[root@localhost]# kubectl get nodes

查看节点，可以发现已经变成Ready状态了。

【5】节点配置

节点加入集群只需要配置join信息即可，即初始化集群输出的kubeadm join命令，复制输入即可

[root@localhost]# kubeadm join X.X.X.X:X --token X.X....

如果token过期或token忘记了，可以到master节点再次生成命令。

[root@master]# kubeadm token generate
2i13l6.t2ub08rgbbhcikut
[root@master]# kubeadm token create 2i13l6.t2ub08rgbbhcikut --print-join-command --ttl=0
kubeadm join x.x.x.x:x --token 2i13l6.t2ub08rgbbhcikut --discovery-token-ca-cert-hash sha256:c82973baff7a8c00d4cbaa83cf8d1cff3620a15f786878833edd98a8b8457703 

此时执行kubectl命令会报错

[root@localhost]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?

kubectl命令需要使用kubernetes-admin来运行，解决方法如下，将主节点中的【/etc/kubernetes/admin.conf】文件拷贝到从节点相同目录下，然后配置环境变量并使其生效

[root@localhost]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@localhost]# source ~/.bash_profile

此时在从节点上即可使用kubectl命令，依次配置剩余虚拟机，开始kubernetes之旅