虚拟机搭建k8s集群

虚拟机搭建k8s集群

实习无聊,搭建个k8s集群玩玩

机器资源介绍

系统	ip	hostname	备注
centos7	192.168.17.129	node1	虚拟机
centos7	192.168.17.130	node2	克隆虚拟机
centos7	192.168.17.131	node3	克隆虚拟机

机器时间同步

yum install -y ntpdate
ntpdate -u ntp.api.bz

修改/etc/hosts文件

192.168.17.129 node1
192.168.17.130 node2
192.168.17.131 node3

修改hostname

vim /etc/sysconfig/network 

NETWORKING=yes 
NETWORKING_IPV6=yes 
HOSTNAME=node1  #每个机器对应起来node1 node2 node3

关闭防火墙，开机不自启动

systemctl stop firewalld
systemctl disable firewalld

禁用selinux

vim /etc/selinux/config
修改以下配置
SELINUX=disabled

关闭swap

swapoff -a  # 临时
vim /etc/fstab #永久
#注释下面这行
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

修改/etc/sysctl.conf文件

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
然后执行命令
[root@localhost ~]# sysctl -p
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

配置ipvs功能

接下来还需要确保各个节点上已经安装了ipset软件包

yum -y install ipset

为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm

yum -y install ipvsadm

如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式

添加需要加载的模块写入脚本文件

cat <<EOF>  /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

为脚本文件添加执行权限

chmod +x /etc/sysconfig/modules/ipvs.modules

执行脚本文件

/bin/bash /etc/sysconfig/modules/ipvs.modules

查看对应的模块是否加载成功

lsmod | grep -e ip_vs -e nf_conntrack_ipv4

docker安装

yum install docker-ce
#开启Docker服务并开机自启
sudo systemctl start docker.service
sudo systemctl enable docker.service

安装kubeadm、kubelet、kubectl

安装kubernetes的yum源，采用阿里的镜像

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

查看可用的kubernetes版本

yum makecache fast
yum list kubeadm --showduplicates | sort -r
#安装最新版本 1.23.1
yum -y install kubeadm kubectl kubelet

验证安装

yum list installed | grep kube

准备kebernetes所需组件的镜像

kubernetes正常工作需要 kubernetes-proxy，kubernetes-apiserver，kubernetes-scheduler，kubernetes-controller-manager以及etcd，pause，coredns，flannel等组件的镜像，由于这些镜像都在google的镜像仓库中，下载较慢，所以需要使用阿里的镜像进行下载，下载完成后通过 docker tag命令修改镜像名称，以便kubernetes能够发现和使用

查看对应版本

[root@localhost ~]# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.23.1
k8s.gcr.io/kube-controller-manager:v1.23.1
k8s.gcr.io/kube-scheduler:v1.23.1
k8s.gcr.io/kube-proxy:v1.23.1
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.23.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.23.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.23.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.23.1
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.5.1-0
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6
docker pull quay.io/coreos/flannel:v0.15.1-amd64

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.23.1  k8s.gcr.io/kube-apiserver:v1.23.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.23.1  k8s.gcr.io/kube-proxy:v1.23.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.23.1  k8s.gcr.io/kube-controller-manager:v1.23.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.23.1  k8s.gcr.io/kube-scheduler:v1.23.1
docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.5.1-0  k8s.gcr.io/etcd:3.5.1-0
docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6  k8s.gcr.io/pause:3.6
docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6  k8s.gcr.io/coredns:v1.8.6

配置kubelet的cgroup drive

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 

加入

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

重新加载

systemctl daemon-reload 

# 编辑/etc/sysconfig/kubelet，添加下面的配置
KUBELET_CGROUP_ARGS="--cgroup-driver=systemd"
KUBE_PROXY_MODE="ipvs"

设置kubelet开机自启

systemctl enable kubelet

---

初始化集群

在node1 上操作(主节点master)

# 由于默认拉取镜像地址 k8s.gcr.io 国内无法访问，这里指定阿里云镜像仓库地址
# apiserver-advertise-address 需要写自己的ip
 kubeadm init \
--apiserver-advertise-address=192.168.17.129 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version=v1.23.1 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 

–apiserver-advertise-address=192.168.17.129 这个参数就是master主机的IP地址，例如我的Master主机的IP是：192.168.17.129

–image-repository=registry.aliyuncs.com/google_containers 这个是镜像地址，由于国外地址无法访问，故使用的阿里云仓库地址：registry.aliyuncs.com/google_containers

–kubernetes-version=v1.23.1 这个参数是下载的k8s软件版本号

–service-cidr=10.96.0.0/12 这个参数后的IP地址直接就套用10.96.0.0/12 ,以后安装时也套用即可，不要更改

–pod-network-cidr=10.244.0.0/16 k8s内部的pod节点之间网络可以使用的IP段，不能和service-cidr写一样，如果不知道怎么配，就先用这个10.244.0.0/16

总之，只修改加粗的两个地方(master主机IP，k8s软件版本)

加入node1的指令，在执行完kubeadm init 之后会自动在控制台打印
接下来,在从节点node2,node3运行这个指令就能加入node1了

设置在kubernetes 本机的环境变量

将主节点（master节点）中的【/etc/kubernetes/admin.conf】文件拷贝到从节点相同目录下:

scp -r /etc/kubernetes/admin.conf node2:/etc/kubernetes/admin.conf
scp -r /etc/kubernetes/admin.conf node3:/etc/kubernetes/admin.conf
配置环境变量：
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
立即生效：
source ~/.bash_profile

初始化失败情况1

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.

检查docker 状态,重启docker

systemctl daemon-reload
systemctl restart docker

重新进行kubeadm初始化

kubeadm reset

初始化集群coredns容器一直处于pending状态

缺少网络插件，部署flannel网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

手动下载https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
内容如下

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: rancher/mirrored-flannelcni-flannel:v0.16.1
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: rancher/mirrored-flannelcni-flannel:v0.16.1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

查看集群是否加入成功：

kubectl get nodes