在virtualbox上搭建kubenetes集群入门

Java红桃峰峰日拱一卒

已于 2022-09-16 11:57:31 修改

阅读量801

点赞数

分类专栏： kubenetes 文章标签： java kubenetes docker kubectl

于 2022-08-23 15:03:07 首次发布

本文链接：https://blog.csdn.net/u011159417/article/details/126485288

版权

kubenetes 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

一、启动虚拟机，准备环境

使用Dockerfile文件自动创建virtualbox

Vagrant.configure("2") do |config|
    (1..3).each do |i|
        config.vm.define "k8s-node#{i}" do |node|
            #设置虚拟机的box
            node.vm.box="centos/7"
            
            #设置虚拟机的主机名
            node.vm.hostname="k8s-node#{i}"
            
            #设置虚拟机的ip
            node.vm.network "private_network",ip:"192.168.56.#{99+i}",netmask:"255.255.255.0"
            
            #VirtualBox相关配置
            node.vm.provider "virtualbox" do |v|
                #设置虚拟机的名称
                v.name="k8s-node#{i}"
                #设置虚拟机的内存大小
                v.memory=4096
                #设置虚拟机的cpu个数
                v.cpus=4
            end
        end
    end
end

进入此window 文件目录

鼠标放入上方文件目录处输入 cmd

如图：第一步

第二步

第三步，然后回车，
在window命令窗口中输入 vagrant up，

virtualbox就会自动创建三台虚拟机

然后打开virtualbox的管理--> 全局设定 --> 网络 -->创建NatNetwork

管理 --> 主机网络管理器，保留一个 host-only 启用状态 192.168.56.1 255.255.255.0

（以下操作，三台机器都要操作）

默认用户名：root

默认密码是：vagrant

使用virtualbox的正常启动

在命令终端输入

vi  etc/ssh/sshd_config

找到 passwordAuthentication no 将no修改为yes

在使用xshell登录就是使用密码，而不是public key了

二、服务器环境初始化

关闭防火墙（清空防火墙规则）

systemctl stop firewalld
systemctl disable firewalld

关闭selinux（关闭安全机制）

setenforce 0                   #临时关闭
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  #永久

关闭swap（必须关闭）

  swapoff -a                  #临时关闭
  sed -ri 's/.*swap.*/#&/' /etc/fstab            #永久

使用 hostname 查看每个机器的机器名

 192.168.56.100  k8s-node1
 192.168.56.101  k8s-node2
 192.168.56.102  k8s-node3

添加host：（在host文件末尾添加）
使用 vi /etc/hosts 打开文件，加入如下内容:

 192.168.56.100  k8s-node1
 192.168.56.101  k8s-node2
 192.168.56.102  k8s-node3

将桥接的ipv4流量传递到iptables的链：

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system   #生效

（以上操作，三台机器都要操作）

三、安装docker

删除之前安装的docker

sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine

安装必须的环境

yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

设置docker repo的yum位置

sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

安装docker 以及docker-cli

sudo yum install -y docker-ce docker-ce-di containerd.io

配置docker加速

sudo mkdir -p /etc/docker


sudo tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors":["https://82m9ar63.mirror.aliyuns.com"]
}
EOF


sudo systemctl daemon-reload
sudo systemctl restart docker

启动docker 和设置docker开机自启动

systemctl enable docker

加速阿里云yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
https://mirrors.aliyun.com/kubemetes/yum/doc/rpm-package-key.gpg
EOF

四、安装kuberadm ,kubelet 和kubectl

yum list|grep kube
yum install -y kubelet-1.17.3 kubeadm-1.17.3 kubectl-1.17.3

执行上面两个yum命令都报错如下：

https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirrors.aliyun.com; Unknown error"
Trying other mirror.

One of the configured repositories failed (Kubernetes Repo),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled
yum --disablerepo=kubernetes ...

4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:

yum-config-manager --disable kubernetes
or
subscription-manager repos --disable=kubernetes

5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:

yum-config-manager --save --setopt=kubernetes.skip_if_unavailable=true

failure: repodata/repomd.xml from kubernetes: [Errno 256] No more mirrors to try.
https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirrors.aliyun.com; Unknown error"

这里是因为上面配置的docker加速中registry-mirrors对应的地址错了
正确的：https://82m9ar63.mirror.aliyuns.com
错误的：https://82m9ar63.mirror.aliyuncs.com

sudo tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors":["https://82m9ar63.mirror.aliyuns.com"]
}
EOF

systemctl enable kubelet

systemctl start kubelet

systemctl status kubelet

五、部署k8s-master

1、master节点初始化

主节点镜像脚本
master_images.sh

#!/bin/bash

images=(
    kube-apiserver:v1.17.3
    kube-proxy:v1.17.3
    kube-controller-manager:v1.17.3
    kube-scheduler:v1.17.3
    coredns:1.6.5
    etcd:3.4.3-0
    pause:3.1
)

for imageName in ${images[@]}; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
#   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done

执行 sh master_images.sh

docker images 查看

使用 ip addr
查看eth0:的ip是： 10.0.2.15

kubeadm 初始化

 kubeadm init \
--apiserver-advertise-address=10.0.2.15 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.244.0.0/16

报错：
W0819 06:30:25.408387 7990 validation.go:28] Cannot validate kubelet config - no validator is available
W0819 06:30:25.408444 7990 validation.go:28] Cannot validate kube-proxy config - no validator is available
[init] Using Kubernetes version: v1.17.3
[preflight] Running pre-flight checks
   [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
   [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.17. Latest validated version: 19.03
error execution phase preflight: [preflight] Some fatal errors occurred:
   [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

以上错误的原因重点看这条：
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

使用下面的命令将ip_forward设置为1即可

sysctl -w net.ipv4.ip_forward=1

然后再次执行

kubeadm init \
--apiserver-advertise-address=10.0.2.15 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.244.0.0/16

成功，下面这个日志信息要先保存下来，后面使用

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.2.15:6443 --token vt4rgu.57cgqbkj1hm7l79b \
    --discovery-token-ca-cert-hash sha256:1668b87c5aaa89bf1c60e4ff3cf59ed10ad54633e3c1df686d13fc8947cda96e

执行

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装pod网络插件（cni）

$ kubectl apply -f \
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f \
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

这里我们在上面这个kube-flannel.yum文件中增加了一些信息，详见：

https://download.csdn.net/download/u011159417/86514665

执行

kubectl apply -f kube-flannel.yml

执行 kubectl get pods
No resources found in default namespace.

[root@k8s-node1 k8s]# kubectl get ns
NAME STATUS AGE
default Active 154m
kube-node-lease Active 154m
kube-public Active 154m
kube-system Active 154m

查看pod是否是running状态
[root@k8s-node1 k8s]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7f9c544f75-pxcdk 1/1 Running 0 155m
kube-system coredns-7f9c544f75-qlpc5 1/1 Running 0 155m
kube-system etcd-k8s-node1 1/1 Running 0 155m
kube-system kube-apiserver-k8s-node1 1/1 Running 0 155m
kube-system kube-controller-manager-k8s-node1 1/1 Running 0 155m
kube-system kube-flannel-ds-amd64-46kpj 1/1 Running 0 4m13s
kube-system kube-proxy-jg9pf 1/1 Running 0 155m
kube-system kube-scheduler-k8s-node1 1/1 Running 0 155m

查看节点信息
[root@k8s-node1 k8s]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready master 157m v1.17.3

2、从节点加入主节点

在k8s-node2 和 k8s-node3上执行
上面执行

kubeadm join 10.0.2.15:6443 --token vt4rgu.57cgqbkj1hm7l79b \
    --discovery-token-ca-cert-hash sha256:1668b87c5aaa89bf1c60e4ff3cf59ed10ad54633e3c1df686d13fc8947cda96e

[root@k8s-node2 k8s]# kubeadm join 10.0.2.15:6443 --token 8mgmlh.cgtgsp3samkvpksn \
> --discovery-token-ca-cert-hash sha256.3cf99aa2e6bfc114c5490a7c6dffcf200b670af21c5a662c299b6de606023f85
W0819 09:25:22.037037 15679 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
   [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
   [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.17. Latest validated version: 19.03
error execution phase preflight: [preflight] Some fatal errors occurred:
   [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

上面的报错
使用下面的命令将ip_forward设置为1即可

sysctl -w net.ipv4.ip_forward=1

[root@k8s-node2 k8s]# kubeadm join 10.0.2.15:6443 --token 8mgmlh.cgtgsp3samkvpksn --discovery-token-ca-cert-hash sha256.3cf99aa2e6bfc114c5490a7c6dffcf200b670af21c5a662c299b6de606023f85
W0819 09:26:20.274076 15869 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.17. Latest validated version: 19.03
error execution phase preflight: couldn't validate the identity of the API Server: invalid public key hash, expected "format:value"
To see the stack trace of this error execute with --v=5 or higher

报这个错误说明之前执行
kubeadm init
命令后面的日志你没有保存下来，或者复制错误了，导致的

如果发生了这种情况，重新从第五步执行一下

查看kube-system命名空间中pod状态