虚拟机搭建 k8s 集群环境

前期准备

虚拟机联网配置

虚拟机设置静态IP

1. 设置网卡连接方式

2. 准备两台虚拟机

机器 IP	hostname
192.168.243.134	k8s-master
192.168.243.136	k8s-node1

3. 配置静态IP (master 和 node 都要配置)

vi /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="d00801e4-2486-4c94-9402-018fdb60fc77"
DEVICE="ens33"
ONBOOT="yes"


#### 以下是静态ip配置时新增内容
IPADDR="192.168.243.134"  # 此处给自己设置一个静态ip, master: 192.168.243.134 node1: 192.168.243.136. 根据自己的实际ip进行填写
NETMASK="255.255.255.0" 
GATEWAY="192.168.243.1" # 网关配置.没有什么特殊需求,前三位和 IPADDR 前三位保持一致. 最后一位使用 1. 如: 192.168.243.1
DNS1="223.5.5.5" # 固定不变

####### 使用命令重启网络
service network restart

• 关闭防火墙 (master 和 node 都要执行)

systemctl stop firewalld
systemctl disable firewalld

• 关闭selinux (master 和 node 都要执行)

setenforce 0  # 临时关闭
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 永久关闭

• 关闭swap(master/node)

swapoff -a    # 临时关闭；关闭swap主要是为了性能考虑
free             # 可以通过这个命令查看swap是否关闭了
sed -ri 's/.*swap.*/#&/' /etc/fstab  # 永久关闭

• 修改host文件(master 和 node 都要执行)

vi /etc/hosts

192.168.243.134 master.com  master  # 主机 host
192.168.243.136 node1.com   node1  # node host
199.232.28.133 raw.githubusercontent.com  # 后面的步骤会在这个网站进行文件下载,如果无法下载文件,可以在 host 文件中添加这个地址

• 修改主机名(master 和 node 都要执行)

master 调整

hostnamectl set-hostname master ##重启后永久生效

node1 调整

hostnamectl set-hostname node1  ##重启后永久生效

• 桥接设置(master 和 node 都要执行)

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF


sysctl --system

• 添加阿里云源 (master 和 node 都要执行)

rm -rf /etc/yum.repos.d/*
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

• 安装常用包 (master 和 node 都要执行)

yum install vim bash-completion net-tools gcc -y

• 安装 docker (master 和 node 都要执行)

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum -y install docker-ce

• 添加aliyundocker仓库加速器(master 和 node 都要执行)

mkdir -p /etc/docker

tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://fl791z1h.mirror.aliyuncs.com"]
}
EOF

systemctl daemon-reload

systemctl restart docker

安装kubectl、kubelet、kubeadm(master 和 node 都要执行)

• 添加阿里kubernetes源(master 和 node 都要执行)

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

• 安装 kubectl、kubelet、kubeadm (master 和 node 都要执行)

yum install kubectl kubelet kubeadm

#### 此时，还不能启动kubelet，因为此时配置还不能.
systemctl enable kubelet

初始化k8s集群(仅 master 需要安装)

• 查看 kubeadm 版本

[root@localhost ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:25:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

## 当前版本: GitVersion:"v1.20.2"

• 初始化集群

kubeadm init --kubernetes-version=1.20.2  \
--apiserver-advertise-address=192.168.243.134   \
--image-repository registry.aliyuncs.com/google_containers  \
--service-cidr=10.10.0.0/16 --pod-network-cidr=10.122.0.0/16

这里注意有两个参数是需要调整的
–kubernetes-version: 引用 kubeadm 的版本号
–apiserver-advertise-address: 需要替换为 master ip 地址

创建kubectl(仅 master 需要安装)

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

• 查看节点 pod

[root@localhost ~]# kubectl get node
NAME                    STATUS   ROLES                  AGE    VERSION
localhost.localdomain   NotReady    control-plane,master   139m   v1.20.2


[root@localhost ~]# kubectl get pod --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-7f89b7bc75-4cvgf         0/1     Pending   0          2m
kube-system   coredns-7f89b7bc75-nfdvg         0/1     Pending   0          2m
kube-system   etcd-master                      1/1     Running   0          2m10s
kube-system   kube-apiserver-master            1/1     Running   0          2m10s
kube-system   kube-controller-manager-master   1/1     Running   0          2m10s
kube-system   kube-proxy-hk47n                 1/1     Running   0          2m
kube-system   kube-scheduler-master            1/1     Running   0          2m10s

node节点为NotReady，因为corednspod没有启动，缺少网络pod

安装calico网络(仅 master 需要安装)

[root@localhost ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

#### 执行结果
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

• 查看pod和node

命令可以等一会执行,因为部分服务在启动,状态不会立马变更为 Running 状态

[root@localhost ~]# kubectl get pod --all-namespaces

#### 执行结果
NAMESPACE              NAME                                            READY   STATUS    RESTARTS   AGE
kube-system            calico-kube-controllers-744cfdf676-djfcb        1/1     Running   0          135m
kube-system            calico-node-r8g7m                               1/1     Running   0          135m
kube-system            coredns-7f89b7bc75-2c8c4                        1/1     Running   0          142m
kube-system            coredns-7f89b7bc75-zl49d                        1/1     Running   0          142m
kube-system            etcd-localhost.localdomain                      1/1     Running   0          142m
kube-system            kube-apiserver-localhost.localdomain            1/1     Running   0          142m
kube-system            kube-controller-manager-localhost.localdomain   1/1     Running   0          142m
kube-system            kube-proxy-lvwhk                                1/1     Running   0          142m
kube-system            kube-scheduler-localhost.localdomain            1/1     Running   0          142m
kubernetes-dashboard   dashboard-metrics-scraper-79c5968bdc-hdzlm      1/1     Running   0          100m
kubernetes-dashboard   kubernetes-dashboard-7448ffc97b-d2q5v           1/1     Running   0          100m

安装kubernetes-dashboard(仅 master 需要安装)

• 官方部署dashboard的服务没使用nodeport，将yaml文件下载到本地，在service里添加nodeport

wget  https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml

如: dashboard 界面报错

namespaces is forbidden: User "system:serviceaccount:kubernetes-dashboard:kubernetes-dashboard" cannot list resource "namespaces" in API group "" at the cluster scope

原因: 发现是dashboard的版本和kubernetes的版本不一致

解决方案: 从 https://github.com/kubernetes/dashboard/releases 找到对应版本的 dashboard 的 yaml 重新部署, 即可解决

如果访问失败: 在 hosts 文件中添加 199.232.28.133 raw.githubusercontent.com

备用下载地址

• 编辑 recommended.yaml 文件

vim recommended.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000
  selector:
    k8s-app: kubernetes-dashboard

• 创建 dashboard

kubectl create -f recommended.yaml

通过 token 方式登录 k8s dashboard

获取 token

### 创建 service account
kubectl create sa dashboard-admin -n kube-system

### 创建角色绑定关系
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

### 查看 dashboard-admin 的 secret 名字
ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')

### 打印 secret 的token
kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}'

• 进入 dashboard 页面

浏览器输入master主机地址 + 30000,使用https协议
如: https://192.168.243.134:30000

node 加入到集群

在 master 主机上生成 token

默认token的有效期为24小时，当过期之后，该token就不可用了，在master节点上执行 kubeadm token create

• 创建token (仅 master 需要执行)

这个 token 可以自动设定,需要按照规则生成token

## token 规则 \\A([a-z0-9]{6})\\.([a-z0-9]{16})\\z
kubeadm token create token1.tokentokentoken1

• 查看 token(仅 master 需要执行)

kubeadm token list

### 执行结果
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION  
token1.tokentokentoken1   23h         2021-01-30T17:33:23+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

• 获取ca证书sha256编码hash值(仅 master 需要执行)

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

### 执行结果
0654fa65a6a2b7fe09cb605f24809e7fe61cdc910d7b2b74165c6c8843c197c7

• 节点加入集群(仅 node 需要执行)

### 清理环境
kubeadm reset 

### 链接集群
kubeadm join 192.168.243.134:6443 --token token1.tokentokentoken1 \
--discovery-token-ca-cert-hash sha256:0654fa65a6a2b7fe09cb605f24809e7fe61cdc910d7b2b74165c6c8843c197c7

重启集群命令

systemctl daemon-reload
systemctl restart kubelet