1. 服务器要求:
建议最小硬件配置:2核CPU、2G内存、20G硬盘。(master内存最好3G,node 2G)
服务器最好可以访问外网,会有从网上拉取镜像需求,如果服务器不能上网,需要提前下载对应镜像并导入节点
[root@k8s-node1 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
服务器规划
架构图
2. 操作系统初始化配置
安装虚拟机:
下载链接:https://download.csdn.net/download/qq_45614471/85722469
下载centos:地址1: https://www.linux.org/pages/download/;地址2: http://mirrors.aliyun.com/centos/7/isos/x86_64/
安装centos:
点击文件-新建虚拟机-自定义-下一步-下一步(选择稍后安装操作系统)-选择客户机操作系统直接下一步-修改虚拟机名字点击下一步-处理器数量(master节点选择2,node可以选1,根据自己的电脑配置来,配置低就都选1)-内存配置(master3个g,node2个g)- 网络选桥接吧 - IO控制器类型直接选推荐的 - 磁盘类型选择默认的-磁盘文件存储(创建文件夹保存系统创建过程中生成的文件)
上面的操作都是傻瓜试的,不怕选错,就怕不知道什么错。
选择镜像
确定后开启此虚拟机
开启后主要配置的几个地方:网络、时间、登录的账户
坑点处理:
centos安装好以后获取不到ip:ens33这个网卡里面没有inet的ip信息。
方法一、
把网卡配置中的ONBOOT修改为YES
[root@k8s-master1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="718c1e27-b4a5-44ac-ab4c-314714b6a5e6"
DEVICE="ens33"
ONBOOT="yes"
ZONE=public
方法二:
借鉴的博客(感谢朋友):https://blog.csdn.net/qq_27525611/article/details/110849786
常用的Linux的网络管理服务有两个:
network服务
NetworkManager服务
network服务是基础服务,通常在服务器中使用,当然在图形界面也有,通常使用命令行配置,而NetworkManager一般在图形界面安装,是为了方便ip的配置,要注意的是两个之中只能有一个生效,因此需要选择一个禁用。在笔者这种情况下应该是两者冲突了,导致不能正常显示本地ip,可以使用
systemctl status network
systemctl status NetworkManager
查看两服务状态,如果两个都是active就是服务冲突,要么应该就是配置出错了的原因
使用network,先把NetworkManager禁用:
systemctl stop NetworkManager
systemctl status NetworkManager
禁用之后重启network:
systemctl restart network
这样就能显示出本地ip了。
若还是不能显示请检查网卡配置,在/etc/sysconfig/network-scripts下,我的是/etc/sysconfig/network-scripts/ifcfg-ens33
3、安装docker(三台主机同时操作):
#查看系统内核版本
uname -r
#查看centos版本
cat /etc/redhat-release
#查看已安装的docker列表
yum list installed | grep docker
#清楚已安装的docker
sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
安装docker
#依赖
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
#镜像源
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#依赖
sudo yum install docker-ce docker-ce-cli containerd.io
#启动
systemctl enable docker && systemctl start docker
#配置镜像下载加速器
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://v27k018o.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
#添加阿里云YUM软件源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#安装kubeadm,kubelet,kubectl
#指定需安装版本号进行部署
yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0
systemctl enable kubelet
docker也可以这样直接安装
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce
systemctl enable docker && systemctl start docker
4、开始重点骚操作和k8s踩坑记录
在三台主机上输入命令
# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
# 关闭selinux
sed -i '/^SELINUX/s/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时
# 关闭swap
swapoff -a # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
# 根据规划设置主机名
hostnamectl set-hostname <hostname>
# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.137.81 k8s-master1
192.168.137.82 k8s-node1
192.168.137.83 k8s-node2
EOF
# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system # 生效
# 时间同步
yum install ntpdate -y && ntpdate time.windows.com
#使配置的内核参数生效
sysctl -p
5、部署Kubernetes Master
在192.168.100.88(Master)执行,–apiserver-advertise-address改为自己的master的ip就行了
kubeadm init \
--apiserver-advertise-address=192.168.100.88 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.20.0 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=all
–apiserver-advertise-address 集群通告地址(mater的ip)
–image-repository 由于默认拉取镜像地址k8s.gcr.io国内无法访问,这里指定阿里云镜像仓库地址
–kubernetes-version K8s版本,与上面安装的一致
–service-cidr 集群内部虚拟网络,Pod统一访问入口
–pod-network-cidr Pod 网络,与下面部署的CNI网络组件yaml中保持一致
或者使用配置文件引导:
vi kubeadm.conf
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
imageRepository: registry.aliyuncs.com/google_containers
networking:
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
kubeadm init --config kubeadm.conf --ignore-preflight-errors=all
初始化完成后,最后会输出一个join命令,先记住,下面用。
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.137.81:6443 --token tix7ff.pnjcwvl6awyaeh8i \
--discovery-token-ca-cert-hash sha256:617842ea5040ce5e7f971d387b58693cbfa79261763b68c589fe8d124f1a5154
坑点来了1:
这里初始化的时候可能会初始化失败。
根据以上关于 kubelet的报错,查看/var/log/messages日志,发现有以下问题
tail -f /var/log/messages
因此kubelet启动失败的原因是:kubelet的cgroup driver是cgroupfs,docker的 cgroup driver是systemd,两者不一致导致
解决方法
参考此博客解决:https://blog.csdn.net/qq_33326449/article/details/119699126
分别修改docker与控制平台的kubelet为systemd 【官方推荐】
鉴于用的k8s版本有点新,本文只记录当前1.18.x的修改方式,其他版本请参详官方:
重置未初始化成功的kubeadm配置
echo y|kubeadm reset
或者
kubeadm reset
我这里直接修改了docker的文件
修改docker,只需在/etc/docker/daemon.json中,添加"exec-opts": [“native.cgroupdriver=systemd”]即可,
"exec-opts": [
"native.cgroupdriver=systemd"
],
修改kubelet:
cat > /var/lib/kubelet/config.yaml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF
重启docker 与 kubelet:
systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
检查 docker info|grep “Cgroup Driver” 是否输出 Cgroup Driver: systemd
[root@k8s-master1 ~]# docker info|grep "Cgroup Driver"
Cgroup Driver: systemd
再次初始化
kubeadm init \
--apiserver-advertise-address=192.168.100.88 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.20.0 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=all
这三个比较重要,记录一下:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown
(
i
d
−
u
)
:
(id -u):
(id−u):(id -g) $HOME/.kube/config
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.100.88:6443 --token g3v4c6.lknh1oa9r3u4pd80 \
--discovery-token-ca-cert-hash sha256:0c31839c8dc76eed694c0f2590514ac1b48fef1e0a55ba8112c7b42c0e5162d7
拷贝kubectl使用的连接k8s认证文件到默认路径:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看工作节点:
[root@k8s-master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 5h55m v1.20.0
k8s-node1 Ready <none> 5h49m v1.20.0
k8s-node2 Ready <none> 5h49m v1.20.0
注:由于网络插件还没有部署,还没有准备就绪 NotReady。我这里已经安装好了,才显示的Ready
6、加入Kubernetes Node节点
在192.168.100.3/132 (Node节点) 执行 向集群添加新节点,执行在kubeadm init输出的kubeadm join命令:
kubeadm join 192.168.100.88:6443 --token g3v4c6.lknh1oa9r3u4pd80 \
--discovery-token-ca-cert-hash sha256:0c31839c8dc76eed694c0f2590514ac1b48fef1e0a55ba8112c7b42c0e5162d7
默认token有效期为24小时,当过期之后,该token就不可用了。这时就需要重新创建token,可以直接使用命令快捷生成:
kubeadm token create --print-join-command
参考资料:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/
7、部署容器网络(CNI)
安装网络插件
Calico是一个纯三层的数据中心网络方案,是目前Kubernetes主流的网络方案。 下载YAML:
wget https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f calico.yaml
Calico要要保持和k8s的版本要一致,不然安装也会报错。
error: unable to recognize "calico.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1"
去官网看看:https://projectcalico.docs.tigera.io/archive/v3.20/getting-started/kubernetes/requirements
我的下载命令:
wget --no-check-certificate https://docs.projectcalico.org/v3.20/manifests/calico.yaml
下载完后还需要修改里面定义Pod网络(CALICO_IPV4POOL_CIDR),与前面kubeadm init的 --pod-network-cidr指定的一样。 修改完后文件后,部署:
kubectl apply -f calico.yaml
kubectl get pods -n kube-system
等Calico Pod都Running,节点也会准备就绪
[root@k8s-master1 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6d9cdcd744-6s5tc 1/1 Running 4 3h29m
calico-node-dklj2 1/1 Running 0 3h29m
calico-node-mj9pf 1/1 Running 0 3h29m
calico-node-xt77z 1/1 Running 0 3h29m
coredns-7f89b7bc75-kdqsn 1/1 Running 0 6h13m
coredns-7f89b7bc75-ltwp6 1/1 Running 0 6h13m
etcd-k8s-master1 1/1 Running 0 6h13m
kube-apiserver-k8s-master1 1/1 Running 0 6h13m
kube-controller-manager-k8s-master1 1/1 Running 2 6h13m
kube-proxy-9p9kp 1/1 Running 0 6h8m
kube-proxy-dcr8s 1/1 Running 0 6h13m
kube-proxy-vfl5t 1/1 Running 0 6h8m
kube-scheduler-k8s-master1 1/1 Running 2 6h13m
参考资料:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network
8、测试kubernetes集群
在Kubernetes集群中创建一个pod,验证是否正常运行:
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
[root@k8s-master1 ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-6799fc88d8-tscdq 1/1 Running 0 4h9m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h15m
service/nginx NodePort 10.97.239.21 <none> 80:30695/TCP 4h9m
访问地址:http://NodeIP:Port
9、部署Dashboard
Dashboard是官方提供的一个UI,可用于基本管理K8s资源。
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml
这个好像下载不下来AI。
后来我直接登录了GitHub,从GitHub里面下载下来的啦。
链接:https://github.com/search?q=dashboard
tag里面内容挺丰富,进去看看:
下载下来-解压
没错就是他-盘他
默认Dashboard只能集群内部访问,修改Service为NodePort类型,暴露到外部:
# vi recommended.yaml
...
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
type: NodePort
...
执行
# kubectl apply -f recommended.yaml
# kubectl get pods -n kubernetes-dashboard
这里遇到一个问题:这里文件里面有两个service,我改错了,改成了我按了shit+g,跳到最后一行,修改了第二个service,改错了,所以我的dashboard一直没有安装成功。改错了怎么办呢:
node 正常
pod 正常
但是dashboard 无法访问
[root@k8s-master1 ~]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-6799fc88d8-tscdq 1/1 Running 0 4h22m
kube-system calico-kube-controllers-6d9cdcd744-6s5tc 1/1 Running 4 3h43m
kube-system calico-node-dklj2 1/1 Running 0 3h43m
kube-system calico-node-mj9pf 1/1 Running 0 3h43m
kube-system calico-node-xt77z 1/1 Running 0 3h43m
kube-system coredns-7f89b7bc75-kdqsn 1/1 Running 0 6h27m
kube-system coredns-7f89b7bc75-ltwp6 1/1 Running 0 6h27m
kube-system etcd-k8s-master1 1/1 Running 0 6h27m
kube-system kube-apiserver-k8s-master1 1/1 Running 0 6h27m
kube-system kube-controller-manager-k8s-master1 1/1 Running 2 6h27m
kube-system kube-proxy-9p9kp 1/1 Running 0 6h22m
kube-system kube-proxy-dcr8s 1/1 Running 0 6h27m
kube-system kube-proxy-vfl5t 1/1 Running 0 6h22m
kube-system kube-scheduler-k8s-master1 1/1 Running 2 6h27m
kubernetes-dashboard dashboard-metrics-scraper-7b59f7d4df-kl2g9 1/1 Running 0 107m
kubernetes-dashboard kubernetes-dashboard-5dbf55bd9d-f8x6x 1/1 Running 0 107m
查一下你的dashboard到那个node上了
kubectl --namespace=kubernetes-dashboard get service kubernetes-dashboard
查看一下dashboard暴露的断点
后来就把dashboard重新卸载了
kubectl delete -f recommended.yaml
#将新修改好的yaml文件上传后重装一下。
kubectl create -f recommended.yaml
访问地址:https://NodeIP:30001 创建service account并绑定默认cluster-admin管理员集群角色:
# 创建用户
$ kubectl create serviceaccount dashboard-admin -n kube-system
# 用户授权
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
# 获取用户Token
$ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
使用输出的token登录Dashboard
[root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name: dashboard-admin-token-cgmld
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-admin
kubernetes.io/service-account.uid: 3208e176-5115-4425-bb67-d45863bc05f7
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1066 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlBnN2lzMk8zYndPX3ZONnc0cnFnRjhsVnczOTVlNGxXSDl4c1Z0OGtmNDAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tY2dtbGQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMzIwOGUxNzYtNTExNS00NDI1LWJiNjctZDQ1ODYzYmMwNWY3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.DY9SLeakUzVUqUHLJjLuaBtv0EOj6l-zsCfmzKTtsiTkaX39bGpLInToKuSpbXHYAmkpCvoZH22ghmOds3LFgvQBCIt6M83rrL83aPzhDjAKtPtPkz9vJGR7K5LnfrB9AX5dVhiU_AkaVIBIFcqTxVlFpl6W1EzTc0uJDM7K8Gr2XnPvRfUMe8WaEWR7tVxMEEhPhP2waEYmcc5uFz5unI_g6lTMYRJnhZCjfqh7lS9NA_8WgmoQnQjxW4cYAsqrdCzbroTEMCslH_pCj-PZNxf7mKVXZwklYL78t8klU_AytuhdaV88iRR3HEuBMYLbfJjy6RLkyt_ORweaXb8npg
10、修改登录方式为账户密码
k8s登录默认使用的是token登录,每次登录的时候需要获取token,我们可以改为账户密码登录的方式
修改流程:修改apiservice的配置文件;修改dashboard的配置文件使其支持配置;添加角色权限关联
1、修改kube-apiserver.yaml
重要:先备份
#备份
cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml-bake-20220623
#修改
vim /etc/kubernetes/manifests/kube-apiserver.yaml
#添加这一段
- --token-auth-file=/etc/kubernetes/pki/basic_auth_file
#手动重启
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kube-apiserver.yaml修改后默认1秒后自动重启。也可以手动重启
在配置好的basic_auth_file文件中添加账号密码
echo 'admin,admin,1' | sudo tee /etc/kubernetes/pki/basic_auth_file
[root@k8s-master1 ~]# cat /etc/kubernetes/pki/basic_auth_file
admin,admin,1
为admin/fengyuqing用户绑定权限:
# admin绑定权限
kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin
# 查看绑定结果
kubectl get clusterrolebinding login-on-dashboard-with-cluster-admin
#查看用户角色绑定情况添加
[root@k8s-master1 ~]# kubectl get clusterrolebinding login-on-dashboard-with-cluster-admin
NAME ROLE AGE
login-on-dashboard-with-cluster-admin ClusterRole/cluster-admin 34m
# admin绑定权限
kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin
# 查看绑定结果
kubectl get clusterrolebinding login-on-dashboard-with-cluster-admin
2、修改dashboard文件recommended.yaml
修改web控制台dashboard文件,使其支持账户密码登录。
- --token-ttl=51600
- --authentication-mode=basic
重启dashboard
kubectl apply -f recommended.yaml
有报错或者进去后查不到节点和pod信息输入下面的命令
kubectl create clusterrolebinding test:anonymous --clusterrole=cluster-admin --user=system:anonymous