组件概览
关于k8s整体架构,可参考:之前文章
Kubernetes主要由以下几个核心组件组成(必须安装):
- etcd保存了整个集群的状态;
- apiserver提供了资源操作的唯一入口,并提供认证、授权、访问控制、API注册和发现等机制;
- controller manager负责维护集群的状态,比如故障检测、自动扩展、滚动更新等;
- scheduler负责资源的调度,按照预定的调度策略将Pod调度到相应的机器上;
- kubelet负责维护容器的生命周期,同时也负责Volume(CVI)和网络(CNI)的管理,安装每台工作节点;
- kube-proxy负责为Service提供cluster内部的服务发现和负载均衡,安装在每台工作节点;
除了核心组件,还有一些推荐的Add-ons(选装): - kube-dns负责为整个集群提供DNS服务,应该使用kubectl添加一个服务到容器,是一个发布的应用服务于
其中kubelet是二进制安装包,其他组件均为docker镜像,kubeadm负责拉取镜像并初始化环境。
kubectl是客户端管理工具,同样是个二进制。
所以kubelet,kubeadm,kubectl需要通过yum|apt-get安装,其他组件通过kubeadm安装。
安装k8s
最简单,成功率最高的安装方式其实是使用rke安装,这里使用kubeadmin是想知道每个组件的单独原理和作用,具体安装教程参考:https://docs.rancher.cn/docs/rke/installation/_index/。
已经安装如果需要清除,可按照此步骤处理(重置k8s,删除所有运行容易和镜像)
kubeadm reset
docker ps -a | awk '{if(NR>1){print $1;system("docker stop "$1);system("docker rm "$1)}}';
docker images | awk '{system("docker rmi "$3)}'
rm -rf $HOME/.kube
同时清除下面安装网络章节的网络相关文件
准备机器
这里使用debian环境,主备一主一从两台机器,最好固定ip,ip最好在同一网段
k8s-master 10.10.0.115
k8s-worker 10.10.0.116
安装kubelet,kubeadm,kubectl,两台机器均相同
确保两台机器提前安装包docker
apt-get install docker-ce
设置阿里云源
sudo vim /etc/apt/sources.list.d/kubernetes.list
# 将下面的阿里源加入文件中
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
# 也可以选择中科大的源
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
里先运行一下 apt update, 会报错,原因是缺少相应的key,可以通过下面的命令添加(E084DAB9 为上面报错的key后8位)
gpg --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
gpg --export --armor E084DAB9 | sudo apt-key add -
下载安装
apt-get update && apt-get install -y kubelet kubeadm kubectl
关闭swap
如果不关闭kubernetes运行会出现错误, 即使安装成功了,node重启后也会出现kubernetes server运行错误。
#暂时关闭,
sudo swapoff -a
# 永久关闭
vim /etc/fstab
注释掉swap那一行就行
虚拟机最好把内存分配调整到2G以上,否则关掉swap会导致图形界面难以进入。
我这里是注释最后一行
root@liaok8s:/home/mainte# more /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=52c86df1-9442-46c0-a29c-b8f7d0a421ab / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
#UUID=e4e792e9-d30b-4e4a-ab3c-26bd488adaae none swap sw 0 0
#/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
获取镜像
由于官方镜像地址被墙,所以我们需要首先获取所需镜像以及它们的版本。然后从国内阿里的镜像站获取。
kubeadm config images list
获取镜像列表后可以通过下面的脚本从阿里云获取:
for i in `kubeadm config images list`; do
#coredns镜像在gcr.io上是k8s.gcr.io/coredns/coredns:v1.8.6 在阿里云上是 registry.aliyuncs.com/google_containers/coredns:v1.8.6,需要特殊处理
if echo $i | grep coredns/coredns;then
imageName=${i#k8s.gcr.io/coredns/}
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/coredns/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
else
imageName=${i#k8s.gcr.io/}
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
fi
done;
主节点
初始化
kubeadm init --service-cidr=10.96.0.0/12 --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
初始化可常用的参数:
参数说明
--apiserver-advertise-address=10.10.0.115 这个参数就是master主机的IP地址,例如我的Master主机的IP是:10.10.0.115
--image-repository=registry.aliyuncs.com/google_containers 这个是镜像地址,由于国外地址无法访问,故使用的阿里云仓库地址:registry.aliyuncs.com/google_containers,如果不指定就需要做下面获取镜像章节的动作。
--kubernetes-version=v1.17.4 这个参数是下载的k8s软件版本号
--service-cidr=10.96.0.0/12 这个参数后的IP地址直接就套用10.96.0.0/12 ,以后安装时也套用即可,不要更改
--pod-network-cidr=10.244.0.0/16 k8s内部的pod节点之间网络可以使用的IP段,不能和service-cidr写一样,如果不知道怎么配,就先用这个10.244.0.0/16
一般都会报错
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
查看官网介绍为 docker 和 kubelet 服务中的 cgroup 驱动不一致,有两种方法
方式一:驱动向 docker 看齐
方式二:驱动为向 kubelet 看齐
如果docker 不方便重启则统一向 kubelet看齐,并重启对应的服务即可
解决方式
docker 配置文件
这里采取的是方式二,docker 默认驱动为 cgroupfs ,只需要添加
“exec-opts”: [
“native.cgroupdriver=systemd”
],
修改后配置文件
root@controlplane:~# cat /etc/docker/daemon.json
{
"exec-opts": [
"native.cgroupdriver=systemd"
],
"bip":"172.12.0.1/24",
"registry-mirrors": [
"http://docker-registry-mirror.kodekloud.com"
]
}
重启docker
systemctl restart docker
kublete 配置文件
grep 截取一下,可以看得出来kubelet默认 cgoup 驱动为systemd
root@controlplane:~# cat /var/lib/kubelet/config.yaml |grep group
cgroupDriver: systemd
修改后重置并重新初始化可正常初始化
kubeadm reset && kubeadm init
成功后提示
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.10.0.115:6443 --token pedwrg.x3ocfkg6ui1t5yht \
--discovery-token-ca-cert-hash sha256:a696588d58710779c758a0cdc4f0da3154af5d62f9b54420b5bb78d63f11e7a2
注意最后一段的kubeadm join是worker节点加入使用的,注意保存,如果忘记了可通过以下命令打印:
kubeadm token create --print-join-command
普通用户安装需要执行如下脚本:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
root用户直接运行
export KUBECONFIG=/etc/kubernetes/admin.conf
此时查看所有二进制安装的进程
root@liaok8s:/home/mainte# ps -ef | grep kube
root 7691 7622 2 16:30 ? 00:00:03 etcd --advertise-client-urls=https://10.10.0.115:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://10.10.0.115:2380 --initial-cluster=liaok8s=https://10.10.0.115:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://10.10.0.115:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://10.10.0.115:2380 --name=liaok8s --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root 7736 7648 8 16:30 ? 00:00:10 kube-apiserver --advertise-address=10.10.0.115 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root 7744 7661 2 16:30 ? 00:00:03 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root 7752 7695 1 16:30 ? 00:00:01 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root 7965 1 2 16:30 ? 00:00:03 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6
root 8294 8274 0 16:31 ? 00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=liaok8s
root 9929 1346 0 16:33 pts/0 00:00:00 grep kube
查看系统pod
root@liaok8s:/home/mainte# kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-64897985d-f9l5d 0/1 ContainerCreating 0 3m18s
coredns-64897985d-gtbp6 0/1 ContainerCreating 0 3m18s
etcd-liaok8s 1/1 Running 2 3m23s
kube-apiserver-liaok8s 1/1 Running 2 3m25s
kube-controller-manager-liaok8s 1/1 Running 2 3m23s
kube-proxy-2phdl 1/1 Running 0 3m18s
kube-scheduler-liaok8s 1/1 Running 4 3m22s
发现dns关于网络的服务都是0个实例
同时检查node节点(目前就一个master节点,版本1.12.3)
root@liaok8s:/home/mainte# kubectl get node
NAME STATUS ROLES AGE VERSION
liaok8s Ready control-plane,master 4m9s v1.23.3
查看版本
root@liaok8s:/home/mainte# kubectl version --short=true
Client Version: v1.23.3
Server Version: v1.23.3
上面安装成功后如果通过查询kube-system下Pod的运行情况,会发现和网络相关的Pod都处于Pending的状态,这是因为缺少相关的网络插件。
安装网络
上面安装成功后如果通过查询kube-system下Pod的运行情况,会放下和网络相关的Pod都处于Pending的状态,这是因为缺少相关的网络插件,而网络插件有很多个(以下任选一个),可以选择自己需要的。
比较流行的几个为:
- flannel
- weave
- calico
这里以calico为例
Calico是一个纯三层的协议,为OpenStack虚机和Docker容器提供多主机间通信。Calico不使用重叠网络比如flannel和libnetwork重叠网络驱动,
它是一个纯三层的方法,使用虚拟路由代替虚拟交换,每一台虚拟路由通过BGP协议传播可达信息(路由)到剩余数据中心。calico原理参考
不要直接使用下面方式下载安装,因为node间通信,默认会选择第一个物理网卡的ip,会导致选错ip,无法通信
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
应该首先下载calico.yaml
wget https://docs.projectcalico.org/manifests/calico.yaml
修改yaml文件容器名称为:calico-node,搜索大概这个内容
containers:
- name: calico-node
image: docker.io/calico/node:v3.22.0
新增一个变量 env部分
- name: IP_AUTODETECTION_METHOD
value: "interface=ens.*"
意思是找物理主机的ip是使用ens开头的网卡,具体的本地物理网卡名称可以通过 ip addr查看
kubectl apply -f ./calico.yaml
检测pod是否安装成功
root@liaok8s:/home/mainte# kubectl get pods --namespace kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-566dc76669-dwlrk 1/1 Running 0 4m4s 192.168.134.129 liaok8s <none> <none>
calico-node-gtpcw 1/1 Running 0 4m4s 10.10.0.115 liaok8s <none> <none>
coredns-64897985d-f9l5d 1/1 Running 0 64m 172.16.134.131 liaok8s <none> <none>
coredns-64897985d-gtbp6 1/1 Running 0 64m 172.16.134.129 liaok8s <none> <none>
etcd-liaok8s 1/1 Running 2 64m 10.10.0.115 liaok8s <none> <none>
kube-apiserver-liaok8s 1/1 Running 2 64m 10.10.0.115 liaok8s <none> <none>
kube-controller-manager-liaok8s 1/1 Running 2 64m 10.10.0.115 liaok8s <none> <none>
kube-proxy-2phdl 1/1 Running 0 64m 10.10.0.115 liaok8s <none> <none>
kube-scheduler-liaok8s 1/1 Running 4 64m 10.10.0.115 liaok8s <none> <none>
如果安装失败很有可能是之前安装reset过没有清除完,清除后重装
1、删除安装插件
kubectl delete -f ./calico.yaml
2、检查所有节点上的网络,看看是否存在Tunl0
若存在Tunl0,将其删除
modprobe -r ipip
3、移除与Calico网络插件有关的网络配置文件
ls /etc/cni/net.d/
rm -rf calico相关文件
rm -rf /etc/cni/net.d/*calico*
删除所有calico的pod
for i in `kubectl get pods --namespace kube-system`; do
if echo $i | grep calico;then
echo $i | awk 'system("kubectl delete --force pod "$1" --namespace kube-system")'
fi
done;
4、重新安装
kubectl apply -f ./calico.yaml
worker节点
同主节点kubelet和docker的驱动类型必须一致
修改/etc/docker/daemon.json 重启docker
"exec-opts": [
"native.cgroupdriver=systemd"
],
注意worker节点也要获取这些镜像,虽然只有部分使用,比如pause,否则kubelet会出现拉取错误
之前是没有在worker节点拉取镜像,在主节点查看calico网络的event日志
kubectl describe pods calico-node-ncwdc -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned kube-system/calico-node-ncwdc to pve-tmpl
Warning FailedCreatePodContainer 15m kubelet unable to ensure pod container exists: failed to create container for [kubepods burstable podb1e24b58-fe92-42d6-8857-61d4d36638a2] : mkdir /sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e24b58_fe92_42d6_8857_61d4d36638a2.slice: no such file or directory
Warning FailedCreatePodSandBox 15m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning FailedCreatePodSandBox 14m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 74.125.204.82:443: i/o timeout
Warning FailedCreatePodSandBox 10m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 142.251.8.82:443: i/o timeout
Warning FailedCreatePodSandBox 8m35s (x13 over 15m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning FailedCreatePodSandBox 8m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 108.177.125.82:443: i/o timeout
使用之前的kubeadm join命令加入,如果忘记了可以在主节点获取
kubeadm token create --print-join-command
kubeadm join 10.10.0.115:6443 --token dprbpd.gdjxay6moqf10d05 --discovery-token-ca-cert-hash sha256:a696588d58710779c758a0cdc4f0da3154af5d62f9b54420b5bb78d63f11e7a2
加入成功后,可通过docker ps 查看
root@pve-tmpl:/home/mainte# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
96170aa46f83 k8s.gcr.io/pause:3.6 "/pause" 13 hours ago Up 13 hours k8s_POD_nginx_default_7c3d47ef-f478-40b4-b154-e9334cff14e3_1
6c627df7a7ca f109b1742d34 "start_runit" 13 hours ago Up 13 hours k8s_calico-node_calico-node-j8x2d_kube-system_3875d252-86cf-47e4-8c86-e74f32356d51_1
c30f1e963eae k8s.gcr.io/pause:3.6 "/pause" 13 hours ago Up 13 hours k8s_POD_calico-node-j8x2d_kube-system_3875d252-86cf-47e4-8c86-e74f32356d51_1
0426d0f18715 9b7cc9982109 "/usr/local/bin/kube…" 13 hours ago Up 13 hours k8s_kube-proxy_kube-proxy-kql69_kube-system_838b79c1-14c1-46e1-a806-868ca3fa87d3_2
773974f94bba k8s.gcr.io/pause:3.6 "/pause" 13 hours ago Up 13 hours k8s_POD_kube-proxy-kql69_kube-system_838b79c1-14c1-46e1-a806-868ca3fa87d3_1
从节点回安装pause,calico,kubeproxy等组件,二进制组件
root@pve-tmpl:/home/mainte# ps -ef | grep kube
root 392 1 1 Feb10 ? 00:13:19 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6
root 967 947 0 Feb10 ? 00:00:08 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=pve-tmpl
异常问题
calico容器异常
如果calico安装出现问题,一般都会导致kube-system下的calico相关的容器无法正常运行
root@liaok8s:/usr/local/bin# kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-566dc76669-dllq7 1/1 Running 0 13h
calico-node-j2ffk 1/1 Running 0 13h
calico-node-j8x2d 1/1 Running 1 (13h ago) 13h
coredns-64897985d-f9l5d 1/1 Running 1 (13h ago) 17h
coredns-64897985d-gtbp6 1/1 Running 1 (13h ago) 17h
etcd-liaok8s 1/1 Running 3 (13h ago) 17h
kube-apiserver-liaok8s 1/1 Running 4 (13h ago) 17h
kube-controller-manager-liaok8s 1/1 Running 3 (13h ago) 17h
kube-proxy-2phdl 1/1 Running 1 (13h ago) 17h
kube-proxy-kql69 1/1 Running 2 (13h ago) 15h
kube-scheduler-liaok8s 1/1 Running 5 (13h ago) 17h
如果任意一个容器出现异常可以通过查看event发现问题
kubectl describe pod calico-node-j2ffk -n kube-system
我这里之前没有修改calico.yaml直接apply导致出现了一个问题
Warning Unhealthy 23s (x3 over 25s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
Warning Unhealthy 20s kubelet Readiness probe failed: 2022-02-10 10:15:15.807 [INFO][250] confd/health.go 180: Number of node(s) with BGP peering established = 0
这个问题就是calico通信无法识别主机间真实的物理ip引起的,可通过安装calicoctl诊断
安装请参考官网:https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
正确的结果为
在master节点,peer_address为多个从节点的ip,一个从节点一个,INFO为Established
root@liaok8s:/usr/local/bin# calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 10.10.0.116 | node-to-node mesh | up | 12:29:59 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
在worker节点,peer_address为多个从节点的ip和主节点的ip,一个从节点一个,INFO为Established
root@pve-tmpl:/home/mainte# calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 10.10.0.115 | node-to-node mesh | up | 12:29:59 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
注意这些ip如果不是物理主机的ip, 变成了某个网卡的内部地址就会出现通信问题。
在主和工作节点上都会有个tunl0的网卡,工作节点的tunl0的ip地址就是pod的节点ip段
无法ping通clusterip和服务名
因为我之前重置过多次集群,没有指定固定的pod ip段,导致master和worker节点tunl0的ip不在同一个网段,导致无法正常通讯。
此时我们可以创建一个dnsutils镜像在kube-system下用来诊断网络
apiVersion: v1
kind: Pod
metadata:
name: dnsutils
spec:
containers:
- name: dnsutils
image: mydlqclub/dnsutils:1.3
imagePullPolicy: IfNotPresent
command: ["sleep","3600"]
通过 Kubectl 工具部署 NDS 工具镜像
通过 Kubectl 工具,将对上面 DNS 工具镜像部署到 Kubernetes 中:
-n:指定应用部署的 Namespace 空间。
$ kubectl create -f ndsutils.yaml -n kube-system
进入 DNS 工具 Pod 的命令行
上面 DNS 工具已经部署完成,我们可也通过 Kubectl 工具进入 Pod 命令行,然后,使用里面的一些工具进行问题分析,命令如下:
exec:让指定 Pod 容器执行某些命令。
-i:将控制台内容传入到容器。
-t:进入容器的 tty 使用 bash 命令行。
-n:指定上面部署 DNS Pod 所在的 Namespace。
$ kubectl exec -it dnsutils /bin/sh -n kube-system
通过 Ping 和 Nsloopup 命令测试
进入容器 sh 命令行界面后,先使用 ping 命令来分别探测观察是否能够 ping 通集群内部和集群外部的地址
首先确认下ndsutils的ip地址:192.168.57.136和第一个coredns相同。
进入容器后 :
/ # ping 192.168.57.136 coredns1能ping通,在同一个worker节点
PING 192.168.57.136 (192.168.57.136): 56 data bytes
64 bytes from 192.168.57.136: seq=0 ttl=64 time=0.058 ms
64 bytes from 192.168.57.136: seq=1 ttl=64 time=0.045 ms
^C
--- 192.168.57.136 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.045/0.051/0.058 ms
/ # ping 192.168.134.132 master对应的coredns无法ping通
PING 192.168.134.132 (192.168.134.132): 56 data bytes
^C
--- 192.168.134.132 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
查看容器的/etc/resolv.conf文件
/ # more /etc/resolv.conf
nameserver 10.96.0.10
search kube-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
发现nameserver指向的是service=kube-dns的集群ip
注意clusterip是不能ping的 可以通过ip:端口组合来访问。
此时ping下外网,不通的
/ # ping www.baidu.com
^C
kube-dns的service实际是负载均衡到两个coredns上,其中一个是通的我们设置dns服务是这个通的试试
#nameserver 10.96.0.10
nameserver 192.168.57.138
search kube-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
再次测试发现正常
/ # ping www.baidu.com
PING www.baidu.com (110.242.68.4): 56 data bytes
64 bytes from 110.242.68.4: seq=0 ttl=52 time=44.204 ms
64 bytes from 110.242.68.4: seq=1 ttl=52 time=44.330 ms
64 bytes from 110.242.68.4: seq=2 ttl=52 time=43.863 ms
64 bytes from 110.242.68.4: seq=3 ttl=52 time=44.107 ms
64 bytes from 110.242.68.4: seq=4 ttl=52 time=44.148 ms
64 bytes from 110.242.68.4: seq=5 ttl=52 time=44.329 ms
64 bytes from 110.242.68.4: seq=6 ttl=52 time=44.210 ms
64 bytes from 110.242.68.4: seq=7 ttl=52 time=44.204 ms
64 bytes from 110.242.68.4: seq=8 ttl=52 time=44.113 ms
64 bytes from 110.242.68.4: seq=9 ttl=52 time=44.161 ms
^C
--- www.baidu.com ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 43.863/44.166/44.330 ms
/ # ping nginx.default
PING nginx.default (10.99.35.228): 56 data bytes
^C
--- nginx.default ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
/ # ping httpd.default
PING httpd.default (10.98.37.38): 56 data bytes
^C
/ # wget nginx.default 注意属于两个不同的namespace所以需要在服务名称后加上.namesapce:端口访问,ip是不能直接ping通的 但是可以获取到dns的地址
Connecting to nginx.default (10.99.35.228:80)
index.html 100% |**********************************************************************************************************************************************************************************************| 615 0:00:00 ETA
/ # more index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
/ #
这里可以想到 ,想着是否我们需要将pod的ip全部都设置为相同的段能访问是否就可行了,重置k8s,init
带上–pod-network-cidr=10.244.0.0/16 发现依然不行,会不是是coredns不应该在master节点上有个pod的了。
重新按照coredns,参考: https://github.com/coredns/deployment/tree/master/kubernetes
下载这两个文件
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
执行命令(-i指定的参数是kube-dns clusterid的地址,默认是10.96.0.10),
注意先提前删除kube-system下的deploy=coredns,service=kube-dns
kubectl delete deploy coredns && kubectl delete svc kube-dns
chmod +x ./deploy.sh && ./deploy.sh -i 10.96.0.10 | kubectl apply -f -
如果命令执行缺少jq,提前用yum或者apt-get或者apk安装,重新安装完成,在测试一切正常了,但是这个命令安装只是产生了一个coredns的pod,依然有点不知所以然虽然正常运行,后续在新增一个worker节点在测试下他的service分发的逻辑。
发布测试
主节点运行
kubectl run nginx --image=nginx
查看nginx状态
root@liaok8s:/usr/local/bin# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 1 (13h ago) 14h 192.168.57.130 pve-tmpl <none> <none>
查看执行过程(如果ready状态一直是0)
kubectl describe pod nginx
通过pod ip访问http地址
root@liaok8s:/usr/local/bin# curl 192.168.57.130
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
创建service
root@liaok8s:/usr/local/bin# kubectl expose pod nginx --port=80
service/nginx exposed
查看service
root@liaok8s:/usr/local/bin# kubectl get service nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
nginx ClusterIP 10.99.35.228 <none> 80/TCP 46s run=nginx
通过集群ip负载访问容器
root@liaok8s:/usr/local/bin# curl 10.99.35.228
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
验证dns正确性
安装一个apahce httpd服务
kubectl run httpd --image httpd
安装成功后进入worker节点宿主机,进入nginx容器
重新启动
如果因为异常断电,导致k8s停止,可设置集群自动启动
配置环境变量
#centos下
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile && source ~/.bash_profile
#debian下
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bashrc && source ~/.bashrc
在主工作节点设置服务自启
systemctl enable kubelet && systemctl restart kubelet
如果kubelet无法启动,很大可能是swap未正常关闭 ,sudo swapoff -a 再试试
图形管理工具
Kubernetes 容器编排已越来越被大家关注,然而使用 Kubernetes 的门槛却依然很高,主要体现在这几个方面:
- 集群的安装复杂,出错概率大
- Kubernetes相较于容器化,引入了许多新的概念,学习难度高
- 需要手工编写 YAML 文件,难以在多环境下管理
- 缺少好的实战案例可以参考
Kuboard,是一款免费的 Kubernetes 图形化管理工具,Kuboard 力图帮助用户快速在 Kubernetes 上落地微服务。
安装
如果您参考 https://kuboard.cn 网站上提供的 Kubernetes 安装文档,可在 master 节点上执行以下命令。
kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
查看 Kuboard 运行状态:
# kubectl get pods -l k8s.eip.work/name=kuboard -n kube-system
NAME READY STATUS RESTARTS AGE
kuboard-756d46c4d4-qh6cm 1/1 Running 0 101m
确保kuboard 处于 Running 状态
获取Token
您可以获得管理员用户、只读用户的Token。
Kuboard 有计划开发权限设置的功能,在这之前,如果您需要更细粒度的权限控制,请参考 RBAC Example
此Token拥有 ClusterAdmin 的权限,可以执行所有操作
# kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d
Kuboard Service 使用了 NodePort 的方式暴露服务,NodePort 为 32567;您可以按如下方式访问 Kuboard。
http://任意一个Worker节点的IP地址:32567/
输入前一步骤中获得的 token,可进入 Kuboard 集群概览页
Kuboard v3.x
Kuboard v3.x 支持 Kubernetes 多集群管理。如果您从 Kuboard v1.0.x 或者 Kuboard v2.0.x 升级到 Kuboard,请注意:
- 您可以同时使用 Kuboard v3.x 和 Kuboard v2.0.x;
- Kuboard v3.x 支持 amd64 (x86) 架构和 arm68 (armv8) 架构的 CPU;
可参考kuboard官网:https://kuboard.cn/install/v3/install-in-k8s.html
在线安装:
kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3.yaml
访问 Kuboard
在浏览器中打开链接 http://your-node-ip-address:30080
输入初始用户名和密码,并登录
用户名: admin
密码: Kuboard123