问题场景:
今日在阿里云部署K8S成功以后,部署了Kubesphere,日志一直卡在minio部署中:
[root@master ~]# kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
... other log
TASK [common : KubeSphere | Checking minio] ************************************
changed: [localhost]
TASK [common : KubeSphere | Deploying minio] ***********************************
问题分析:
查看kubesphere中minio相关pod日志看看有什么错误日志。
查看minio安装日志日志:
[root@master ~]# kubectl logs minio-68df76c95f-qb7sp -n kubesphere-system
Endpoint: http://10.244.166.133:9000 http://127.0.0.1:9000
Browser Access:
http://10.244.166.133:9000 http://127.0.0.1:9000
Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
看出没有什么问题,没有错误日志
查看minio-make-bucket-job安装日志:
[root@master ~]# kubectl logs minio-make-bucket-job-llxc2 -n kubesphere-system
Connecting to Minio server: http://minio:9000
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-xv41utfnfvgs/?location=: dial tcp: i/o timeout.
"Failed attempts: 1"
可以看到,有一个访问9000端口的失败了,看到kubesphere官网有人说网络问题,首先排查网络。
找到minio的网络IP:
[root@master ~]# kubectl get pods -A -o wide -w
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...其他pod
kubesphere-system minio-68df76c95f-qb7sp 1/1 Running 0 93s 10.244.166.133 node1 <none> <none>
kubesphere-system minio-make-bucket-job-llxc2 1/1 Running 1 (2s ago) 31m 10.244.104.5 node2 <none> <none>
...其他pod
可以看到,minio 的 IP 为:10.244.166.133
尝试排查网络问题:
尝试ping,结果不通:
[root@master ~]# ping 10.244.166.133
PING 10.244.166.133 (10.244.166.133) 56(84) bytes of data.
^C
--- 10.244.166.133 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
查看 route 信息:
[root@master ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 eth0
10.244.104.0 node2 255.255.255.192 UG 0 0 0 eth0
10.244.166.128 node1 255.255.255.192 UG 0 0 0 eth0
10.244.219.64 0.0.0.0 255.255.255.192 U 0 0 0 *
10.244.219.71 0.0.0.0 255.255.255.255 UH 0 0 0 calib18ee40d62c
10.244.219.72 0.0.0.0 255.255.255.255 UH 0 0 0 cali1b2a2cf9dcf
10.244.219.73 0.0.0.0 255.255.255.255 UH 0 0 0 calia93ec013639
10.244.219.74 0.0.0.0 255.255.255.255 UH 0 0 0 cali79d02065363
10.244.219.75 0.0.0.0 255.255.255.255 UH 0 0 0 calif78f7290b65
10.244.219.76 0.0.0.0 255.255.255.255 UH 0 0 0 cali0b3aa4ca066
link-local 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
172.21.144.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
ping都不通:
[root@master ~]# ping 10.244.104.0
PING 10.244.104.0 (10.244.104.0) 56(84) bytes of data.
卡住...不通
[root@master ~]# ping 10.244.166.128
PING 10.244.166.128 (10.244.166.128) 56(84) bytes of data.
卡住...不通
最终解决:
解决分两步:阿里云配置(非阿里云可能不需要此步骤)、iptable规则修改。
阿里云配置
现在这是阿里云环境,我就在本地部署了一下,ping 了一下各个node之间能通,这证明是个别环境问题,找了阿里云客服问了,是交换机路由表问题。需要添加路由表,将各个node ip添加到专有网络 -> 路由表 -> 自定义路由条目即可,重点是:
转发IP段设置好IP段和子网数。
转发目的实例选择好IP所在实例
iptable规则修改
因为阿里云centos镜像会有防火墙,先关闭防火墙,再添加规则放行TCP 0-65535端口、UDP1-65535端口以及IPENCAP 和IPIP。
关闭防火墙、开机不启动防火墙:
[root@master ~]# sudo systemctl stop firewalld && sudo systemctl disable firewalld
添加iptable放行规则 并 重启电脑:
[root@node1 ~]# sudo iptables -A INPUT -p tcp --dport 0:65535 -j ACCEPT && \
sudo iptables -A INPUT -p udp --dport 1:65535 -j ACCEPT && \
sudo iptables -A INPUT -p 41 -j ACCEPT && \
sudo iptables -A INPUT -p 4 -j ACCEPT && \
sudo iptables -L -n -v && \
sudo iptables -L -n -v |grep 4 && \
sudo iptables-save && reboot
这个Iptable没有详细纠结是端口问题还是IPENCAP或者IPIP的问题,有时间纠结下找出...