gluster Issues: Readiness probe failed or Liveness probe failed

问题描述：

glusterfs的Pod启动失败如下（实际为gluster-blockd启动失败）

 

log

查看gluster需要的几个状态：
systemctl status gluster-blockd

 

Mar 21 08:37:42 k8s-worker03 systemd[1]: Dependency failed for Gluster block storage utility.
Mar 21 08:37:42 k8s-worker03 systemd[1]: Job gluster-blockd.service/start failed with result 'dependency'.

systemctl status tcmu-runner.service：

● tcmu-runner.service - LIO Userspace-passthrough daemon
Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-12-03 17:22:14 UTC; 8s ago
Process: 5151 ExecStart=/usr/bin/tcmu-runner (code=exited, status=1/FAILURE)
Main PID: 5151 (code=exited, status=1/FAILURE)

Dec 03 17:22:14 vva-er-k8s0 systemd[1]: Starting LIO Userspace-passthrough daemon...
Dec 03 17:22:14 vva-er-k8s0 tcmu-runner[5151]: The logdir option from the tcmu.conf will be ignored
Dec 03 17:22:14 vva-er-k8s0 tcmu-runner[5151]: Inotify is watching "/etc/tcmu/tcmu.conf", wd: 1, mask: IN_ALL_EVENTS
Dec 03 17:22:14 vva-er-k8s0 systemd[1]: tcmu-runner.service: main process exited, code=exited, status=1/FAILURE
Dec 03 17:22:14 vva-er-k8s0 systemd[1]: Failed to start LIO Userspace-passthrough daemon.
Dec 03 17:22:14 vva-er-k8s0 systemd[1]: Unit tcmu-runner.service entered failed state.
Dec 03 17:22:14 vva-er-k8s0 systemd[1]: tcmu-runner.service failed.

journalctl -xeu tcmu-runner主要报错

Failed to start LIO Userspace-passthrough daemon.

查看tcmu-runner的报错：
tcmu-runner -dd

[root@glusterfs2 /]# tcmu-runner -dd
The logdir option from the tcmu.conf will be ignored
2019-01-16 03:16:55.012 288 [INFO] dyn_config_start:409: Inotify is watching "/etc/tcmu/tcmu.conf", wd: 1, mask: IN_ALL_EVENTS
2019-01-16 03:16:55.012 288 [DEBUG] main:1018: handler path: /usr/lib64/tcmu-runner
2019-01-16 03:16:55.013 288 [INFO] load_our_module:498: no modules directory '/lib/modules/4.15.0-43-generic', checking module target_core_user entry in '/sys/modules/'
2019-01-16 03:16:55.013 288 [ERROR] load_our_module:503: stat() on '/sys/module/target_core_user' failed: No such file or directory
2019-01-16 03:16:55.013 288 [ERROR] main:1022: couldn't load module

gluster的git上解决方式(有兴趣可进去查看下git上的issues)：将内核文件的volume路径由/usr/lib/module改为/lib/module

---

若报错如下

[root@node5 /]# tcmu-runner -dd
The logdir option from the tcmu.conf will be ignored
Inotify is watching "/etc/tcmu/tcmu.conf", wd: 1, mask: IN_ALL_EVENTS
2019-10-22 01:51:20.445 508 [DEBUG] main:1018: handler path: /usr/lib64/tcmu-runner
2019-10-22 01:51:20.514 508 [ERROR] load_our_module:515: kmod_module_new_from_lookup() failed to find module target_core_user

报错显示没有加载target_core_user内核模块
首先尝试modprobe target_core_user没有报错，lsmod |grep target_core_user查看模块已加载
若
返回如下

[root@node5 /]# modprobe target_core_user
modprobe: FATAL: Module target_core_user not found.

查看ls /lib/modules/$(uname -r)/kernel/drivers/target/是否存在target_core_file.ko 、target_core_iblock.ko 、 target_core_mod.ko 、target_core_pscsi.ko 、target_core_user.ko缺少文件的话，尝试执行depmod再次进行查看，若没有解决只有更新内核系统了，可以尝试小版本更新yum update kernel && reboot查看。之后执行modprobe target_core_user,lsmod |grep target_core_user查看模块已加载。
最后
移除旧版本内核
1. 查看在使用内核uname -a或uname -r
2. 查看全部内核rpm -qa | grep kernel如下

[root@node4 ~]# rpm -qa | grep kernel
kernel-tools-3.10.0-1062.1.2.el7.x86_64
kernel-3.10.0-327.el7.x86_64
kernel-tools-libs-3.10.0-1062.1.2.el7.x86_64
kernel-3.10.0-1062.1.2.el7.x86_64
kernel-devel-3.10.0-1062.1.2.el7.x86_64

3. 移除旧版本内核yum remove kernel-3.10.0-327.el7.x86_64