nfs客户端进程变D,延伸linux的lock

最新推荐文章于 2023-01-07 15:53:06 发布

weixin_33905756

最新推荐文章于 2023-01-07 15:53:06 发布

阅读量429

点赞数

文章标签： python 网络

原文链接：https://segmentfault.com/a/1190000009056709

版权

问题现象

客户端是opentack，通过nfs挂载我们存储，发现有个服务进程变D了，长时间无法恢复，如下

[root@ECM-043 ~]# ps aux|grep nova-compute
nova     21409  1.4  0.0 2117692 110916 ?      Dl   15:06   1:24 /opt/server/python27/bin/python /usr/bin/nova-compute --logfile /var/log/nova/compute.log

排查步骤

客户端排查，查找该进程hang 在哪里，lsof 查找进程打开的文件


lsof -p 21409 
nova-comp 21409 nova   22w   REG               0,29        0 2199054015793 /var/lib/nova/instances/locks/nova-storage-registry-lock (block.beijing.wocloud.cn:/var/share/ezfs/shareroot/block-bj)
可以看到客户端的进程打开了nfs上的nova-storage-registry-lock文件，下一步在服务端看下该文件的状态

服务端查看nova-storage-registry-lock 文件的inode号

知道了inode号

在所有的nfs服务端，看下是否有其他进程拿着该lock 文件

root@Storage5:~# cat /proc/locks |grep inode
1: POSIX  ADVISORY  WRITE 28331 00:14:2199053494291 0 EOF
2: POSIX  ADVISORY  WRITE 28165 00:14:1099511627783 0 0
3: POSIX  ADVISORY  READ  360769 00:0f:21590 4 4
4: POSIX  ADVISORY  WRITE 28331 00:14:2199054015793 0 EOF
5: POSIX  ADVISORY  WRITE 1022761 00:0f:673303453 0 EOF
6: POSIX  ADVISORY  WRITE 98842 00:0f:47181 0 EOF
7: POSIX  ADVISORY  WRITE 688677 00:0f:1186403778 0 EOF
8: POSIX  ADVISORY  READ  38754 00:0f:21590 4 4

服务端确认哪些 nfs客户端连接

nfs 是无状态协议，和cifs不一样，不是TCP 吗？

root@Storage5:/var/lib/nfs/sm# ll
total 48
drwxr-xr-x 2 statd root 4096 Apr 13 09:03 ./
drwxr-xr-x 5 statd root 4096 Apr 10 18:21 ../
-rw------- 1 statd root   88 Apr 11 16:08 10.55.4.1
-rw------- 1 statd root   89 Apr 10 14:47 10.55.4.15
-rw------- 1 statd root   89 Apr 10 14:41 10.55.4.16
-rw------- 1 statd root   90 Apr  7 02:10 10.55.4.199
-rw-r----- 1 statd root 1350 Apr 11 13:42 10.55.4.205
-rw------- 1 statd root   89 Apr 11 16:00 10.55.4.31
-rw------- 1 statd root   89 Apr 10 14:44 10.55.4.32
-rw------- 1 statd root   89 Apr 10 14:34 10.55.4.37
-rw------- 1 statd root   89 Apr 11 16:04 10.55.4.54
-rw-r----- 1 statd root 2288 Apr 13 09:03 10.55.4.9
root@Storage5:/var/lib/nfs/sm# pwd
/var/lib/nfs/sm

延伸

linux下lock类型

文件锁，主要分为 flock 和fcntl

2者的粒度不一样

flock

锁住整个文件，如下，

root@scal61:/usr/share/pyshared/ezs3# cat /proc/locks 
1: FLOCK  ADVISORY  WRITE 97862 00:12:229635 0 EOF
2: FLOCK(lock类型）  ADVISORY（建议锁，非强制执行）  WRITE（holder能写lock文件） 4246（holder的pid） 00:12:20704（锁文件的MAJOR-DEVICE:MINOR-DEVICE:INODE-NUMBER） 0 EOF（锁文件的范围，0到EOF代表整个文件）

fcntl

lock文件的某一部分，如下 posix

root@scal61:/usr/share/pyshared/ezs3# cat /proc/locks 
3: POSIX  ADVISORY  WRITE 97830 08:03:524404 0 EOF
4: POSIX  ADVISORY  WRITE 4056 00:12:41028 0 EOF
5: POSIX  ADVISORY  READ  2351 00:12:10095 4 4
6: POSIX  ADVISORY  READ  2783 00:12:10052 4 4
7: POSIX  ADVISORY  READ  2783 00:12:14432 4 4
8: POSIX  ADVISORY  WRITE 2783 00:12:10050 0 0

* centos

FLOCK signifying the older-style UNIX file locks from a flock system
call
POSIX representing the newer POSIX locks from the lockf system call.

ADVISORY means that the lock does not prevent other people from accessing the data; it only prevents other attempts to lock it
MANDATORY means that no other access to the data is permitted while the lock is held
The fourth column reveals whether the lock is allowing the holder READ or WRITE access to the file