Ceph OSD因为xfs fadvise(DONTNEED)问题无法启动

环境说明

Ceph 0.87

Centos 6.5 X86_64

问题描述

今天在启动一台OSD Server的时候，该OSD Server里面包含3个osds，分别为osd.5，osd.6，osd.7，osd.7无法正常启动

[root@ceph-osd-2 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      11.65   root default
-2      5.28            host ceph-osd-1
0       0.98                    osd.0   up      1
1       0.98                    osd.1   up      1
2       0.98                    osd.2   up      1
3       0.98                    osd.3   up      1
4       1.36                    osd.4   up      1
-3      2.73            host ceph-osd-2
5       0.91                    osd.5   up      1
6       0.91                    osd.6   up      1
7       0.91                    osd.7   down    1
-4      3.64            host ceph-osd-3
8       1.82                    osd.8   up      1
9       0.91                    osd.9   up      1
10      0.91                    osd.10  up      1

查看osd.7的日志信息

[root@ceph-osd-2 ~]# tail -f /var/log/ceph/ceph-osd.7.log
2014-12-23 10:08:36.481558 7fe21c3d9800  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 3554
2014-12-23 10:08:36.553743 7fe21c3d9800  0 filestore(/var/lib/ceph/osd/ceph-7) backend xfs (magic 0x58465342)
2014-12-23 10:08:36.553761 7fe21c3d9800  1 filestore(/var/lib/ceph/osd/ceph-7)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-12-23 10:08:36.625752 7fe21c3d9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is supported and appears to work
2014-12-23 10:08:36.625768 7fe21c3d9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-12-23 10:08:36.683817 7fe21c3d9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: syscall(SYS_syncfs, fd) fully supported
2014-12-23 10:08:36.683911 7fe21c3d9800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_feature: extsize is disabled by conf
2014-12-23 10:08:36.930178 7fe21c3d9800  0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-12-23 10:08:38.130686 7fe21c3d9800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-12-23 10:08:38.130694 7fe21c3d9800  1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 20: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0

解决方法

    在OSD Server中osd.7对应的硬盘为/dev/sdc，xfs挂载参数为rw,noexec,nodev,noatime,nodiratime,barrier=0，重新挂载sdc，并重启osd服务

#/etc/init.d/ceph stop osd
#umount /dev/sdc 
#mount/dev/sdc /osd3  ##/osd3为osd.7的挂载目录
#/etc/init.d/ceph start osd

  查看ceph osd状态，可以发现osd.7已经正常启动。

[root@ceph-osd-2 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      11.65   root default
-2      5.28            host ceph-osd-1
0       0.98                    osd.0   up      1
1       0.98                    osd.1   up      1
2       0.98                    osd.2   up      1
3       0.98                    osd.3   up      1
4       1.36                    osd.4   up      1
-3      2.73            host ceph-osd-2
5       0.91                    osd.5   up      1
6       0.91                    osd.6   up      1
7       0.91                    osd.7   up      1
-4      3.64            host ceph-osd-3
8       1.82                    osd.8   up      1
9       0.91                    osd.9   up      1
10      0.91                    osd.10  up      1