问题解决记录 - KRBD Map失败

问题：

centos6.5，升级kernel到3.10.84

rbd create test1无报错

rbd info test1无报错显示正常

rbd map test1出现问题

开始是没有响应，等待一段时间后，出现：

rbd: sysfs write failed

rbd: map failed: (5) Input/output error

解决过程：

应该是内核rbd模块无法读取rbd信息所致。

怀疑可能是image 数据损坏，重新建了两个image，都出现一样的问题，排除这个可能。

查看dmesg发现有错误输出：

mon1 xxxxxxx:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 1000000000000

经过一番搜索，在下面这个文章找到了解药：

http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client/

对照feature表，应该是当前内核版本缺少  CEPH_FEATURE_CRUSH_V4 **** 这个特性支持。

而这个特性在4.1版本内核才被支持，升级内核不现实，只能在服务端关闭这个特性。

通过ceph osd crush dump看到：

"tunables": {
        "choose_local_tries": 0,
        "choose_local_fallback_tries": 0,
        "choose_total_tries": 50,
        "chooseleaf_descend_once": 1,
        "chooseleaf_vary_r": 0,
        "straw_calc_version": 1,
        "allowed_bucket_algs": 22,
        "profile": "unknown",
        "optimal_tunables": 0,
        "legacy_tunables": 0,
        "require_feature_tunables": 1,
        "require_feature_tunables2": 1,
        "require_feature_tunables3": 0,
        "has_v2_rules": 0,
        "has_v3_rules": 0,
        "has_v4_buckets": 1
    }

最后这个特性显得比较诡异，为什么被打开了呢？

尝试把crushmap导出，用crushtool反编译成文本，加上一行：

tunbale has_v4_buckets 0

重新编译发现不能编译，提示这并不是一个合法的配置项。

于是连蒙带猜，认为这个配置项不是手工配置的，而是由其它配置自动产生的。

straw2是一个跟crush相关的新特性，很有可能与之有关。

检查crushmap，发现其中的bucket的alg全部配置成了straw2(不知道哪个哥们干的)

于是将其改回straw，重新导入crushmap。

发现has_v4_buckets特性的值已经变为0了。

尝试重新map image:  rbd map test1

终于成功了~~

收获：

以后用KRBD时，可以通过查表清楚的知道哪些版本支持什么样的特性啦。

为防止老外删除文章，我还是把表格拷贝过来比较保险：

Feature	BIT	OCT	3.8	3.9	3.10	3.14	3.15	3.18	4.1
CEPH_FEATURE_UID	0	1
CEPH_FEATURE_NOSRCADDR	1	2	R	R	R	R	R	R	R
CEPH_FEATURE_MONCLOCKCHECK	2	4
CEPH_FEATURE_FLOCK	3	8
CEPH_FEATURE_SUBSCRIBE2	4	10
CEPH_FEATURE_MONNAMES	5	20
CEPH_FEATURE_RECONNECT_SEQ	6	40			-R-	R	R	R	R
CEPH_FEATURE_DIRLAYOUTHASH	7	80
CEPH_FEATURE_OBJECTLOCATOR	8	100
CEPH_FEATURE_PGID64	9	200		R	R	R	R	R	R
CEPH_FEATURE_INCSUBOSDMAP	10	400
CEPH_FEATURE_PGPOOL3	11	800		R	R	R	R	R	R
CEPH_FEATURE_OSDREPLYMUX	12	1000
CEPH_FEATURE_OSDENC	13	2000		R	R	R	R	R	R
CEPH_FEATURE_OMAP	14	4000
CEPH_FEATURE_MONENC	15	8000
CEPH_FEATURE_QUERY_T	16	10000
CEPH_FEATURE_INDEP_PG_MAP	17	20000
CEPH_FEATURE_CRUSH_TUNABLES	18	40000	S	S	S	S	S	S	S
CEPH_FEATURE_CHUNKY_SCRUB	19	80000
CEPH_FEATURE_MON_NULLROUTE	20	100000
CEPH_FEATURE_MON_GV	21	200000
CEPH_FEATURE_BACKFILL_RESERVATION	22	400000
CEPH_FEATURE_MSG_AUTH	23	800000						-S-	S
CEPH_FEATURE_RECOVERY_RESERVATION	24	1000000
CEPH_FEATURE_CRUSH_TUNABLES2	25	2000000		S	S	S	S	S	S
CEPH_FEATURE_CREATEPOOLID	26	4000000
CEPH_FEATURE_REPLY_CREATE_INODE	27	8000000		S	S	S	S	S	S
CEPH_FEATURE_OSD_HBMSGS	28	10000000
CEPH_FEATURE_MDSENC	29	20000000
CEPH_FEATURE_OSDHASHPSPOOL	30	40000000		S	S	S	S	S	S
CEPH_FEATURE_MON_SINGLE_PAXOS	31	80000000
CEPH_FEATURE_OSD_SNAPMAPPER	32	100000000
CEPH_FEATURE_MON_SCRUB	33	200000000
CEPH_FEATURE_OSD_PACKED_RECOVERY	34	400000000
CEPH_FEATURE_OSD_CACHEPOOL	35	800000000				-S-	S	S	S
CEPH_FEATURE_CRUSH_V2	36	1000000000				-S-	S	S	S
CEPH_FEATURE_EXPORT_PEER	37	2000000000				-S-	S	S	S
CEPH_FEATURE_OSD_ERASURE_CODES***	38	4000000000
CEPH_FEATURE_OSD_TMAP2OMAP	38*	4000000000
CEPH_FEATURE_OSDMAP_ENC	39	8000000000					-S-	S	S
CEPH_FEATURE_MDS_INLINE_DATA	40	10000000000
CEPH_FEATURE_CRUSH_TUNABLES3	41	20000000000					-S-	S	S
CEPH_FEATURE_OSD_PRIMARY_AFFINITY	41*	20000000000					-S-	S	S
CEPH_FEATURE_MSGR_KEEPALIVE2	42	40000000000
CEPH_FEATURE_OSD_POOLRESEND	43	80000000000
CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2	44	100000000000
CEPH_FEATURE_OSD_SET_ALLOC_HINT	45	200000000000
CEPH_FEATURE_OSD_FADVISE_FLAGS	46	400000000000
CEPH_FEATURE_OSD_REPOP	46*	400000000000
CEPH_FEATURE_OSD_OBJECT_DIGEST	46*	400000000000
CEPH_FEATURE_OSD_TRANSACTION_MAY_LAY	46*	400000000000
CEPH_FEATURE_MDS_QUOTA	47	800000000000
CEPH_FEATURE_CRUSH_V4 ****	48	1000000000000							-S-
CEPH_FEATURE_OSD_MIN_SIZE_RECOVERY	49	2000000000000
CEPH_FEATURE_OSD_PROXY_FEATURES	49*	4000000000000