osd 故障测试

1. 创建的 volumes 是否可以导出数据

 

查询当前池

[root@tt-ceph-048146 ~]# ceph osd  lspools
0 rbd,1 volumes,

查询 volumes 池中的卷

[root@tt-ceph-048144 ceph]# rbd -p volumes ls
volume-be34745e-165f-4fc6-86c4-6248173abc00

导出卷

[root@tt-ceph-048144 ceph]# rbd -p volumes export  be34745e-165f-4fc6-86c4-6248173abc00  /tmp/volume-test
2015-08-04 17:07:46.071512 7fcec6ada880 -1 librbd::ImageCtx: error finding header: (2) No such file or directory
rbd: error opening image be34745e-165f-4fc6-86c4-6248173abc00: (2) No such file or directory
[root@tt-ceph-048144 ceph]# rbd -p volumes export volume-be34745e-165f-4fc6-86c4-6248173abc00  /tmp/volume-test

导出的卷只能够用于 ceph 导出, 而不可以令 linux 直接执行读写

[root@tt-ceph-048144 ceph]#  dumpe2fs  /tmp/volume-test
dumpe2fs 1.42.9 (28-Dec-2013)
dumpe2fs: Bad magic number in super-block 当尝试打开 /tmp/volume-test 时
找不到有效的文件系统超级块.

 

2. vm 进行数据写入时候,  是否会对 mon 节点保持长连接,  是否会对 ceph  osd 节点保持长连接

 

经测试, 当 vm 云盘后, 对应物理机即会产生一常连接到  mon 节点,  只有在 vm 产生数据写时候, 才会与 osd 节点产生对应的 TCP 连接

MON 节点的常连接并不会全部都集中到其中的一台 MON 节点中, 常连接会比较平均地分布到各个 mon 节点中

 

参考生产环境上的 osd 连接

第一个 mon 节点, 有 13 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.212; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.128.30:47557
10.198.128.35:53771
10.198.128.37:58485
10.198.128.32:33849
10.198.128.38:46272
10.198.128.32:33819
10.198.128.33:40363
10.198.128.31:45110
10.198.128.32:56079
10.198.128.30:59187
10.198.128.32:33836
10.198.128.30:34132
10.198.128.38:46288
 
第二个 mon 节点, 有 7 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.214; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.129.72:44896
10.198.128.38:47764
10.198.128.33:51101
10.198.128.38:56206
10.198.128.32:44929
10.198.128.31:37976
10.198.128.37:56534
 
第三个 mon 节点, 有 12 个常连接
[root@sh-ceph-128212 ~]# for ip in 10.198.128.216; do  ssh $ip  "netstat -nt | grep 6789" | awk '{print $5}' |  grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219'   ; done
10.198.128.31:37590
10.198.128.32:49009
10.198.128.36:52632
10.198.128.36:52615
10.198.128.30:54464
10.198.128.32:45783
10.198.128.36:46733
10.198.128.35:34866
10.198.128.36:46793
10.198.128.37:54421
10.198.128.33:52135
10.198.128.36:46713

 

3. 当 vm 进行数据写入时候,  断开一个 mon 节点, 是否会对数据写入造成影响

 

当前测试 compute 192.168.209.106 mon 192.168.48.144, 192.168.48.146, 192.168.48.148

1.  当前只有 192.168.48.144 节点中出现与客户的常连接
2.  让 compute 节点中的 vm 进行大数据复制 (1gb 左右数据复制), 产生了磁盘 IO
3.  这个时候利用 kill 命令删除 192.168.48.144 的 mon 进程, 并利用 ps 命令进行验证
4.  当 mon 进程被杀死后, 192.168.209.106 与 192.168.48.148 创建了新的 mon 常连接
5.  数据复制没有被影响

mon 进程不会自动恢复,  需要手动对 mon 进行启动

[root@tt-ceph-048144 ceph]# ceph -s
  cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
  health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
    osdmap e245: 30 osds: 30 up, 30 in
    pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
         46119 MB used, 854 GB / 899 GB avail
         10304 active+clean

重启启动了 mon 服务后, 恢复正常

[root@tt-ceph-048144 ceph]# ceph -s
  cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
  health HEALTH_OK
    monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e246: 30 osds: 30 up, 30 in
     pgmap v1016: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
         46119 MB used, 854 GB / 899 GB avail
         10304 active+clean

 

4. 当 vm 进行数据写入时候, 断开一个 osd 节点, 是否会对数据写入造成影响

 

1. 在 vm 写入数据的过程中, 关闭一台 osd 节点
2. 在数据完成后, 验证数据完成性
3. 重启启动 osd 节点,  发现 ceph 集群会自动恢复
4. ceph 集群完整性没有被破坏

 

在进行数据复制过程中, 把其中一台的 osd 进程全部 kill 掉

[root@tt-ceph-048144 ceph]# ceph osd tree
2015-08-04 14:57:32.063320 7fccf41a9700 0 -- :/1007979 >> 192.168.48.144:6789/0 pipe(0x7fccf00298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fccf0029b70).fault
# id weight type name up/down reweight
-1 30 root default
-2 5 host tt-ceph-048144
0 1 osd.0 up 1
1 1 osd.1 up 1
2 1 osd.2 up 1
3 1 osd.3 up 1
4 1 osd.4 up 1
-3 5 host tt-ceph-048145
5 1 osd.5 up 1
6 1 osd.6 up 1
7 1 osd.7 up 1
8 1 osd.8 up 1
9 1 osd.9 up 1
-4 5 host tt-ceph-048146
10 1 osd.10 up 1
11 1 osd.11 up 1
12 1 osd.12 up 1
13 1 osd.13 up 1
14 1 osd.14 up 1
-5 5 host tt-ceph-048147
15 1 osd.15 up 1
16 1 osd.16 up 1
17 1 osd.17 up 1
18 1 osd.18 up 1
19 1 osd.19 up 1
-6 5 host tt-ceph-048148
20 1 osd.20 up 1
21 1 osd.21 up 1
22 1 osd.22 up 1
23 1 osd.23 up 1
24 1 osd.24 up 1
-7 5 host tt-ceph-048149
25 1 osd.25 down 1
26 1 osd.26 down 1
27 1 osd.27 down 1
28 1 osd.28 down 1
29 1 osd.29 down 1

 

检查 ceph 集群, 发现发生故障

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 5176 pgs degraded; 5171 pgs stuck unclean; 5176 pgs undersized; recovery 471/2598 objects degraded (18.129%); 5/30 in osds are down; 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e239: 30 osds: 25 up, 30 in
      pgmap v924: 10304 pgs, 2 pools, 3392 MB data, 866 objects
            42279 MB used, 858 GB / 899 GB avail
            471/2598 objects degraded (18.129%)
                5128 active+clean
                5176 active+undersized+degraded

重新启动 osd, 发现有数据恢复现象

[root@tt-ceph-048144 ceph]# ceph -s
2015-08-04 14:58:26.309245 7f7eb818b700  0 -- :/1008324 >> 192.168.48.144:6789/0 pipe(0x7f7eb40298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7eb4029b70).fault
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 3089 pgs degraded; 3085 pgs stuck unclean; 3089 pgs undersized; recovery 325/2835 objects degraded (11.464%); 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v941: 10304 pgs, 2 pools, 3708 MB data, 945 objects
            43958 MB used, 856 GB / 899 GB avail
            325/2835 objects degraded (11.464%)
                7215 active+clean
                3089 active+undersized+degraded
recovery io 12881 kB/s, 3 objects/s
  client io 3969 kB/s wr, 2 op/s

恢复后, 发现, 数据正常了

[root@tt-ceph-048144 ceph]# ceph -s
2015-08-04 14:58:44.340787 7fd72bd78700  0 -- :/1008446 >> 192.168.48.144:6789/0 pipe(0x7fd7300298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd730029b70).fault
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v952: 10304 pgs, 2 pools, 3925 MB data, 999 objects
            45058 MB used, 855 GB / 899 GB avail
               10304 active+clean
  client io 13701 kB/s rd, 20372 kB/s wr, 119 op/s

重启一下故障的 mon 服务

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148
     osdmap e245: 30 osds: 30 up, 30 in
      pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects
            46119 MB used, 854 GB / 899 GB avail
               10304 active+clean

 

第二次测试
数据复制正常

[root@tt-ceph-048144 ceph]# ceph -s
    cluster 60391f4d-734f-425d-89b2-1fcb26c876c8
     health HEALTH_OK
     monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148
     osdmap e248: 30 osds: 30 up, 30 in
      pgmap v1052: 10304 pgs, 2 pools, 4680 MB data, 1188 objects
            46095 MB used, 854 GB / 899 GB avail
               10304 active+clean
  clien