服务器硬盘变missing了,AIX 磁盘状态为missing问题的解决

背景

一年前由于本地备份空间不够，加入了一个存储上面的磁盘(LUN),这样就造成了备份用的这个VG是本地磁盘和存储磁盘混用了，最近通过监控发现存储其中一个控制器存在故障，初步诊断为光纤模块存在问题，这个时候如果另外一个控制器出现故障就会导致整个存储不能用，防止单点故障需要停存储进行更换控制器，在维护过程中我们只把存储停了，但是AIX主机并没有关机，当存储启来后我们重启了操作系统

AIX操作系统版本:

# oslevel -s

6100-06-01-1043

存储类型

IBM DS5200

问题描述和诊断过程

1.当重启操作系统后发现/backup挂载不上，手动挂载报错如下：

bash-3.00#

mount /backup

Replaying

log for /dev/backuplv.

Fatal:

I/O error

mount:

0506-324 Cannot mount /dev/backuplv on /backup: The media is not formatted or

the format is not correct.

0506-342

The superblock on /dev/backuplv is dirty.  Run a full fsck to fix.

bash-3.00#

2.根据上面提示手动执行fsck后问题还是一样，查看对应用的LV，再根据对应的LV查看对应的磁盘，发现其中一磁盘，也就是存储上面的一个磁盘状态为missing

bash-3.00#

more /etc/filesystems

/backup:

dev             =

/dev/backuplv

vfs             =

jfs2

log             =

/dev/loglv00

mount           = true

options         = rw

account         = false

bash-3.00# lsvg -l  backupvg

backupvg:

LV NAME          TYPE

LPs    PPs   PVs  LV

STATE     MOUNT POINT

backuplv           jfs2

3600    3600   3    open/syncd    /backup

loglv00             jfs2log    1

1       1    open/syncd    N/A

bash-3.00# lspv|grep backupvg

hdisk2          00f6886135525b61                  backupvg

active

hdisk3          00f6886135525c3b                  backupvg

active

hdisk20         00f6886132cfde40                   backupvg

missing

bash-3.00# lspv hdisk20

PHYSICAL VOLUME:

hdisk20                  VOLUME

GROUP:     backupvg

PV IDENTIFIER:

00f6886132cfde40 VG IDENTIFIER

00f6886100004c000000012f35525ca0

PV STATE:           missing

STALE PARTITIONS:

0

ALLOCATABLE:      yes

PP SIZE:

256 megabyte(s)          LOGICAL

VOLUMES:  0

TOTAL PPs:

1459 (373504 megabytes)  VG

DESCRIPTORS:   1

FREE PPs:

1459 (373504 megabytes)  HOT

SPARE:        no

USED PPs:

0 (0 megabytes)          MAX

REQUEST:      256 kilobytes

FREE DISTRIBUTION:

292..292..291..292..292

USED DISTRIBUTION:

00..00..00..00..00

MIRROR POOL:

None

bash-3.00#

3.试图使用smit chpv把Physical volume STATE从no active改为active,执行过程中没有报错，但是执行成功后，状态还是没有变，说明这个方法行不通

上面执行成功了，状态和原来一样，说明这个方法无效

bash-3.00# lspv hdisk20

PHYSICAL VOLUME:    hdisk20                  VOLUME GROUP:     backupvg

PV IDENTIFIER:      00f6886132cfde40 VG IDENTIFIER     00f6886100004c000000012f35525ca0

PV STATE:           missing

STALE PARTITIONS:   0                        ALLOCATABLE:      yes

PP SIZE:            256 megabyte(s)          LOGICAL VOLUMES:  0

TOTAL PPs:          1459 (373504 megabytes)  VG DESCRIPTORS:   1

FREE PPs:           1459 (373504 megabytes)  HOT SPARE:        no

USED PPs:           0 (0 megabytes)          MAX REQUEST:      256 kilobytes

FREE DISTRIBUTION:  292..292..291..292..292

USED DISTRIBUTION:  00..00..00..00..00

MIRROR POOL:        None

bash-3.00#

解决方案

因为这个目录都是备份用的，里面数据不是很重要，所以打算重建，我想一定有不需要破坏原数据的就能把这个磁盘改为active的方法，先记录一下，等有空再找一下

重建步骤如下：

先使用smit

rmfs把backup文件和LV删除再把这个这个磁盘从backupvg中卸载，最后把这个磁盘添加到backupvg中去，以下为操作日志

bash-3.00# smit reducevg

Remove a Physical Volume from a Volume Group

Type or select values in entry

fields.

Press Enter AFTER making all

desired changes.

[Entry Fields]

* VOLUME GROUP name                                   backupvg

* PHYSICAL VOLUME names                              [hdisk20]

+

FORCE deallocation of all partitions on             no

+

this physical volume?

F1=Help

F2=Re+------------------------------------------------------+     F4=List

Esc+5=Reset                      Esc+6| Running

command...

|     Esc+8=Image

Esc+9=Shell                      Esc+0+------------------------------------------------------+

COMMAND STATUS

Command: running       stdout: no            stderr: no

Before command completion,

additional instructions may appear below.

Reduce a Volume Group

Move cursor to desired item and

press Enter.

Remove a Physical Volume from a Volume Group

Remove a Volume Group

F1=Help                          F2=Refresh                       F3=Cancel                        Esc+8=Image

Esc+9=Shell                      Esc+0=Exit                       Enter=Do

bash-3.00# lspv

hdisk0          00f68861b9e00f54                    rootvg          active

hdisk1          00f688611026530f                    rootvg          active

hdisk2          00f6886135525b61                    backupvg        active

hdisk3          00f6886135525c3b                    backupvg        active

hdisk20         00f6886132cfde40                    None

bash-3.00# smit extendvg

Add a Physical Volume to a Volume Group

Type or select values in entry

fields.

Press Enter AFTER making all

desired changes.

[Entry Fields]

Force the creation of a volume group?               no

+

* VOLUME GROUP name

[backupvg]                                                                  +

* PHYSICAL VOLUME names                              [hdisk20]

+

COMMAND STATUS

Command: running       stdout: yes           stderr: no

Before command completion,

additional instructions may appear below.

0516-1398 /usr/sbin/extendvg: The

physical volume hdisk20, appears to belong to

another volume group. Use the

force option to add this physical volume

to a volume group.

0516-792 /usr/sbin/extendvg:

Unable to extend volume group.

Add a

Physical Volume to a Volume Group

Type or select values in entry

fields.

Press Enter AFTER making all desired

changes.

[Entry Fields]

Force the creation of a volume group?               yes

* VOLUME GROUP name                          [backupvg]

* PHYSICAL VOLUME names                              [hdisk20]

bash-3.00# lspv hdisk20

PHYSICAL VOLUME:    hdisk20           VOLUME GROUP:     backupvg

PV IDENTIFIER: 00f6886132cfde40

VG IDENTIFIER

00f6886100004c000000012f35525ca0

PV STATE:

active

STALE PARTITIONS:   0                        ALLOCATABLE:      yes

PP SIZE:            256 megabyte(s)          LOGICAL VOLUMES:  0

TOTAL PPs:          1459 (373504 megabytes)  VG DESCRIPTORS:   1

FREE PPs:           1459 (373504 megabytes)  HOT SPARE:        no

USED PPs:           0 (0 megabytes)          MAX REQUEST:      256 kilobytes

FREE DISTRIBUTION:  292..292..291..292..292

USED DISTRIBUTION:  00..00..00..00..00

MIRROR POOL:        None

bash-3.00#

总结

在维护存储时，如果有服务器挂载了这些磁盘，务必停掉服务器或者umount相关的文件系统再进行操作