rac一台实例突然无法启动

今天开虚拟机,rac一台实例突然无法启动:

[oracle@rac2 bdump]$ crs_stat -t
HA Resource                                   Target     State            
--------------------------------------------------------------------------
[oracle@rac2 bdump]$ crs_stat
HA Resource                                   Target     State            
--------------------------------------------------------------------------
ora.rac1.ASM1.asm                             ONLINE     ONLINE on rac1   
ora.rac1.LISTENER_RAC1.lsnr                   ONLINE     ONLINE on rac1   
ora.rac1.gsd                                  ONLINE     ONLINE on rac1   
ora.rac1.ons                                  ONLINE     ONLINE on rac1   
ora.rac1.vip                                  ONLINE     ONLINE on rac1   
ora.rac2.ASM2.asm                             ONLINE     ONLINE on rac2   
ora.rac2.LISTENER_RAC2.lsnr                   ONLINE     ONLINE on rac2   
ora.rac2.gsd                                  ONLINE     ONLINE on rac2   
ora.rac2.ons                                  ONLINE     ONLINE on rac2   
ora.rac2.vip                                  ONLINE     ONLINE on rac2   
ora.racdb.db                                  ONLINE     ONLINE on rac2   
ora.racdb.hchservice1.cs                      ONLINE     ONLINE on rac1   
ora.racdb.hchservice1.racdb2.srv              ONLINE     ONLINE on rac1   
ora.racdb.racdb1.inst                         ONLINE     ONLINE on rac1   
ora.racdb.racdb2.inst                         OFFLINE    OFFLINE          

 

第一反应就是去手动启动下实例:

SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/racdb/spfileracdb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/spfileracdb.ora
ORA-15077: could not locate ASM instance serving a required diskgroup

 

看这里就发现，从spfile启动开始就有问题了:

猜测是asm出问题:

[oracle@rac2 bdump]$ export ORACLE_SID=+ASM2
[oracle@rac2 bdump]$ sqlplus / as sysdba;

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Nov 22 15:12:18 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> set line 200;
SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA1                          MOUNTED
DATA2                          MOUNTED

 

看到这里就明白了，这里还有一个DATA没有挂载成功，在正常启动的rac1中:

[oracle@rac1 raw]$ export ORACLE_SID=+ASM1
[oracle@rac1 raw]$ sqlplus / as sysdba;

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Nov 22 15:14:09 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> select name ,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA1                          MOUNTED
DATA2                          MOUNTED
DATA                           MOUNTED

 

手动重启下rac2的asm实例，看下会报什么错:

SQL> shutdown immediate;
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup
ASM instance started

Total System Global Area  130023424 bytes
Fixed Size                  2094544 bytes
Variable Size             102763056 bytes
ASM Cache                  25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

 

这里显示data磁盘组加载出现问题，查看对应的alert:

NOTE: cache dismounting group 3/0x79818E72 (DATA)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DATA was not mounted
NOTE: cache opening disk 0 of grp 1: DATA1_0000 path:/dev/raw/raw8
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x5A318E70 (DATA1)
Thu Nov 22 15:15:17 CST 2012

 

对比实例rac1的日志可以发现，rwa/raw6裸设备没有加载:

[oracle@rac2 raw]$ ls
raw1  raw2  raw3  raw4  raw5  raw7  raw8

从这里可以看出来确实少了裸设备6:

[root@rac2 /]# cat /etc/sysconfig/rawdevices
# raw device bindings
# format:  <rawdev> <major> <minor>
#          <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
#          /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/sde1
/dev/raw/raw2 /dev/sdf1
/dev/raw/raw3 /dev/sdg1
/dev/raw/raw4 /dev/sdh1
/dev/raw/raw5 /dev/sdi1
/dev/raw/raw6 /dev/sdb1
/dev/raw/raw7 /dev/sdc1
/dev/raw/raw8 /dev/sdd1

查看资源情况：

手动启动下裸设备配置:

[root@rac2 /]# service rawdevices restart
Assigning devices:
           /dev/raw/raw1  -->   /dev/sde1
Error setting raw device (Device or resource busy)
           /dev/raw/raw2  -->   /dev/sdf1
Error setting raw device (Device or resource busy)
           /dev/raw/raw3  -->   /dev/sdg1
Error setting raw device (Device or resource busy)
           /dev/raw/raw4  -->   /dev/sdh1
Error setting raw device (Device or resource busy)
           /dev/raw/raw5  -->   /dev/sdi1
Error setting raw device (Device or resource busy)
           /dev/raw/raw6  -->   /dev/sdb1
/dev/raw/raw6:  bound to major 8, minor 17
           /dev/raw/raw7  -->   /dev/sdc1
Error setting raw device (Device or resource busy)
           /dev/raw/raw8  -->   /dev/sdd1
Error setting raw device (Device or resource busy)
done
[root@rac2 /]# cd /dev/raw
[root@rac2 raw]# ls
raw1  raw2  raw3  raw4  raw5  raw6  raw7  raw8

 

可以看到，现在有了raw6:

进入asm实例，mount下就可以了:

SQL> select name ,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA1                          DISMOUNTED
DATA2                          DISMOUNTED
DATA                           DISMOUNTED

SQL> ALTER DISKGROUP ALL MOUNT
  2  ;

Diskgroup altered.

SQL> select name ,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA1                          MOUNTED
DATA2                          MOUNTED
DATA                           MOUNTED