oracle中asm磁盘不足,一次关于Oracle RAC 11g ASM磁盘组剩余空间不足导致数据库无法启动的解决...

前段时间在VM ESX上安装了两台Linux Server(版本:RHEL5U4 64Bit )做了Oracle RAC 11g，测试了一段时间RAC 的TAF特性，周一上班后重启两个节点，发现Oracle DB无法启动，两个节点分别报如下错误：

节点1：

[oracle@rac01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Nov 28 15:38:20 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE

--------------------

MOUNTED

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-16038: log 4 sequence# 56 cannot be archived

ORA-19504: failed to create file ""

ORA-00312: online log 4 thread 2: '+FRA/ora11g/onlinelog/group_4.271.767987875'

ORA-00312: online log 4 thread 2: '+FRA/ora11g/onlinelog/group_4.272.767987877'

预警文件中出现如下错误：

*************************************************************

ARC2: Error 19504 Creating archive log file to '+FRA'

Errors in file /u01/product/diag/rdbms/ora11g/ora11g1/trace/ora11g1_arc1_5550.trc:

ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.

ORA-17502: ksfdcre:4 Failed to create file +FRA

ORA-15041: diskgroup "FRA" space exhausted

*************************************************************

WARNING: A file of type ARCHIVED LOG may exist in

db_recovery_file_dest that is not known to the database.

Use the RMAN command CATALOG RECOVERY AREA to re-catalog

any such files. If files cannot be cataloged, then manually

delete them using OS command. This is most likely the

result of a crash during file creation.

*************************************************************

节点2：

[root@rac02 ~]# su - oracle

[oracle@rac02 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Nov 28 15:21:39 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;

ORACLE instance started.

Total System Global Area  425897984 bytes

Fixed Size                  2229024 bytes

Variable Size             268438752 bytes

Database Buffers          150994944 bytes

Redo Buffers                4235264 bytes

Database mounted.

ORA-03113: end-of-file on communication channel

Process ID: 5250

Session ID: 34 Serial number: 3

预警文件中出现如下错误：

Errors in file /u01/product/diag/rdbms/ora11g/ora11g2/trace/ora11g2_ora_6964.trc:

ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.

ORA-17502: ksfdcre:4 Failed to create file +FRA

ORA-15041: diskgroup "FRA" space exhausted

Oracle DB的归档日志路径设置在ASM磁盘组“FRA"下，究其原因是由于"FRA"剩余空间不足，导致归档无法生成。使用ASM的磁盘管理工具：ASMCMD ，可以直接查看ASM磁盘组的内容并对其做相应的操作。这里我们可以删除磁盘组”FRA"下的归档(或其它不再需要的文件)，操作过程如下

1) 以oracle grid infrastructure软件所有者的身份登录到节点1，设置ASM实例名：

Normal

0

0

2

false

false

false

MicrosoftInternetExplorer4

export ORACLE_SID=+ASM1

2)运行asmcmd工具：asmcmd

3)查看剩余空间：lsdg

ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512   4096  1048576     10236     9840                0            9840              0             Y  CRS/

MOUNTED  EXTERN  N         512   4096  1048576     20473    20378                0           20378              0             N  DATA/

MOUNTED  EXTERN  N         512   4096  1048576     10236        6                0               6              0             N  FRA/

4)进入相应路径删除归档：ASMCMD> cd +FRA/ORA11G/ARCHIVELOG

ASMCMD> ls

2011_11_23/

2011_11_24/

2011_11_25/

2011_11_26/

ASMCMD> rm -rf *

5)退出asmcmd工具后，即可正常开启Oracle DB