前段时间在VM ESX上安装了两台Linux Server(版本:RHEL5U4 64Bit )做了Oracle RAC 11g,测试了一段时间RAC 的TAF特性,周一上班后重启两个节点,发现Oracle DB无法启动,两个节点分别报如下错误:
节点1:
[oracle@rac01 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Mon Nov 28 15:38:20 2011
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-16038: log 4 sequence# 56 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 4 thread 2: '+FRA/ora11g/onlinelog/group_4.271.767987875'
ORA-00312: online log 4 thread 2: '+FRA/ora11g/onlinelog/group_4.272.767987877'
预警文件中出现如下错误:
*************************************************************
ARC2: Error 19504 Creating archive log file to '+FRA'
Errors in file /u01/product/diag/rdbms/ora11g/ora11g1/trace/ora11g1_arc1_5550.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-17502: ksfdcre:4 Failed to create file +FRA
ORA-15041: diskgroup "FRA" space exhausted
*************************************************************
WARNING: A file of type ARCHIVED LOG may exist in
db_recovery_file_dest that is not known to the database.
Use the RMAN command CATALOG RECOVERY AREA to re-catalog
any such files. If files cannot be cataloged, then manually
delete them using OS command. This is most likely the
result of a crash during file creation.
*************************************************************
节点2:
[root@rac02 ~]# su - oracle
[oracle@rac02 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Mon Nov 28 15:21:39 2011
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup;
ORACLE instance started.
Total System Global Area 425897984 bytes
Fixed Size 2229024 bytes
Variable Size 268438752 bytes
Database Buffers 150994944 bytes
Redo Buffers 4235264 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 5250
Session ID: 34 Serial number: 3
预警文件中出现如下错误:
Errors in file /u01/product/diag/rdbms/ora11g/ora11g2/trace/ora11g2_ora_6964.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-17502: ksfdcre:4 Failed to create file +FRA
ORA-15041: diskgroup "FRA" space exhausted
Oracle DB的归档日志路径设置在ASM磁盘组“FRA"下,究其原因是由于"FRA"剩余空间不足,导致归档无法生成。使用ASM的磁盘管理工具:ASMCMD ,可以直接查看ASM磁盘组的内容并对其做相应的操作。这里我们可以删除磁盘组”FRA"下的归档(或其它不再需要的文件),操作过程如下
1) 以oracle grid infrastructure软件所有者的身份登录到节点1,设置ASM实例名:
Normal
0
0
2
false
false
false
MicrosoftInternetExplorer4
export ORACLE_SID=+ASM1
2)运行asmcmd工具:asmcmd
3)查看剩余空间:lsdg
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 10236 9840 0 9840 0 Y CRS/
MOUNTED EXTERN N 512 4096 1048576 20473 20378 0 20378 0 N DATA/
MOUNTED EXTERN N 512 4096 1048576 10236 6 0 6 0 N FRA/
4)进入相应路径删除归档:ASMCMD> cd +FRA/ORA11G/ARCHIVELOG
ASMCMD> ls
2011_11_23/
2011_11_24/
2011_11_25/
2011_11_26/
ASMCMD> rm -rf *
5)退出asmcmd工具后,即可正常开启Oracle DB