oracle中asm磁盘不足,【问题处理】因ASM磁盘组空间不足导致数据库实例无法启动的故障处理...

1.问题现象数据库无法启动，错误号ORA-03113RACDB1@rac1 /home/oracle$ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.1.0 - Production on Sat Nov 6 15:52:49 2010Copyright (c) 1982, 2005, Oracle.  All rights reserved.Connected to an idle instance.SQL> startup;ORACLE instance started.Total System Global Area  167772160 bytesFixed Size                  1218292 bytesVariable Size              88082700 bytesDatabase Buffers           71303168 bytesRedo Buffers                7168000 bytesDatabase mounted.ORA-03113: end-of-file on communication channel2.问题分析1)alert日志文件中记录了如下错误信息alert中记录的信息ARC2 started with pid=28, OS id=11242Sat Nov  6 15:54:31 2010Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbfORA-15041: diskgroup space exhaustedSat Nov  6 15:54:31 2010ARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'ARC1: Failed to archive thread 1 sequence 21 (19504)ARCH: Archival stopped, error occurred. Will continue retryingSat Nov  6 15:54:33 2010ORACLE Instance RACDB1 - Archival ErrorSat Nov  6 15:54:33 2010ORA-16038: log 2 sequence# 21 cannot be archivedORA-19504: failed to create file ""ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'Sat Nov  6 15:54:33 2010Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:ORA-16038: log 2 sequence# 21 cannot be archivedORA-19504: failed to create file ""ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'Sat Nov  6 15:54:34 20102)trace文件中记录的信息如下+ASM1@rac1 /home/oracle$ cat /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc/u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trcOracle Database 10g Enterprise Edition Release 10.2.0.1.0 - ProductionWith the Partitioning, Real Application Clusters, OLAP and Data Mining optionsORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1System name:    LinuxNode name:      rac1Release:        2.6.18-53.el5xenVersion:        #1 SMP Wed Oct 10 17:06:12 EDT 2007Machine:        i686Instance name: RACDB1Redo thread mounted by this instance: 1Oracle process number: 26Unix process pid: 11177, image: oracle@rac1 (ARC1)*** SERVICE NAME:() 2010-11-06 15:54:30.927*** SESSION ID:(144.1) 2010-11-06 15:54:30.927kcrrwkx: work to do 0x1 (start)Failed to create file '+DATA/racdb/1_21_733447664.dbf' (file not accessible?)ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbfORA-15041: diskgroup space exhausted*** 2010-11-06 15:54:31.432 60679 kcrr.cARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'*** 2010-11-06 15:54:31.432 58941 kcrr.ckcrrfail: dest:1 err:19504 force:0 blast:1ARCH: Connecting to console port...ARCH: Connecting to console port...*** 2010-11-06 15:54:33.979 20145 kcrr.cORA-16038: log 2 sequence# 21 cannot be archivedORA-19504: failed to create file ""ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'3.问题原因导致数据库实例无法启动的原因是Oracle ASM存放归档日志的磁盘组已被耗尽。因无法完成归档，导致数据库无法启动。4.处理方法1)第一种处理方法如果可以将数据库调整为非归档模式，可以保证数据库实例正常启动。如果是生产环境不建议这样操作。2)第二种处理方法使用asmcmd工具清理归档日志，释放磁盘组上的空间具体操作过程如下：RACDB1@rac1 /home/oracle$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1RACDB1@rac1 /home/oracle$ export ORACLE_SID=+ASM1ASMCMD [+] > lsdgState    Type    Rebal  Unbal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  NameMOUNTED  EXTERN  N      N         512   4096  1048576      2047        7                0               7              0  DATA/MOUNTED  EXTERN  N      N         512   4096  1048576      2047     1825                0            1825              0  FRA/ASMCMD [+] > cd DATAASMCMD [+DATA] > cd RACDBASMCMD [+DATA/RACDB] > cd ARCHIVELOGASMCMD [+DATA/RACDB/ARCHIVELOG] > ls2010_10_26/2010_10_27/2010_10_28/2010_10_29/2010_10_30/2010_11_01/2010_11_04/ASMCMD [+DATA/RACDB/ARCHIVELOG] > duUsed_MB      Mirror_used_MB841                 841ASMCMD [+DATA/RACDB/ARCHIVELOG] > rm -rf *ASMCMD [+data/racdb] > lsdgState    Type    Rebal  Unbal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  NameMOUNTED  EXTERN  N      N         512   4096  1048576      2047      848                0             848              0  DATA/MOUNTED  EXTERN  N      N         512   4096  1048576      2047     1825                0            1825              0  FRA/此时数据库实例便可正常启动。3)第三种处理方法调整归档路径，或增加调整磁盘组大小。5.小结在数据库系统部署之前我们要对磁盘组大小充分的评估，以避免出现因空间不足导致系统故障，以及分配过多的空间造成磁盘和系统资源的浪费。Good luck.secooler10.11.07-- The End --