【问题处理】因ASM磁盘组空间不足导致数据库实例无法启动的故障处理

1.问题现象
数据库无法启动，错误号ORA-03113
RACDB1@rac1 /home/oracle$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Nov 6 15:52:49 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area  167772160 bytes
Fixed Size                  1218292 bytes
Variable Size              88082700 bytes
Database Buffers           71303168 bytes
Redo Buffers                7168000 bytes
Database mounted.
ORA-03113: end-of-file on communication channel

2.问题分析
1）alert日志文件中记录了如下错误信息
alert中记录的信息
ARC2 started with pid=28, OS id=11242
Sat Nov  6 15:54:31 2010
Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:
ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbf
ORA-15041: diskgroup space exhausted
Sat Nov  6 15:54:31 2010
ARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'
ARC1: Failed to archive thread 1 sequence 21 (19504)
ARCH: Archival stopped, error occurred. Will continue retrying
Sat Nov  6 15:54:33 2010
ORACLE Instance RACDB1 - Archival Error
Sat Nov  6 15:54:33 2010
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'
Sat Nov  6 15:54:33 2010
Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'
Sat Nov  6 15:54:34 2010

2）trace文件中记录的信息如下
+ASM1@rac1 /home/oracle$ cat /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc
/u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name:    Linux
Node name:      rac1
Release:        2.6.18-53.el5xen
Version:        #1 SMP Wed Oct 10 17:06:12 EDT 2007
Machine:        i686
Instance name: RACDB1
Redo thread mounted by this instance: 1
Oracle process number: 26
Unix process pid: 11177, image: oracle@rac1 (ARC1)

*** SERVICE NAME:() 2010-11-06 15:54:30.927
*** SESSION ID:(144.1) 2010-11-06 15:54:30.927
kcrrwkx: work to do 0x1 (start)
Failed to create file '+DATA/racdb/1_21_733447664.dbf' (file not accessible?)
ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbf
ORA-15041: diskgroup space exhausted
*** 2010-11-06 15:54:31.432 60679 kcrr.c
ARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'
*** 2010-11-06 15:54:31.432 58941 kcrr.c
kcrrfail: dest:1 err:19504 force:0 blast:1
ARCH: Connecting to console port...
ARCH: Connecting to console port...
*** 2010-11-06 15:54:33.979 20145 kcrr.c
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'

3.问题原因
导致数据库实例无法启动的原因是Oracle ASM存放归档日志的磁盘组已被耗尽。因无法完成归档，导致数据库无法启动。

4.处理方法
1）第一种处理方法
如果可以将数据库调整为非归档模式，可以保证数据库实例正常启动。
如果是生产环境不建议这样操作。

2）第二种处理方法
使用asmcmd工具清理归档日志，释放磁盘组上的空间
具体操作过程如下：
RACDB1@rac1 /home/oracle$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
RACDB1@rac1 /home/oracle$ export ORACLE_SID=+ASM1
ASMCMD [+] > lsdg
State    Type    Rebal  Unbal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  EXTERN  N      N         512   4096  1048576      2047        7                0               7              0  DATA/
MOUNTED  EXTERN  N      N         512   4096  1048576      2047     1825                0            1825              0  FRA/
ASMCMD [+] > cd DATA
ASMCMD [+DATA] > cd RACDB
ASMCMD [+DATA/RACDB] > cd ARCHIVELOG
ASMCMD [+DATA/RACDB/ARCHIVELOG] > ls
2010_10_26/
2010_10_27/
2010_10_28/
2010_10_29/
2010_10_30/
2010_11_01/
2010_11_04/
ASMCMD [+DATA/RACDB/ARCHIVELOG] > du
Used_MB      Mirror_used_MB
    841                 841
ASMCMD [+DATA/RACDB/ARCHIVELOG] > rm -rf *
ASMCMD [+data/racdb] > lsdg
State    Type    Rebal  Unbal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  EXTERN  N      N         512   4096  1048576      2047      848                0             848              0  DATA/
MOUNTED  EXTERN  N      N         512   4096  1048576      2047     1825                0            1825              0  FRA/

此时数据库实例便可正常启动。

3）第三种处理方法
调整归档路径，或增加调整磁盘组大小。

5.小结
在数据库系统部署之前我们要对磁盘组大小充分的评估，以避免出现因空间不足导致系统故障，以及分配过多的空间造成磁盘和系统资源的浪费。

Good luck.

secooler
10.11.07

-- The End --

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/519536/viewspace-677785/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/519536/viewspace-677785/