一次数据库服务器DOWN处理

[oracle@rac1 dbs]$ cd /opt/app/oracle/diag/rdbms/racdb/racdb1/trace
[oracle@rac1 trace]$ ll alert*

-rw-r----- 1 oracle asmadmin 17600371 07-05 10:02 alert_racdb1.log

RAC1:

Thread 1 cannot allocate new log, sequence 23033
Checkpoint not complete
  Current log# 5 seq# 23032 mem# 0: +DATA/racdb/onlinelog/group_5.322.817383997
  Current log# 5 seq# 23032 mem# 1: +RECOVERY/racdb/onlinelog/group_5.374.817383999
Thread 1 advanced to log sequence 23033 (LGWR switch)
  Current log# 2 seq# 23033 mem# 0: +DATA/racdb/onlinelog/group_2.282.817384559
  Current log# 2 seq# 23033 mem# 1: +RECOVERY/racdb/onlinelog/group_2.3280.817384559
Thu Jul 04 05:33:36 2013
Archived Log entry 41881 added for thread 1 sequence 23032 ID 0x3045bbae dest 1:
Thu Jul 04 05:34:06 2013
Thread 1 cannot allocate new log, sequence 23034
Checkpoint not complete
  Current log# 2 seq# 23033 mem# 0: +DATA/racdb/onlinelog/group_2.282.817384559
  Current log# 2 seq# 23033 mem# 1: +RECOVERY/racdb/onlinelog/group_2.3280.817384559
Thread 1 advanced to log sequence 23034 (LGWR switch)
  Current log# 1 seq# 23034 mem# 0: +DATA/racdb/onlinelog/group_1.281.817384703
  Current log# 1 seq# 23034 mem# 1: +RECOVERY/racdb/onlinelog/group_1.3281.817384703
Thu Jul 04 05:34:10 2013
Archived Log entry 41885 added for thread 1 sequence 23033 ID 0x3045bbae dest 1:
Thu Jul 04 05:40:00 2013
Errors in file /opt/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_j000_8047.trc:
ORA-12012: 自动执行作业 "SYS"."MONITOR_ORACLE_SYSTEM" 出错
ORA-06553: PLS-306: 调用 'MP_DBA_PERFORMANCE_WARNING' 时参数个数或类型错误
Thu Jul 04 05:40:33 2013

RAC2

Thu Jul 04 05:03:19 2013
Archived Log entry 41572 added for thread 2 sequence 18695 ID 0x3045bbae dest 1:
Thread 2 advanced to log sequence 18697 (LGWR switch)
  Current log# 4 seq# 18697 mem# 0: +DATA/racdb/onlinelog/group_4.290.817384729
  Current log# 4 seq# 18697 mem# 1: +RECOVERY/racdb/onlinelog/group_4.3277.817384729
Thu Jul 04 05:03:25 2013
Archived Log entry 41574 added for thread 2 sequence 18696 ID 0x3045bbae dest 1:
Thu Jul 04 05:03:27 2013
Thread 2 advanced to log sequence 18698 (LGWR switch)
  Current log# 6 seq# 18698 mem# 0: +DATA/racdb/onlinelog/group_6.302.817384029
  Current log# 6 seq# 18698 mem# 1: +RECOVERY/racdb/onlinelog/group_6.375.817384029
Thu Jul 04 05:03:30 2013
Archived Log entry 41575 added for thread 2 sequence 18697 ID 0x3045bbae dest 1:
Thread 2 cannot allocate new log, sequence 18699
Checkpoint not complete
  Current log# 6 seq# 18698 mem# 0: +DATA/racdb/onlinelog/group_6.302.817384029
  Current log# 6 seq# 18698 mem# 1: +RECOVERY/racdb/onlinelog/group_6.375.817384029
Thread 2 advanced to log sequence 18699 (LGWR switch)
  Current log# 3 seq# 18699 mem# 0: +DATA/racdb/onlinelog/group_3.289.817384603
  Current log# 3 seq# 18699 mem# 1: +RECOVERY/racdb/onlinelog/group_3.3279.817384603
Thu Jul 04 05:03:35 2013
Archived Log entry 41576 added for thread 2 sequence 18698 ID 0x3045bbae dest 1:
Thu Jul 04 05:03:39 2013
Thread 2 cannot allocate new log, sequence 18700

两个节点出现不能归档，CHECKPOINT 错误。

导致数据库挂掉了。

更深原因分析：

每天的归档日志删除代码执行出错，无法删除过期归档日志，

RMAN-08138: WARNING: archived log not deleted - must create more backups


可看到如下设置：

CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY

修改配置为：

CONFIGURE ARCHIVELOG DELETION POLICY TO none;

再删除归档日志

crosscheck archivelog all;
delete noprompt expired archivelog all;
delete noprompt archivelog until time 'sysdate-5';