环境:CentOS6.5 + Oracle RAC 11.2.0.4.0(xxxdb1,xxxdb2两个实例)
问题介绍:实例xxxdb1节点多次因ORA-000227和ORA-00202导致宕机,宕机后通过srvctl命令可以重新启动实例。
RAC集群大部分业务只访问这个节点,控制文件current.257.1024919441是近期通过rman新生成的。
操作系统级别未发现有硬件的故障日志,另外一个实例xxxdb2运行正常。
请教一下,因为这个控制文件是刚刚备份后添加以及故障后实例可以正常启动,感觉集群的控制文件本身是没有问题的,应该怎么解决此问题?
Alter Log:
---------------------------------------------------------
Fri Nov 22 19:58:33 2019
Hex dump of (file 0, block 4) in trace file /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/trace/xxxdb1_lmon_79295.trc
Corrupt block relative dba: 0x00000004 (file 0, block 4)
Bad check value found during control file block read
Data in bad block:
type: 21 format: 2 rdba: 0x00000004
last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x00001501
check value in block header: 0x3b08
computed block checksum: 0x1
Control file block read for fixed table access has failed due to concurrent controlfile updates.
This is not a real data corruption. Please retry the query.
Errors in file /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/trace/xxxdb1_lmon_79295.trc:
ORA-00202: control file: '+CTL/xxxdb/controlfile/current.257.1024919441'
Errors in file /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/trace/xxxdb1_lmon_79295.trc (incident=69689):
ORA-00227: corrupt block detected in control file: (block 4, # blocks 1)
ORA-00202: control file: '+CTL/xxxdb/controlfile/current.257.1024919441'
Incident details in: /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/incident/incdir_69689/xxxdb1_lmon_79295_i69689.trc
Errors in file /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/trace/xxxdb1_lmon_79295.trc:
ORA-00227: corrupt block detected in control file: (block 4, # blocks 1)
ORA-00202: control file: '+CTL/xxxdb/controlfile/current.257.1024919441'
LMON (ospid: 79295): terminating the instance due to error 227
xxxdb1_lmon_79295.trc:
-----------------------------------
***************** End of Resource Limits Dump ******************
----- Java Stack -----
----- VKTM Time Drifts Circular Buffer -----
Incident 69689 created, dump file: /u01/app/oracle/diag/rdbms/xxxdb/xxxdb1/incident/incdir_69689/xxxdb1_lmon_79295_i69689.trc
ORA-00227: corrupt block detected in control file: (block 4, # blocks 1)
ORA-00202: control file: '+CTL/xxxdb/controlfile/current.257.1024919441'
LMON caught an error 227 in the main loop
error 227 detected in background process
ORA-00227: corrupt block detected in control file: (block 4, # blocks 1)
ORA-00202: control file: '+CTL/xxxdb/controlfile/current.257.1024919441'
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465
----- End of Abridged Call Stack Trace -----
*** 2019-11-22 19:58:35.165
LMON (ospid: 79295): terminating the instance due to error 227
ksuitm: waiting up to [5] seconds before killing DIAG(79285)
srvctl命令成功恢复实例:
----------------------------------------------------
srvctl start instance -d xxxdb -i xxxdb1
备份ASM上控制文件到本地磁盘后,DBV检测备份文件无坏块:
-----------------------------------------------------------------------------
ASMCMD> cp current.257.1024919441 /home/grid/controlfile_20191123_bak.ctl
copying +CTL/xxxdb/controlfile/current.257.1024919441 -> /home/grid/controlfile_20191123_bak.ctl
ASMCMD> exit
[grid@rac2 ~]$ dbfsize controlfile_20191123_bak.ctl
Database file: controlfile_20191123_bak.ctl
Database file type: file system
Database file size: 2384 16384 byte blocks
[grid@rac2 ~]$ dbv file=/home/grid/controlfile_20191123_bak.ctl blocksize=16384
DBVERIFY: Release 11.2.0.4.0 - Production on Sat Nov 23 16:19:12 2019
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /home/grid/controlfile_20191123_bak.ctl
DBVERIFY - Verification complete
Total Pages Examined : 2384
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1365
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 1019
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 1154908 (65535.1154908)