oracle 11g RAC ASM磁盘被强制下线抢修一例

本文链接：https://blog.csdn.net/luoyeyuan/article/details/53243654

又有一段时间没写有关oracle的文章了，恐怕这也是不能成为高手的原因之一，偶有典型性拖延症和懒癌。
今天要讲的一个栗子又一个现场case，中午吃饭的时候看到来自同事的未接，处于职业敏感感觉是数据中心出问题了，拨过去第一通电话告诉我存储坏掉一块盘，然后NBU有个备份作业没完成，感觉问题不大通知厂商维修便可以了。刚吃完饭，第二通电话进来了，这次告知我又一个双活控制器存储报警，同时说数据库RAC集群看不到状态了。心中暗暗祈祷不要被近期“比特币勒索”事件波及，上周二刚刚通知各位厂家技术人员排查过，可别害我啊，这玩意儿我真修复不了。
原本打算中午休息，忍了赶快处理吧，下午还有3节c#的实验课呢，周末不太平啊。赶到现场，如我猜想的一样，同事赶着中午在巡检，询问最近的巡检情况，同事告知我周六没有做巡检，周一到周五都做了，没发现问题。打开终端连到db server上，果然是有问题：

[12:59:29][root: ~]#ckrac   自己写个一个小脚本
[12:59:31]CRS-4535: Cannot communicate with Cluster Ready Services
[12:59:31]CRS-4000: Command Status failed, or completed with errors.

首先第一印象判断：集群垮了
马上检查数据库

su - oracle
sqlplus / as sysdba
select * from v$instance;

还好，实例还活着，数据算是基本上不会丢了。
尝试启动集群

13:07:48][root: /u01/app/11.2.0/grid/bin]#./crsctl start cluster
[13:07:48]CRS-2672: Attempting to start 'ora.crsd' on 'rac01'
[13:07:49]CRS-2676: Start of 'ora.crsd' on 'rac01' succeeded

接下就是检查日志了，先检查节点实例的日志，没看出问题

[13:13:33][root: /u01/app/11.2.0/grid/bin]# cd /u01/app/oracle/diag/rdbms/gbf1/GBF11/trace
[13:13:52][root: /u01/app/oracle/diag/rdbms/gbf1/GBF11/trace]#tail -500  alert_GBF11.log |more

有点懵了，确定是集群的问题，这会是什么问题呢？
查看集群日志：没做功课和笔记下场，只有find一下了

[13:19:33][root: ~]#find /u01 -name 'crsd.log'
[13:19:45]/u01/app/11.2.0/grid/log/rac01/crsd/crsd.log

[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: COMMCRS  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: COMMNS  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: CSSCLNT  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCLIB  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCXBAD  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCLXPT  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCUNDE  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPC  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCGEN  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCTRAC  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCWAIT  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCXCPT  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCOSD  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCBASE  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCCLSA  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCCLSC  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCEXMP  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCGMOD  0
[13:20:42]2016-11-20 13:23:50.303: [    CRSD][3134007024] Logging level for Module: GIPCHEAD  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCMUX  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCNET  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCNULL  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCPKT  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCSMEM  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHAUP  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHALO  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHTHR  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHGEN  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHLCK  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHDEM  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: GIPCHWRK  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSMAIN  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: clsdmt  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: clsdms  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSUI  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSCOMM  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSRTI  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSPLACE  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSAPP  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSRES  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSTIMER  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSEVT  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSD  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CLUCLS  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CLSVER  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CLSFRAME  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSPE  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSSE  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSRPT  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSOCR  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: UiServer  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: AGFW  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: SuiteTes  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSSHARE  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSSEC  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSCCL  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: CRSCEVT  0
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: AGENT  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRAPI  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRCLI  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRSRV  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRMAS  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRMSG  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRCAC  1
[13:20:42]2016-11-20 13:23:50.304: [    CRSD][3134007024] Logging level for Module: OCRRAW  1
[13:20:42]2016-11-20 13:23:50.305: [    CRSD][3134007024] Logging level for Module: OCRUTL  1
[13:20:42]2016-11-20 13:23:50.305: [    CRSD][3134007024] Logging level for Module: OCROSD  1
[13:20:42]2016-11-20 13:23:50.305: [    CRSD][3134007024] Logging level for Module: OCRASM  1
[13:20:42]2016-11-20 13:23:50.305: [ CRSMAIN][3134007024] Checking the OCR device
[13:20:42]2016-11-20 13:23:50.305: [ CRSMAIN][3134007024] Sync-up with OCR
[13:20:42]2016-11-20 13:23:50.305: [ CRSMAIN][3134007024] Connecting to the CSS Daemon
[13:20:42]2016-11-20 13:23:50.305: [ CRSMAIN][3134007024] Getting local node number
[13:20:42]2016-11-20 13:23:50.306: [ CRSMAIN][3134007024] Initializing OCR
[13:20:42]2016-11-20 13:23:50.307: [ CRSMAIN][3127560512] Policy Engine is not initialized yet!
[13:20:42][   CLWAL][3134007024]clsw_Initialize: OLR initlevel [70000]
[13:20:42]2016-11-20 13:23:50.622: [  OCRASM]
****[3134007024]proprasmo: Error in open/create file in dg [OCRVDISK]**** 
[13:20:42][  OCRASM][3134007024]SLOS : SLOS: cat=8, opn=kgfoOpen01, dep=15056, loc=kgfokge
[13:20:42]
[13:20:42]2016-11-20 13:23:50.623: [  OCRASM][3134007024]ASM Error Stack : 
[13:20:42]2016-11-20 13:23:50.667: [  OCRASM][3134007024]proprasmo: kgfoCheckMount returned [6]
[13:20:42]2016-11-20 13:23:50.667: [  OCRASM][3134007024]proprasmo: The ASM disk group OCRVDISK is not found or not mounted
[13:20:42]2016-11-20 13:23:50.667: [  OCRRAW][3134007024]proprioo: Failed to open [+OCRVDISK]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
[13:20:42]2016-11-20 13:23:50.667: [  OCRRAW][3134007024]proprioo: No OCR/OLR devices are usable
[13:20:42]2016-11-20 13:23:50.667: [  OCRASM][3134007024]proprasmcl: asmhandle is NULL
[13:20:42]2016-11-20 13:23:50.668: [    GIPC][3134007024] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5343]
[13:20:42]2016-11-20 13:23:50.668: [ default][3134007024]clsvactversion:4: Retrieving Active Version from local storage.
[13:20:42]2016-11-20 13:23:50.670: [ CSSCLNT][3134007024]clssgsgrppubdata: group (ocr_rac-cluster01) not found
[13:20:42]
[13:20:42]2016-11-20 13:23:50.670: [  OCRRAW][3134007024]proprio_repairconf: Failed to retrieve the group public data. CSS ret code [20]
[13:20:42]2016-11-20 13:23:50.672: [  OCRRAW][