【kingsql分享】RAC节点故障修复一例

很久之前安装的虚拟机,今天开机之后,rac2和rac3集群服务无法启动
[grid@rac2 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
[grid@rac3 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac1 ~]$ srvctl status database -d kingsql
Instance kingsql1 is running on node rac1
Instance kingsql2 is not running on node rac2
Instance kingsql3 is not running on node rac3


rac2的cssd日志发现问题
2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 356447081, wrtcnt, 294277, LATS 4294697710, lastSeqNo 294274, uniqueness 1461034467, timestamp 1461056437/21174804
2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 3, rac3, has a disk HB, but no network HB, DHB has rcfg 356447080, wrtcnt, 41563, LATS 4294697710, lastSeqNo 41560, uniqueness 1463643843, timestamp 1463643914/18290394

参考文章
http://www.itpub.net/thread-1766984-1-1.html

[grid@rac1 ~]$ crs_stop -all
CRS-2500: Cannot stop resource 'ora.gsd' as it is not running
Attempting to stop `ora.oc4j` on member `rac1`
Attempting to stop `ora.CRS.dg` on member `rac1`
Attempting to stop `ora.DATA.dg` on member `rac1`
Attempting to stop `ora.kingsql.db` on member `rac1`
Attempting to stop `ora.ons` on member `rac1`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac1'
Stop of `ora.ons` on member `rac1` succeeded.
Attempting to stop `ora.cvu` on member `rac1`
Attempting to stop `ora.rac2.vip` on member `rac1`
Attempting to stop `ora.rac3.vip` on member `rac1`
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.rac3.vip` on member `rac1` succeeded.
Stop of `ora.rac2.vip` on member `rac1` succeeded.
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Stop of `ora.oc4j` on member `rac1` succeeded.
Stop of `ora.cvu` on member `rac1` succeeded.
Attempting to stop `ora.net1.network` on member `rac1`
Stop of `ora.net1.network` on member `rac1` succeeded.
Stop of `ora.kingsql.db` on member `rac1` succeeded.
Stop of `ora.DATA.dg` on member `rac1` succeeded.
Stop of `ora.CRS.dg` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
Stop of `ora.asm` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
--卡在这不动了,于是执行crsctl stop crs

CRS-0184: Cannot communicate with the CRS daemon.
[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac3'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'rac3'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac3'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac3' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac3'
CRS-2677: Stop of 'ora.asm' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac3'
CRS-2677: Stop of 'ora.ons' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac3'
CRS-2677: Stop of 'ora.net1.network' on 'rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac3'
CRS-2673: Attempting to stop 'ora.asm' on 'rac3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac3'
CRS-2677: Stop of 'ora.evmd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac3'
CRS-2677: Stop of 'ora.cssd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac3'
CRS-2677: Stop of 'ora.crf' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac3'
CRS-2677: Stop of 'ora.gipcd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac3'
CRS-2677: Stop of 'ora.gpnpd' on 'rac3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.

启动
[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
--还是有问题
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[grid@rac1 ~]$ crs_start -all
[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3        
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM3.asm application    ONLINE    ONLINE    rac3        
ora....C3.lsnr application    ONLINE    ONLINE    rac3        
ora.rac3.gsd   application    OFFLINE   OFFLINE               
ora.rac3.ons   application    ONLINE    ONLINE    rac3        
ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3   

重启rac2操作系统
[root@rac2 ~]# reboot
[root@rac2 ~]#
Broadcast message from root@rac2
        (/dev/pts/0) at 17:40 ...

The system is going down for reboot NOW!

[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3        
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....SM3.asm application    ONLINE    ONLINE    rac3        
ora....C3.lsnr application    ONLINE    ONLINE    rac3        
ora.rac3.gsd   application    OFFLINE   OFFLINE               
ora.rac3.ons   application    ONLINE    ONLINE    rac3        
ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3     

[grid@rac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.DATA.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.asm
               ONLINE  ONLINE       rac1                     Started             
               ONLINE  ONLINE       rac2                     Started             
               ONLINE  ONLINE       rac3                     Started             
ora.gsd
               OFFLINE OFFLINE      rac1                                         
               OFFLINE OFFLINE      rac2                                         
               OFFLINE OFFLINE      rac3                                         
ora.net1.network
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.ons
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac3                                         
ora.cvu
      1        ONLINE  ONLINE       rac3                                         
ora.kingsql.db
      1        ONLINE  ONLINE       rac3                     Open                
      2        ONLINE  ONLINE       rac2                     Open                
      3        ONLINE  ONLINE       rac1                     Open                
ora.oc4j
      1        ONLINE  ONLINE       rac3                                         
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                         
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                         
ora.rac3.vip
      1        ONLINE  ONLINE       rac3                                         
ora.scan1.vip
      1        ONLINE  ONLINE       rac3                                        

结论:当
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs

[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
还是无法重启的时候,最后的大招就是reboot


kingsql分享
2016年5月31日
转载请注明出处

Oracle Young Expert查询网址
http://www.oracle.com/technetwork/cn/community/user-groups/young-expert-list-2395298-zhs.html

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/28389881/viewspace-2109655/,如需转载,请注明出处,否则将追究法律责任。

转载于:http://blog.itpub.net/28389881/viewspace-2109655/

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值