【kingsql分享】RAC节点故障修复一例

最新推荐文章于 2023-09-06 17:26:31 发布
最新推荐文章于 2023-09-06 17:26:31 发布
阅读量254 收藏
点赞数
文章标签： 数据库 运维 操作系统
很久之前安装的虚拟机，今天开机之后，rac2和rac3集群服务无法启动
[grid@rac2 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
[grid@rac3 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac1 ~]$ srvctl status database -d kingsql
Instance kingsql1 is running on node rac1
Instance kingsql2 is not running on node rac2
Instance kingsql3 is not running on node rac3


rac2的cssd日志发现问题
2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 356447081, wrtcnt, 294277, LATS 4294697710, lastSeqNo 294274, uniqueness 1461034467, timestamp 1461056437/21174804
2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 3, rac3, has a disk HB, but no network HB, DHB has rcfg 356447080, wrtcnt, 41563, LATS 4294697710, lastSeqNo 41560, uniqueness 1463643843, timestamp 1463643914/18290394

参考文章
http://www.itpub.net/thread-1766984-1-1.html

[grid@rac1 ~]$ crs_stop -all
CRS-2500: Cannot stop resource 'ora.gsd' as it is not running
Attempting to stop `ora.oc4j` on member `rac1`
Attempting to stop `ora.CRS.dg` on member `rac1`
Attempting to stop `ora.DATA.dg` on member `rac1`
Attempting to stop `ora.kingsql.db` on member `rac1`
Attempting to stop `ora.ons` on member `rac1`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac1'
Stop of `ora.ons` on member `rac1` succeeded.
Attempting to stop `ora.cvu` on member `rac1`
Attempting to stop `ora.rac2.vip` on member `rac1`
Attempting to stop `ora.rac3.vip` on member `rac1`
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.rac3.vip` on member `rac1` succeeded.
Stop of `ora.rac2.vip` on member `rac1` succeeded.
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Stop of `ora.oc4j` on member `rac1` succeeded.
Stop of `ora.cvu` on member `rac1` succeeded.
Attempting to stop `ora.net1.network` on member `rac1`
Stop of `ora.net1.network` on member `rac1` succeeded.
Stop of `ora.kingsql.db` on member `rac1` succeeded.
Stop of `ora.DATA.dg` on member `rac1` succeeded.
Stop of `ora.CRS.dg` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
Stop of `ora.asm` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
--卡在这不动了，于是执行crsctl stop crs
CRS-0184: Cannot communicate with the CRS daemon.
[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac3'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'rac3'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac3'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac3' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac3'
CRS-2677: Stop of 'ora.asm' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac3'
CRS-2677: Stop of 'ora.ons' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac3'
CRS-2677: Stop of 'ora.net1.network' on 'rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac3'
CRS-2673: Attempting to stop 'ora.asm' on 'rac3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac3'
CRS-2677: Stop of 'ora.evmd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac3'
CRS-2677: Stop of 'ora.cssd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac3'
CRS-2677: Stop of 'ora.crf' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac3'
CRS-2677: Stop of 'ora.gipcd' on 'rac3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac3'
CRS-2677: Stop of 'ora.gpnpd' on 'rac3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.

启动
[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
--还是有问题
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[grid@rac1 ~]$ crs_start -all
[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3        
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM3.asm application    ONLINE    ONLINE    rac3        
ora....C3.lsnr application    ONLINE    ONLINE    rac3        
ora.rac3.gsd   application    OFFLINE   OFFLINE               
ora.rac3.ons   application    ONLINE    ONLINE    rac3        
ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3   

重启rac2操作系统
[root@rac2 ~]# reboot
[root@rac2 ~]#
Broadcast message from root@rac2
        (/dev/pts/0) at 17:40 ...

The system is going down for reboot NOW!

[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3        
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....SM3.asm application    ONLINE    ONLINE    rac3        
ora....C3.lsnr application    ONLINE    ONLINE    rac3        
ora.rac3.gsd   application    OFFLINE   OFFLINE               
ora.rac3.ons   application    ONLINE    ONLINE    rac3        
ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3     

[grid@rac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.DATA.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.asm
               ONLINE  ONLINE       rac1                     Started             
               ONLINE  ONLINE       rac2                     Started             
               ONLINE  ONLINE       rac3                     Started             
ora.gsd
               OFFLINE OFFLINE      rac1                                         
               OFFLINE OFFLINE      rac2                                         
               OFFLINE OFFLINE      rac3                                         
ora.net1.network
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
ora.ons
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
               ONLINE  ONLINE       rac3                                         
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac3                                         
ora.cvu
      1        ONLINE  ONLINE       rac3                                         
ora.kingsql.db
      1        ONLINE  ONLINE       rac3                     Open                
      2        ONLINE  ONLINE       rac2                     Open                
      3        ONLINE  ONLINE       rac1                     Open                
ora.oc4j
      1        ONLINE  ONLINE       rac3                                         
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                         
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                         
ora.rac3.vip
      1        ONLINE  ONLINE       rac3                                         
ora.scan1.vip
      1        ONLINE  ONLINE       rac3                                        

结论：当
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
和
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
还是无法重启的时候，最后的大招就是reboot


kingsql分享
2016年5月31日
转载请注明出处

Oracle Young Expert查询网址
http://www.oracle.com/technetwork/cn/community/user-groups/young-expert-list-2395298-zhs.html

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/28389881/viewspace-2109655/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/28389881/viewspace-2109655/

cuishitan3439
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Oracle 11gr2 rac 修复单节点故障（删除和添加节点）
jisen_huang的专栏
10-31 6313
由于本机硬盘故障导致单个节点的/u01目录丢失。导致单节点的rac
RAC坏节点重建
09-12
Oracle集群，rac节点损坏修复，数据库工程师，操作步骤
RAC坏节点重建与卸载
01-21
Oracle ocr损坏修复，集群卸载，重装，OCR损坏修复问题
oracle rac单节点恢复,Oracle_RAC恢复一个节点总结
weixin_42131443的博客
04-11 397
Oracle_RAC恢复一个节点总结Rac1已坏Rac3正常先在rac3上把rac1的信息删干净,然后重新填加rac1步骤如下:1,在rac１上运行DBCA,删除instance;2,如果有ASM,删除ASM实例,srvctl stop asm -n rac1srvctl remove asm -n rac1;3,在rac3上执行updateNodeList脚本/u01/oracle/oracle...
rac10g 还原损坏节点
q195136130的专栏
06-22 294
场景介绍 系统环境 oel 5.6 Oracle环境 10.2.0.1 磁盘绑定 裸设备+asmlib Asm 使用 场景：2节点本地磁盘损坏，系统与oracle软件均丢失。实验使用虚拟机快照，恢复到二节点配置前的场合。 主机名 oel1,oel2 基础环境复原 复原hosts vi /etc/hosts 192.168.7.221 oel1 192.168.7.222 oel2 192.168.7.2...
RAC节点宕机故障分析
06-27
RAC 节点宕机故障分析是一个复杂的过程，需要对节点宕机故障的原因、事件、ORA-600 错误、Bug 和 LMS 进程进行分析和确认。通过对这些方面的分析，可以获取节点宕机故障的相关信息，诊断和解决 RAC 节点宕机故障。
Oracle RAC故障节点删除步骤
01-12
一、RAC 故障节点删除步骤 在 RAC-1 节点宕机，RAC-2 节点正常的情况下，需要在 RAC-2 节点上删除 RAC-1 节点信息。删除步骤如下： 1. 查看 RAC-2 节点数据库状态，确保一切正常。 2. 在 RAC-2 节点上，检查 ...
Oracle RAC 新增节点 操作步骤
最新发布
09-22
运维Oracle RAC 如果遇到故障，如何删除节点再添加节点，资源不足如何扩容，本篇文章告诉你。
Oracle rac 两节点升级.docx
06-17
在本文中，我们将详细探讨如何将一个拥有两个节点的Oracle RAC 11.2.0.1.0系统升级到11.2.0.4.0版本。Oracle RAC（Real Application Clusters）是一种数据库高可用性解决方案，允许在多台服务器之间共享单个数据库...
rac 添加节点.docx
07-22
详细描述了 rac 如何添加节点，添加节点的详细步骤，安装集群软件，安装oracle软件， 挂载磁盘组，创建数据库
ADG搭建，RAC--＞单节点，搭建过程及故障描述
a651557007的博客
09-06 200
lsnrctl status #状态为unknow 开启成功，静态监听的作用是在duplicate的时候备库状态为nomount依然可以连接。sqlplus sys/password@ecologydg as sysdba #测试主库至备库的连通性。登录rman ,开始duplicate，主库备库都可以执行。备库切换主库下次再写。
如何Oracle_RAC恢复一个节点总结
热门推荐
badman250的专栏
07-05 3万+
如何Oracle_RAC恢复一个节点总结 作者：51cto出处：博客2013-08-01 13:39 　　Rac1 已坏 　　Rac3 正常 　　先在rac3上把rac1的信息删干净,然后重新填加rac1 　　步骤如下: 　　1,在rac1上运行DBCA,删除instance; 　　2,如果有ASM,删除ASM实例, 　　srVCtl
记录解决一次Listener状态为Not All Endpoints Registered的故障
weixin_33809981的博客
01-12 2964
检查发现监听异常C:\Users\Administrator>crsctl status res -t NAME TARGET STATE SERVER STATE_DETAILS Local Resources ora.DATA.dgONLINE ONLINE rac2ora.LISTENER.lsnrO...
ORA-27300,ORA-27301,ORA-27302,ORA-27303导致实例宕的问题分析
wzw_dba的博客
06-29 9856
4.29上午，生产环境数据库用户无法登陆，提示归档日志错误，只能以超级管理员登陆，但是关于日志的相关操作都hang住，由于时间紧迫，未做hanganzlyze操作，正常停库过程也hang，只能以shutdown abort停掉实例，然后在重新启库过程中正常，未遇到错误提示，数据库正常打开，操作暂时恢复，持续有10分钟左右，中间日志切换无反应，用户登录又hang，于是再做shutdown abort
模拟RAC两个节点内联网不通
wonder_2012的专栏
07-16 3734
环境：RHEL5.8 RAC 11.2.0.3.0 1：2节点集群是正常的 [grid@rac1 ~]$ crs_stat -t Name           Type           Target    State     Host         ------------------------------------------------------------ ora.
记录一些RAC下CRS错误，备用。
weixin_34013044的博客
02-05 168
[oracle@racdb6 racdb6]$ tail -200 alertracdb6.log 2010-02-03 23:45:34.772[cssd(7582)]CRS-1610:node racdb5 (1) at 90% heartbeat fatal, eviction in 4.134 seconds2010-02-03 23:45:35.774[cssd(7...
ORACLE RAC中一个实例不能随crs自动启动的解决
还不算晕的专栏
09-16 3507
现象：在两个节点上做CRS的重启，这个实例都不能随CRS的启动而启动。CRS启动后做crs_start -all可以把没启动的资源起来，而且无报错。 分析：去crsd.log中找原因，发现CRS根本就没有去尝试启动该instance以及相关的HA资源。用crs_stat -p把资源详细信息输出后发现该实例的所有资源的auto_start=2，意味着该资源不会自动启动 解决：更新
记一次Oracle RAC一节点重启后出现故障的处理
范一刀的博客
03-08 5892
因为存储的相关操作，客户需要手动重启rac节点，然而，这个重启导致了接下来的事故。。。。 由于是远程跟我沟通，我回复rac环境下可以重启一个节点，客户就自信重启了，出现的故障如下所示： [grid@hxdb01 ~]$ srvctl start nodeapps -n hxdb01 PRKH-1010 : 无法与 CRS 服务通信。 PRKH-3003 : 尝试与 CSS 守护程序通...
一次关于scan IP 的故障
瀚宇的技术之路
02-08 7446
环境：HP-UNIX , oracle 11.2.0.3 版本的三 node RAC 问题描述：问题发生前几天公司机房网络升级万兆网络，整个数据库服务器停机。网络升级完成后， RAC环境开启正常。三天后应用的同事反映部分应用不能连接数据库，应用后台报：Listener refused the connection with the following error:ORA-12514,
oracle 12c rac故障节点删除
09-16
Oracle 12c RAC故障节点删除主要包括以下步骤： 1. 检查故障节点：首先，我们需要确认故障节点是否真的无法恢复。可以使用集群管理工具(如CRSCTL或SRVCTL)来检查节点状态和资源的可用性。 2. 卸载软件：如果节点...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值