rac集群由于ora.cluster_interconnect.haip资源无法启动，导致集群启动失败的两种解决方法

老苏畅谈运维

已于 2024-07-18 13:24:27 修改

阅读量316

点赞数 5

分类专栏： oracle rac 文章标签： oracle 运维 rac

于 2024-07-18 13:20:59 首次发布

文章来源于老苏畅谈运维，欢迎您转载，转载声明来源老苏畅谈运维

本文链接：https://blog.csdn.net/zhirongsu/article/details/140520538

版权

oracle 同时被 2 个专栏收录

38 篇文章 2 订阅

订阅专栏

rac

1 篇文章 0 订阅

订阅专栏

作者介绍：老苏，10余年DBA工作运维经验，擅长Oracle、MySQL、PG、Mongodb数据库运维（如安装迁移，性能优化、故障应急处理等）
公众号：老苏畅谈运维
欢迎关注本人公众号，更多精彩与您分享。

生产一套Oracle 11.2.0.4 的 RAC DG数据库，由于心跳网络的调整需要，原本能正常运行的，发现网络调整后，集群服务启动不正常。启动过程报错如下：

2024-07-16 13:21:34.094:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(13408)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/agent/ohasd/orarootagent_root//orarootagent_root.log.
2024-07-16 13:21:38.097:
[ohasd(13172)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/ohasd/ohasd.log.
2024-07-16 13:22:38.117:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(13408)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/agent/ohasd/orarootagent_root//orarootagent_root.log.
2024-07-16 13:22:42.120:
[ohasd(13172)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/ohasd/ohasd.log.
2024-07-16 13:23:42.133:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(13408)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/agent/ohasd/orarootagent_root//orarootagent_root.log.
2024-07-16 13:23:46.136:
[ohasd(13172)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/ohasd/ohasd.log.
2024-07-16 13:24:46.149:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(13408)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/agent/ohasd/orarootagent_root//orarootagent_root.log.
2024-07-16 13:24:50.152:
[ohasd(13172)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/sbhis1/ohasd/ohasd.log.
2024-07-16 13:24:50.158:
[ohasd(13172)]CRS-2807:Resource 'ora.asm' failed to start automatically.

从上面的具体报错，可以看到ora.cluster_interconnect.haip这个资源启动失败，导致集群启动失败，查看/u01/app/11.2.0/grid/log/sbhis1/agent/ohasd/orarootagent_root//orarootagent_root.log 这个日志，查找ora.cluster_interconnect.haip启动失败原因，从该日志可以看到以下报错：

2024-07-16 13:20:35.030: [ USRTHRD][1570182912]{0:0:2} PROBE:  got conflicting source ip 169.254.98.99, addr fa-16-3e-8e-ec-6e
2024-07-16 13:20:35.030: [ USRTHRD][1570182912]{0:0:2} PROBE:  conflict detected src { 169.254.98.99, fa-16-3e-8e-ec-6e }, target { 0.0.0.0, 08-3a-88-cd-66-34 }
2024-07-16 13:20:35.092: [ora.ctssd][1585514240]{0:0:2} [start] with returnbuf
[  clsdmc][1585514240]CLSDMC.C returnbuflen=8, extraDataBuf=40, returnbuf=440A21A0
2024-07-16 13:20:35.094: [ora.ctssd][1585514240]{0:0:2} [start] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7fa5440a21a0], buflen=8
2024-07-16 13:20:35.094: [ora.ctssd][1585514240]{0:0:2} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/11.2.0/grid
2024-07-16 13:20:35.094: [ora.ctssd][1585514240]{0:0:2} [start] Utils::getOracleHomeAttrib oracle_home:/u01/app/11.2.0/grid
2024-07-16 13:20:35.095: [ora.ctssd][1585514240]{0:0:2} [start] PID 13890 from /u01/app/11.2.0/grid/ctss/init/sbhis1.pid
2024-07-16 13:20:35.095: [ora.ctssd][1585514240]{0:0:2} [start] }DaemonAgent::start
2024-07-16 13:20:35.095: [ora.ctssd][1585514240]{0:0:2} [start] translateReturnCodes, return = 0, state detail = Checkcb data [0x7fa5440a21a0]: mode[0x40] offset[0 ms].
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2} Failed to check 169.254.111.70 on bond2
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2} (null) category: 0, operation: , loc: , OS error: 0, other:
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2} Starting Probe for ip 169.254.111.70
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2} Transitioning to Probe State
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2}  Arp::sProbe {
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2} Arp::sSend:  sending type 1
2024-07-16 13:20:35.530: [ USRTHRD][1570182912]{0:0:2}  Arp::sProbe }
2024-07-16 13:20:35.531: [ USRTHRD][1570182912]{0:0:2} PROBE:  got conflicting source ip 169.254.111.70, addr fa-16-3e-39-76-01
2024-07-16 13:20:35.531: [ USRTHRD][1570182912]{0:0:2} PROBE:  conflict detected src { 169.254.111.70, fa-16-3e-39-76-01 }, target { 0.0.0.0, 08-3a-88-cd-66-34 }
2024-07-16 13:20:36.031: [ USRTHRD][1570182912]{0:0:2} Failed to check 169.254.205.245 on bond2
2024-07-16 13:20:36.031: [ USRTHRD][1570182912]{0:0:2} (null) category: 0, operation: , loc: , OS error: 0, other:
2024-07-16 13:20:36.031: [ USRTHRD][1570182912]{0:0:2} Starting Probe for ip 169.254.205.245
2024-07-16 13:20:36.031: [ USRTHRD][1570182912]{0:0:2} Transitioning to Probe State
[  clsdmc][1585514240]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=440A2520
2024-07-16 13:20:36.096: [ora.ctssd][1585514240]{0:0:2} [start] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7fa5440a2520], buflen=8
2024-07-16 13:20:36.096: [ora.ctssd][1585514240]{0:0:2} [start] Start: Extended check return buffer: "Ì" with length of 8

从上面的关键字报错“PROBE: got conflicting source”，可以从MOS上找到相关的解决方案：

HAIP fails to start if default gateway is configured for VLAN for private network on network switch
Issue: HAIP fails to start if default gateway is configured for VLAN for private network on network switch

orarootagent_root.log shows: PROBE: conflict detected src { 169.254.12.247, <gateway MAC on switch> }, target { 0.0.0.0, <private NIC MAC> }

The solution is to remove default gateway setting on network switch for private network (VLAN), refer to Note 1366211.1 for more details.

MOS 1366211.1文档有说明报错原因是该心跳网络所在的VLAN配置了网关，HAIP无法获取169.254.x.x 这个段的IP，也就导致ora.cluster_interconnect.haip启动失败。

1、解决方法一：调整网络

知道了原因，解决起来也就简单了，让网络部把这个RAC的心跳单独弄到一个vlan上，并把该vlan的网关设置去掉，这样调整后，就顺利启动集群。

2、解决方法二：禁用HAIP

上面是一种解决方式，有没有另外的解决方法呢，答案是肯定的，把HAIP这个特性禁用了，是不是就可以了。HAIP这个特性是从11.0.0.2版本开始，Oracle提供的私网的高可用性和负载均衡，需要有多块网卡来做配置。如果你在操作系统层面已经做了网卡绑定配置，如linux bonding ，那其实是可以禁用的。要该如何禁用这个HAIP呢？

（1）禁用HAIP资源

--使用root用户执行以下命令。

--停止所有节点的CRS

/u01/app/11.2.0/grid/bincrsctl stop crs

--依次在每个节点中执行以下命令（节点1执行完毕后再在节点2执行）

# /u01/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
# /u01/app/11.2.0/grid/bin/crsctl stop res ora.asm -init
# /u01/app/11.2.0/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init

(2)修改ASM资源的依赖关系
```language
--查看当前ASM资源的关联关系：
# /u01/app/11.2.0/grid/bin/crsctl stat res ora.asm -p -init
NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:grid:rw-,pgrp:oinstall:rw-,other::r--,user:grid:rwx
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_ARGS=
CHECK_COMMAND=
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLEAN_ARGS=
CLEAN_COMMAND=
DAEMON_LOGGING_LEVELS=
DAEMON_TRACING_LEVELS=
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION="ASM instance"
DETACHED=true
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=3
FAILURE_THRESHOLD=5
GEN_USR_ORA_INST_NAME=+ASM2
HOSTING_MEMBERS=
LOAD=1
LOGGING_LEVEL=1
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
ORA_VERSION=11.2.0.4.0
PID_FILE=
PLACEMENT=balanced
PROCESS_TO_MONITOR=
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
SERVER_POOLS=
SPFILE=
START_ARGS=
START_COMMAND=
START_DEPENDENCIES=hard(ora.cssd,ora.cluster_interconnect.haip,ora.ctssd)pullup(ora.cssd,ora.cluster_interconnect.haip,ora.ctssd)weak(ora.drivers.acfs)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_ARGS=
STOP_COMMAND=
STOP_DEPENDENCIES=hard(intermediate:ora.cssd,shutdown:ora.cluster_interconnect.haip)
STOP_TIMEOUT=600
UNRESPONSIVE_TIMEOUT=180
UPTIME_THRESHOLD=1h
USR_ORA_ENV=
USR_ORA_INST_NAME=
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.3.0
##可以看到ASM资源和HAIP资源的依赖关系。


--修改ASM的关联关系（所有节点执行）
# /u01/app/11.2.0/grid/bin/crsctl modify resource ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.drivers.acfs)'" -f -init 
# /u01/app/11.2.0/grid/bin/crsctl modify resource ora.asm -attr "STOP_DEPENDENCIES=hard(intermediate:ora.cssd)" -f –init

在这里插入图片描述

（3）修改cluster_interconnects参数

--修改ASM实例的cluster_interconnects参数
--grid用户执行，修改具体的私网地址
SQL> alter system set cluster_interconnects='10.10.10.11' scope=spfile sid='+ASM1'; 
SQL> alter system set cluster_interconnects='10.10.10.12' scope=spfile sid='+ASM2';


--修改DB实例的cluster_interconnects参数
--oracle用户执行，修改具体的私网地址
SQL> alter system set cluster_interconnects='10.10.10.11' scope=spfile sid='sbhis1'; 
SQL> alter system set cluster_interconnects='10.10.10.12' scope=spfile sid='sbhis2';

(4)重启集群

# /u01/app/11.2.0/grid/bin/crsctl stop crs -f
# /u01/app/11.2.0/grid/bin/crsctl start crs

--重启完，检查HAIP是否禁用
$ crsctl stat res -t -init

若ora.cluster_interconnect.haip为offline则为禁用状态。然后执行如下命令：

ifconfig -a 或ip a |grep 169.254

查看是否还有169.254开头的地址，如果没有了，那么说明已经禁用成功。

关注我，学习更多的数据库知识
请添加图片描述

老苏畅谈运维

关注

5
点赞
踩
5

收藏

觉得还不错? 一键收藏
打赏
0
评论
rac集群由于ora.cluster_interconnect.haip资源无法启动，导致集群启动失败的两种解决方法

生产一套Oracle 11.2.0.4 的 RAC DG数据库，由于心跳网络的调整需要，原本能正常运行的，发现网络调整后，集群服务启动不正常。MOS 1366211.1文档有说明报错原因是该心跳网络所在的VLAN配置了网关，HAIP无法获取169.254.x.x 这个段的IP，也就导致ora.cluster_interconnect.haip启动失败。
复制链接

扫一扫