the work directory /tmp/ oracle,Oracle 11.2.0.3 RAC ON Linux 6.4 无法启动 CRS-4124: Oracle High Availabi...

背景：半夜临时需求，虚拟平台升级，涉及该数据库集群，主机重启后，集群无法启动。

报错信息：CRS-4124: Oracle High Availability Services startup failed。

集群日志无任何信息

2021-03-26 01:51:36.668:

[ohasd(3279)]CRS-0715:Oracle High Availability Service has timed out waiting for init.ohasd to be started.

OHAS几乎也没有信息

Changing directory to /oracle/11g/grid/11.2.0.3/log/soadbjx2/ohasd

OHASD starting

OHASD stderr redirected to ohasdOUT.log

正常信息应该是这样的：

2020-06-05 22:25:31

Changing directory to /oracle/11g/grid/11.2.0.3/log/soadbjx2/ohasd

OHASD starting

Handshake with the watchdog’s been completed <<<<<< 会出现如下信息

OHASD stderr redirected to ohasdOUT.log

尝试过的操作：

1、关闭防火墙

2、删除socket文件

cd /var/tmp/

rm -rf .oracle

3、重置权限

./rootcrs.pl -unlock

./rootcrs.pl -patch

以上做法均无效,差点想要重建集群，索性跟踪看看

跟踪启动日志

strace -f -t -o /tmp/crschk.trc crsctl start crs

日志中循环打印

open("/var/tmp/.oracle/npohasd", O_WRONLY|O_NONBLOCK) = -1 ENXIO (No such device or address)

读不到socket的文件

命中

Bug 18330979 - Improved diagnostics for when ohasd / init.ohasd handshake failures (Doc ID 18330979.8)

Description

This is a diagnostic enhancement intended to aid in diagnosing a particular class of problem that is sometimes (rarely)

encountered in the Oracle Clusterware start up sequence. Specifically, the required synch up/handshake that

must successfully complete between the ohasd daemon and the init.ohasd watchdog script.

This bug fix adds code to collect diagnostics that are useful towards diagnosing these problems.

Rediscovery Notes

Oracle Clusterware stack stalls at ohasd start up. An strace of ohasd finds that it is stalled waiting on

open("/var/tmp/.oracle/npohasd", O_WRONLY),

ie it is waiting for init.ohasd to complete a read on the same named pipe.

Workaround

Manually read from pipe /var/tmp/.oracle/npohasd

通过cat该文件，恢复正常

cat /var/tmp/.oracle/npohasd