PROC-26 CRS Does not Start on 2nd Node But on 1st Node is up (Doc ID 603398.1)-CSDN博客

PROC-26 CRS Does not Start on 2nd Node But on 1st Node is up (Doc ID 603398.1)

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.1.0.7 [Release 10.2 to 11.1]
Information in this document applies to any platform.
***Checked for relevance on 24-Dec-2013***

Symptoms

2 node RAC cluster, CRS is up on node 1, but not up on the 2nd node.

ps -ef | grep css

root 11813 1 0 12:28 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 15228 11813 0 12:28 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 15246 12448 0 12:28 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 15991 13672 0 12:28 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

cat /tmp/crsctl.15228 shows:

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage

As oracle user, run "crsctl check boot" failed with same error.

But OCR is owned by root:oinstall and permission is 644. There is no issue for oracle user to run dd against OCR device: /dev/raw/raw1 and /dev/raw/raw2.

Changes

Just restored OCR and Voting disk from backup due to storage problem before starting CRS on both nodes.

Cause

It is caused by mismatch of /etc/oracle/ocr.loc file on the two nodes.

On node 1, /etc/oracle/ocr.loc does not have ocrmirror defined:

ocrconfig_loc=/dev/raw/raw1
local_only=false

On node2, /etc/oracle/ocr.loc has ocrmirror defined:

ocrconfig_loc=/dev/raw/raw1
ocrmirrorconfig_loc=/dev/raw/raw2
local_only=false

CRS already started on node 1 without OCR mirror, that is why on 2nd node CRS report PROC-26 for OCR mirror as it is not same as OCR.

Solution

1. On node 2, run following command as root:

# ocrconfig -repair ocrmirror

This command will remove the entry ocrmirrorconfig_loc from /etc/oracle/ocr.loc, it can only be run when CRS is down on that node.

2. Wait for upto 60 seconds, CRS should be started automatically on the 2nd node.

3. If you want to add OCR mirror again, run following command as root user on either node (assume CRS is up on all nodes):

# ocrconfig -replace ocrmirror /dev/raw/raw2

It will add ocrmirror device in /etc/oracle/ocr.loc.

Note: ocrconfig -replace ocr|ocrmirror command only propagates the changes in /etc/oracle/ocr.loc to nodes which CRS is running (at the time command is issued). By comparison, ocrconfig -repair ocr|ocrmirror command only modifies /etc/oracle/ocr.loc on the node where the command is run from (while CRS is shutdown)."