Troubleshooting CRSD Start up Issue (Doc ID 1323698.1)

In this Document

Purpose

Troubleshooting Steps

A. Find out whether OCR master is healthy

B. Enable further tracing

C. Take pstack

References

---

This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process and therefore has not been subject to an independent technical review.

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

PURPOSE

When crsd.bin fails to start, resource ora.crsd  status from "crsctl stat res -t -init" does not show ONLINE, and "crsctl check crs" may report "CRS-4535: Cannot communicate with Cluster Ready Services".

Common cause of crsd.bin start up failure is in note 1050908.1 - "Case 4: CRSD.BIN does not start". This note provides further steps to troubleshoot as in some cases crsd.bin dumps with no clear reason which makes it difficult to identify the cause. For example, from $GRID_HOME/log//crsd/crsd.log:

2011-05-11 10:54:17.636: [    AGFW][11828] Initializing the resource ora.db1.db 1 1 for type ora.database.type
2011-05-11 10:54:17.636: [    AGFW][11828] SR: acl = owner: oradb3:rwx,pgrp:oinstall:rwx,other::r--
2011-05-11 10:54:17.652: [    CRSD][11828] Dump State Starting ...
..
2011-05-11 10:54:17.686: [   CRSPE][11828] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
..
2011-05-11 10:54:17.689: [   CRSPE][11828] Dumping ICE contents...:ICE operation count: 0

By reading the log, it's unclear why crsd.bin fails to come up. The cause is bug 10638125 (duplicates: bug 14769944 bug 11925853), it happens as OS user "oradb3" is removed.

TROUBLESHOOTING STEPS

When crsd.bin fails to start, the following can be tried to identify the cause:

A. Find out whether OCR master is healthy

A1. Find out which node is OCR Master:

grep "OCR MASTER" $GRID_HOME/log//crsd/crsd.log

OCR Slave will have "th_master: NEW OCR MASTER IS 1" in latest crsd.log entry while OCR Master will have "th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2". For details, refer to note 1281982.1

A2. Find out whether OCR Master is healthy:

Log on to OCR Master node and execute the following:

$GRID_HOME/bin/crsctl check crs

It should finish quick and report "CRS-4537: Cluster Ready Services is online"

$GRID_HOME/bin/crsctl stat res -t

It should finish quick, it's ok to have some resources to have other than ONLINE status as long as those are expected. For example, a database resource may have OFFLINE status as it's intended to be offline.

If OCR Master is healthy, the issue may be limited to OCR Slave node.

 

 

B. Enable further tracing

Execute the following steps as root:

B1. List all crsd modules: 

# $GRID_HOME/bin/crsctl lsmodules crs

B2. Find out the current trace level for all crsd modules - the output can be used to to revert back to original trace level once the issue is solved:

# $GRID_HOME/bin/crsctl get log crs all

B3. Set trace level to 5 for all modules: 

# $GRID_HOME/bin/crsctl set log crs all:5

Alternatively trace level for each modules can be set individually: 

# $GRID_HOME/bin/crsctl set log crs CRSPE:5

Note: The module name is case sensitive


B4. Once the issue is solved, set trace level back to original value for all crs modules with output from B2

# $GRID_HOME/bin/crsctl set log crs AGENT:1
..

C. Take pstack

When crsd.bin dumps, it will store a thread stack in $GRID_HOME/log//crsd/crsdOUT.log

Most of the time, there may not be a time window to take pstack as crsd.bin aborts very quick; but in case crsd.bin stays up for a short while, take a few pstack at interval of 30 seconds as root:

Find out pid of crsd.bin: ps -ef| grep crsd.bin

Take pstack:

  AIX        : /bin/procstack
  Linux        : /usr/bin/pstack
  Solaris    : /usr/bin/pstack

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/15747463/viewspace-774900/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/15747463/viewspace-774900/