How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure (CRS) [ID 969254.1]


How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure (CRS) [ID 969254.1]


Modified 10-OCT-2010     Type HOWTO     Status PUBLISHED


In this Document
     Failure before is ran:
     Failure during
     Failure during plug-in:

Applies to:

Oracle Server - Enterprise Edition - Version: to - Release: 11.2 to 11.2
Generic UNIX
Generic Linux


This goal of this note is to provide steps to proceed from a failed upgrade to 11gR2  (11.2) Grid Infrastructure(CRS), so another attempt can be pursued.



Failure before is ran:

Case applies here is runInstaller failed before is executed.

You will need identify cause of the failure from screen output and installation logs in $ORACLE_BASE/oraInventory/logs (three files .log, .err and .out will be generated each time you run runInstaller) before execute clean up steps.

  A. As grid user, remove the failed installation by running $GRID_HOME/deinstall/deinstall

  B. If step A succeeded, skip to step C, otherwise perform the following:


## To detach failed GRID_HOME:
$OLD_CRS_HOME/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=$GRID_HOME

## To confirm $GRID_HOME is removed from central inventory:
$OLD_CRS_HOME/OPatch/opatch lsinventory -all  

## To remove files in failed GRID_HOME on all nodes manually:
/bin/rm -rf $GRID_HOME



  C. Detach and attach pre-11gR2 CRS home to set CRS flag to true on all nodes:



$OLD_CRS_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_CRS_HOME

$OLD_CRS_HOME/oui/bin/runInstaller -silent -local -ignoreSysPrereqs -attachHome ORACLE_HOME=$OLD_CRS_HOME ORACLE_HOME_NAME=$OLD_CRS_HOME_NAME LOCAL_NODE=local_nodename CLUSTER_NODES=local_nodename,remote_nodename CRS=true


  D. Restart runInstaller to upgrade to 11.2 Grid Infrastructure

Failure during

You will need identify cause of the failure by reviewing logs in $GRID_HOME/cfgtoollogs/crsconfig, $GRID_HOME/log and $ORACLE_BASE/oraInventory/logs; once cause is identified and issue is fixed, please proceed with steps below:

  0. Restartable root script -

For and above, is restartable. Once cause is identified and fixed, can be executed again on the failed node, if it succeeds, you can continue with the installation, otherwise proceed to next step (Step 1) of the note.

  1. Identify "last node"

Mark the node as "last node" where you find pre-upgrade OCR backup in $GRID_HOME/cdata/ocr$OLD_CRS_VERSION (for example, /ocw/grid/cdata/ocr10.; in most case it's first node where you execute

  2. As root, on all but "last node", execute:

$GRID_HOME/crs/install/ -downgrade -force

      You can skip the command on node(s) where you never executed
  3. As root, on "last node", execute:


$GRID_HOME/crs/install/ -downgrade -force -lastnode -oldcrshome $OLD_CRS_HOME -version $OLD_CRS_VERSION


      Please note $OLD_CRS_VERSION need to be in format of n.n.n.n.n, for example,

      If OCR downgraded successfully, skip to step 5, otherwise go to step 4.

  4. Deconfigure 11gR2
      Execute this step only if step 3 failed.

        4A. As root, on all but last node, execute:

$GRID_HOME/crs/install/ -verbose -deconfig -force

        4B. As root, on last node, execute:

$GRID_HOME/crs/install/ -verbose -deconfig -force -lastnode

  • If fails with following error:


Using configuration parameter file: /ocw/grid/crs/install/crsconfig_params
Can't open /etc/oracle/scls_scr/node1/root/ohasdrun for write: No such file or directory at /ocw/grid/crs/install/ line 1021.

        As root, please create a empty ohasdrun and try the same again:

mkdir -p /etc/oracle/scls_scr/$HOSTNAME/root
touch /etc/oracle/scls_scr/$HOSTNAME/root/ohasdrun


  • If hang forever, please disable 11gR2 from auto startup, reboot the node, and re-execute the same command.  To disable CRS from auto startup, as root execute:

    $GRID_HOME/bin/crsctl disable crs

  • Once Step 4 finishes, checkpoint file shouldn't exist. To verify:

              ls -l $ORACLE_BASE/Clusterware/ckptGridHA_<nodename>.xml

If it's still there, please remove it manually with "rm" command.

  5. Execute $OLD_CRS_HOME/

     - If  new node were added or any of the following were changed after original setup of pre-11.2 CRS, please engage Oracle Support for instruction on how to modify pre-11.2 configuration manually.

            OCR or Voting disk location
            Network information for public, VIP or private

     - If OCR or Voting Disk are located on Cluster File System or Network File System, please make sure they exist before execution of If they don't exist, please create dummy OCR/Voting Disk files with /bin/touch.

      As root, execute $OLD_CRS_HOME/ on each node sequentially, it will start old CRS at end of the script.

      If failed to bring up pre-11.2 CRS, please go to step 6; otherwise skip to step 7.

  6. Deconfigure and reconfigure pre-11.2 CRS

      Execute this step only if step 5 failed.

        6A. As root, on all nodes, execute $OLD_CRS_HOME/install/
        6B. As root, on first node, execute $OLD_CRS_HOME/install/
        6C. As root, execute $OLD_CRS_HOME/ on all nodes sequentially.

  7. Restore OCR

    If you execute either step 4 or step 6, backup of OCR need to be restored.

    If you backed up OCR prior to upgrade, you can use that to restore, otherwise go through $OLD_CRS_HOME/cdata/{CLUSTERNAME} on all nodes to identify a good automatic backup

      7A. As root, on all nodes, stop CRS:

$OLD_CRS_HOME/bin/crsctl stop crs

      7B. As root, on node where identified OCR backup resides:

$OLD_CRS_HOME/bin/ocrconfig -restore name_of_ocr_backup

  8. At this point, pre-11.2 CRS should be running, as CRS user, execute following on all nodes to validate:


$OLD_CRS_HOME/bin/crsctl query crs activeversion
$OLD_CRS_HOME/bin/crsctl query crs softwareversion
$OLD_CRS_HOME/bin/crsctl check crs
$OLD_CRS_HOME/bin/crs_stat -t
$OLD_CRS_HOME/bin/oifcfg iflist -p -n
$OLD_CRS_HOME/bin/oifcfg getif
cat /etc/oracle/ocr.loc /var/opt/oracle/ocr.loc
$OLD_CRS_HOME/bin/crsctl get css diagwait
$OLD_CRS_HOME/bin/srvctl config nodeapps -n `hostname | cut -d'.' -f1 | awk '{ print tolower($0); }'` -a
$OLD_CRS_HOME/OPatch/opatch lsinventory oh=$OLD_CRS_HOME
ifconfig              ## location and exact syntax in
note 1054902.1


  9. Execute

      Unset ORACLE_HOME, ORACLE_BASE, ORACLE_SID, ORA_CRS_HOME, TNS_ADMIN in both root and oracle's profile and current shell environment; ensure firewall is down on all nodes, and execute $GRID_HOME/ sequentially on all nodes.

  10. Post steps:

      If any configuration assistant already finished successfully previously and you'd like to skip it, please be sure RERUN is set to false in $GRID_HOME/cfgtoollogs/configToolAllCommands, for example:


$ cat $GRID_HOME/cfgtoollogs/configToolAllCommands
/ogrid/gbase/oui/bin/ ORACLE_HOME=/ogrid/gbase MODE=perform ACTION=configure RERUN=false $*


      As GRID user, execute $GRID_HOME/cfgtoollogs/configToolAllCommands on first node.

Failure during plug-in:

    Please identify the cause of failed plug-in and go to Step 10.


NOTE:1050908.1 - How to Troubleshoot Grid Infrastructure Startup Issues
NOTE:1053147.1 - 11gR2 Clusterware and Grid Home - What You Need to Know
NOTE:1054902.1 - How to Validate Network and Name Resolution Setup for the Clusterware and RAC
NOTE:1056322.1 - How to Troubleshoot 11gR2 Installation Issue
NOTE:942166.1 - How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation
NOTE:948456.1 - Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment