In this Document
Goal |
Solution |
Rolling upgrade Oracle Clusterware from 10.1.0.x to 10.1.0.4/5 |
Rolling upgrade Oracle Clusterware from 10.1.0.x to 10.2.0.1 |
Rolling upgrade Oracle Clusterware from 10.2.0.1 to 10.2.0.2/x |
Rolling upgrade Oracle Clusterware from 10.1.0.x / 10.2.0.x to 11.1.0.x. |
Rolling upgrade Oracle Clusterware from 11.1.0.6 to 11.1.0.7/x |
Rolling upgrade from 10.1/10.2/11.1 to 11.2 |
Rolling upgrade to 11.2.0.2/11.2.0.3/11.2.0.4 |
Rolling upgrade to 12c |
References |
APPLIES TO:
Oracle Database - Enterprise Edition - Version 10.1.0.2 to 12.1.0.2 [Release 10.1 to 12.1]Information in this document applies to any platform.
GOAL
The purpose of this document is to provide information about the Oracle Clusterware (formerly CRS)
rolling upgrade feature and how to install a patchset or major release without Oracle Clusterware downtime.
While Oracle generally supports rolling upgrades for Oracle Clusterware, Oracle does not support
'slow, delayed, or extended' rolling upgrades, whereby any node in the cluster remains at the older
version for an extended period of time.
Once the rolling upgrade is initiated, by design the expectation would be that the upgrade rapidly
rolls through each node in the cluster and finishes within 24 hours.
While fully operational, the cluster will provide some restricted functionally during the time of
the rolling upgrade. For example: Oracle Clusterware will not accept the creation of new cluster
resources during the upgrade. ASM (if used in the cluster) will not perform an automatic
re-balancing until all nodes in the cluster have been upgraded successfully.
SOLUTION
Starting with Oracle Database 10.1.0.2, Rolling upgrade of the Oracle Clusterware is possible.
- Rolling upgrade of Oracle Clusterware is possible between patchset releases . E.g. applying 10.1.0.4 CRS Patchset to a 10.1.0.3 CRS_home.
- Rolling upgrade of Oracle Clusterware is possible between different releases. E.g upgrade from 10.1.0.x to 10.2.0.x
Rolling upgrade Oracle Clusterware from 10.1.0.x to 10.1.0.4/5
The following describe the 10.1.0.4 Oracle Clusterware patchset installation on top of 10.1.0.3 Oracle Clusterware version.
- Welcome Screen
Click Next
- Specify File Locations
choose the CRS_home and click Next (selection the CRS_home will automatically select and upgrade the CRS software)
- Selected Nodes
All nodes should be automatically selected click Next
- Summary
Click Next
- Install
- End of Installation
(in a new window the final patch install instructions will displayed)
The below example is a screen output from installing Oracle Clusterware patchset 10.1.0.4.
------------------------------------------------------------------------------------
Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patch your Oracle Clusterware installation in a rolling manner, one node at a time.
1. Log in as the root user.
2. As the root user, perform the following tasks:
a. Shutdown the CRS daemons by issuing the following command:
/etc/init.d/init.crs stop.
b. Run the shell script located at:
/opt/crs/install/root10104.sh
This script will automatically start the CRS daemons on the patched
node upon completion.
3. After completing this procedure, proceed to the next node and repeat.
Perform the root10104.sh on all nodes one after each other and so a rolling Oracle Clusterware upgrade will be done.
Rolling upgrade Oracle Clusterware from 10.1.0.x to 10.2.0.1
During an Oracle Clusterware installation, if OUI detects an existing Oracle Database 10g Release 1 (10.1) Cluster Ready Services, then you are given the option to perform a rolling upgrade by installing Oracle Database 10g Release 2 (10.2) Oracle Clusterware on a subset of cluster member nodes.
If only a subset of the cluster nodes are upgraded uncheck the one who shouldn't be upgraded at the "Specify Hardware Cluster Installation Mode" OUI screen.
During the "Product-Specific Prerequisite Checks" OUI will detect which nodes having the CSS/CRS stack running and will post a message like below:
Actual Result:
CSS Stack is running on the following nodes : node1,node2.
Check complete. The overall result of this check is: Failed <<<<
Problem:
The cluster is not in a state suitable for upgrade. One or more of the following conditions have not been met: The Cluster Synchronization Service (CSS) is running on one or more nodes in the cluster and/or appropriate directories within the Oracle Home are not writable.
Recommendation:
To upgrade Oracle 10g Release 1 Cluster Ready Services (CRS), you must shutdown CSS on all nodes, and ensure that specific directories within the Oracle 10g Release 1 CRS_homeare writable on all nodes. In the upgrade/ directory at the root of the Oracle Clusterware 10g Release 2 media there is a shell script 'preupdate.sh' that should be run as root on each node that you wish to upgrade prior to launching the installer. This script will shutdown the Oracle 10g Release1 CRS stack on that node and prepare the Oracle 10g Release 1 CRS_home such that the upgrade can take place. The script can be invoked using the following command: 'preupdate.sh -crshome <Oracle 10g Release 1 CRS_home location> -crsuser <Oracle 10g Release 1 CRS user name>'.
If you intend to perform a rolling upgrade, then you should shut down the Oracle Clusterware stack on the nodes you intend to upgrade, and unlock the CRS_home using the scriptmountpoint/clusterware/upgrade/preupdate.sh, which is available on the 10g Release 2 (10.2) installation media.
After the software is distributed to the node(s) the $CRS_home/install/rootupgrade.sh must be run as root to start the CSS/CRS stack.
When you run OUI and select the option to install Oracle Clusterware on a subset of nodes, OUI installs Oracle Database 10g Release 2 (10.2) Oracle Clusterware software into the existing CRS_home on the local and remote node subset. When you run the rootupgrade.sh script, it starts the Oracle Clusterware 10g Release 2 (10.2) stack on the subset cluster nodes, but lists it as an inactive version.
When all member nodes of the cluster are running Oracle Clusterware 10g Release 2 (10.2), then the new clusterware becomes the active version, otherwise the older 10.1.0.x version is still be used.
To check which active version is running on which node perform the following as root:
$CRS_home/bin/crsctl query crs softwareversion [<nodename>] - lists the version of CRS software installed
$CRS_home/bin/crsctl query crs activeversion - lists the CRS software operating version
If you intend to install RAC, then you must first complete the upgrade to Oracle Clusterware 10g Release 2 (10.2) on all cluster member nodes before you install the Oracle Database 10g Release 2 (10.2) version of RAC.
Rolling upgrade Oracle Clusterware from 10.2.0.1 to 10.2.0.2/x
The following describe the 10.2.0.2 Oracle Clusterware patchset installation on top of 10.2.0.1 Oracle Clusterware version.
- Welcome Screen
Click Next
- Specify File Locations
choose the CRS_home and click Next (selection the CRS_home will automatically select and upgrade the CRS software)
- Selected Nodes
All nodes should be automatically selected click Next
- Summary
Click Next
- Install
- End of Installation
(in a new window the final patch install instructions will displayed)
The below example is a screen output from installing Oracle Clusterware patchset 10.2.0.2
node1
node2
Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patch your Oracle Clusterware installation in a rolling manner, one node at a time.
To complete the installation of this patchset, you must perform the following tasks on each node:
1. Log in as the root user.
2. As the root user, perform the following tasks:
a. Shutdown the CRS daemons by issuing the following command:
/opt/crs/bin/crsctl stop crs
b. Run the shell script located at:
/opt/crs/install/root102.sh
This script will automatically start the CRS daemons on the
patched node upon completion.
3. After completing this procedure, proceed to the next node and repeat.
Perform the root102.sh on all nodes one after each other and so a rolling Oracle Clusterware upgrade will be done.
After a successful upgrade on all nodes the softwareversion and activeversion should be like below:
$CRS_Home/bin/crsctl query crs softwareversion
CRS software version on node [node1] is [10.2.0.2.0]
$CRS_Home/bin/crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.2.0]
Rolling upgrade Oracle Clusterware from 10.1.0.x / 10.2.0.x to 11.1.0.x.
Requirements for Rolling Upgrade of Oracle Clusterware
When upgrading Oracle Real Application Clusters from release 10.2.0.1 or 10.2.0.2 to release 11.1, rolling upgrade of Oracle Clusterware will require some additional steps (reference unpublished bug 5198903). Do the following to successfully complete the rolling upgrade:
- Upgrade Oracle Clusterware (that is, the CRS_home) to release 10.2.0.3, or 10.2.0.2 with CRS Bundle Patch #2 (reference unpublished bug 5256865).
Upgrading the database Oracle Home is not required. - Upgrade Oracle Clusterware from release 10.2.0.3 to release 11.1.
Note that each of the CRS upgrades in Steps 1 and 2 are rolling upgrades. Therefore, cluster outage is not required for these upgrades.
There is a similar requirement for rolling upgrade of 10.1.0.2 to 11.1 (reference unpublished Bug 5860479). To successfully perform a rolling upgrade of Oracle Clusterware from 10.1.0.2, do the following:
- Upgrade Oracle Clusterware (that is, the CRS_home) to release 10.1.0.3.
- Upgrade Oracle Clusterware from release 10.1.0.3 to release 11.1.
The Oracle Clusterware upgrades in Steps 1 and 2 are rolling upgrades, therefore no cluster outage is required.
This is a problem for Oracle Real Application Clusters only. For single instance database installations with ASM, Oracle Clusterware may be upgraded directly from release 10.1.x or 10.2.x to release 11.1.
Manual check
To check wether a node has the fix installed or not the following script could help to determine this.
RC=$?
if [ $RC -ne 0 ]; then
echo Please upgrade Oracle Clusterware to 10.2.0.3 first, to support rolling upgrade
exit 1
fi
---------
If the script does not work, run nm manually and depending on the OS it should return something like this:
Solaris:
[5492] | 4296074891| 23|OBJT |GLOB |0 |13 |$XBX9QuLTfeRFWsa.clssgmSuspendAllGrocks.__func__
[857] | 4295271840| 604|FUNC |LOCL |0 |9 |clssgmSuspendAllGrocks
Linux:
0000000000442fbc t clssgmSuspendAllGrocks
The upgrade steps
During an Oracle Clusterware installation, if Oracle Universal Installer (OUI) detects an existing Oracle Clusterware 10g Release 1 or Release 2 installation, then you are given the option to perform a rolling upgrade by installing Oracle Clusterware 11g Release 1 on a subset of cluster member nodes.
If only a subset of the cluster nodes are to be upgraded, uncheck the ones which shouldn't be upgraded at the "Specify Hardware Cluster Installation Mode" OUI screen.
During the "Product-Specific Prerequisite Checks" OUI will detect which nodes have the CSS/CRS stack running and will post a message like:
Actual Result: CSS Stack is running on the following nodes : node_2.
Check complete. The overall result of this check is: Failed <<<<
Problem: The cluster is not in a state suitable for upgrade. One or more of the following
conditions have not been met: The Cluster Synchronization Service (CSS) is running on one or more nodes in the cluster and/or appropriate directories within the Oracle Home are not writable.
Recommendation: To upgrade Oracle 10g Release Clusterware, you must shutdown CSS on all nodes, and ensure that specific directories within the Oracle 10g Release Clusterware Home are writable on all nodes. In the upgrade/ directory at the root of the Oracle Clusterware 11g Release 1 media there is a shell script 'preupdate.sh' that should be run as root on each node that you wish to upgrade prior to launching the installer.
This script will shutdown the Oracle 10g Release Clusterware stack on that node and prepare the Oracle 10g Release Clusterware Home such that the upgrade can take place. The script can be invoked using the following command:
'preupdate.sh -crshome <Oracle 10g Release Clusterware Home location> -crsuser <Oracle 10g Release Clusterware user name>'.
If you intend to perform a rolling upgrade, then you should run the script mountpoint/clusterware/upgrade/preupdate.sh, which is available on the 11g Release 1 installation media:
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is down now.
On the "Product-Specific Prerequisite Checks" screen click Retry, and now the message for the CSS check should be:
Check complete. The overall result of this check is: Passed
Continue with Next.
After the software is distributed to the node(s), the CRS_home/install/rootupgrade.sh must be run as root to start the Oracle Clusterware.
When you run OUI and select the option to install Oracle Clusterware on a subset of nodes, OUI installs Oracle Clusterware software into the existing CRS_home on the local and remote node subset. When you run the rootupgrade.sh script, it starts the Oracle Clusterware 11g Release 1 stack on the subset cluster nodes, but lists it as an inactive version.
When all member nodes of the cluster are running Oracle Clusterware 11g Release 1, then the new clusterware becomes the active version, otherwise the older 10.1.0.x or 10.2.0.x version is still be used.
To check which active version is running on which node perform the following as root:
$CRS_home/bin/crsctl query crs softwareversion [<nodename>] - lists the version of Oracle Clusterware software installed
$CRS_home/bin/crsctl query crs activeversion - lists the Oracle Clusterware software operating version
If you intend to install (or upgrade) RAC, then you must first complete the upgrade to Oracle Clusterware 11g Release 1 on all cluster member nodes before you install the Oracle Database 11g Release 1 version of RAC.
Rolling upgrade Oracle Clusterware from 11.1.0.6 to 11.1.0.7/x
The following describe the 11.1.0.7 Oracle Clusterware patchset installation on top of 11.1.0.6 Oracle Clusterware version.
- Welcome Screen
Click Next
- Specify File Locations
choose the CRS_home and click Next (selection the CRS_home will automatically select and
upgrade the CRS software)
- Selected Nodes
All nodes should be automatically selected click Next
- Product-Specific Prerequisite Checks
- Summary
Click Next
- Install
- End of Installation
(in the installer window the final patch install instructions will displayed)
The below example is a screen output from installing Oracle Clusterware patchset 11.1.0.7
node1
node2
Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patch your CRS installation in a rolling manner, one node at a time.
To complete the installation of this patchset, you must perform the following tasks on each node:
1. Log in as the root user.
2. As the root user, perform the following tasks:
a. Shutdown the CRS daemons by issuing the following command:
/opt/crs/bin/crsctl stop crs
b. Run the shell script located at:
/opt/crs/install/root111.sh
This script will automatically start the CRS daemons on the
patched node upon completion.
3. After completing this procedure, proceed to the next node and repeat.
Perform the root111.sh on all nodes one after each other and so a rolling Oracle Clusterware upgrade will be done.
After a successful upgrade on all nodes the softwareversion and activeversion should be like below:
$CRS_Home/bin/crsctl query crs softwareversion
CRS software version on node [node1] is [11.1.0.7]
$CRS_Home/bin/crsctl query crs activeversion
CRS active version on the cluster is [11.1.0.7]
Rolling upgrade from 10.1/10.2/11.1 to 11.2
Oracle Clusterware and Automatic Storage Management (ASM) are installed into a single Oracle home called the Grid home. This installation is referred to as the Oracle grid infrastructure for a cluster installation.
When upgrading, you must upgrade both Oracle Clusterware and ASM at the same time.
Before starting the upgrade please read the Oracle® Database Readme 11g Release 2 (11.2), to be aware of known issues.
When upgrading Oracle Real Application Clusters from release 10.2.0.1 or 10.2.0.2 to release 11.2, rolling upgrade of Oracle Clusterware will require some additional steps (reference Bug 5198903). Complete the following to successfully complete the rolling upgrade:
1. Upgrade Oracle Clusterware (that is, the CRS home) to release 10.2.0.3, or 10.2.0.2 with CRS Bundle Patch#2 (reference Bug 5256865).
2. Upgrade Oracle Clusterware from release 10.2.0.3 to release 11.2.
There is a similar requirement for rolling upgrade of 10.1.0.2 to 11.2 (reference Bug 5860479). To
successfully perform a rolling upgrade of Oracle Clusterware from 10.1.0.2, complete the following:
1. Upgrade Oracle Clusterware (that is, the CRS home) to release 10.1.0.3.
2. Upgrade Oracle Clusterware from release 10.1.0.3 to release 11.2.
The Oracle Clusterware upgrades in Steps 1 and 2 are rolling upgrades, therefore no cluster outage is
required.
This is a problem for Oracle Real Application Clusters only. For single-instance database installations with Oracle ASM, Oracle Clusterware may be upgraded directly from release 10.1.x or 10.2.x to release 11.2.
The upgrade steps
During an Oracle Clusterware installation, if Oracle Universal Installer (OUI) detects an existing Oracle
Clusterware 10gR1/10gR2 or 11gR1, then you are given the option to perform a rolling upgrade by installing Oracle Clusterware 11gR2.
The upgrade to Oracle Clusterware 11gR2 will be an out of place upgrade hence a new CRS_HOME including ASM will be created which will be refered as the new GRID_Infrastructure home. If an existing ASM_HOME is present in 10gR1/10gR2 or 11gR1 then ASM will be upgraded automatically as well.
After the new 11gR2 software is installed on all nodes the following configuration scripts need to be executed as the "root" user in each cluster node.
After this script run successfully on all nodes in the cluster the active and softwareversion on each node should be 11.2.0.1.0.
$GRID_Home/bin/crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.1.0]
$GRID_Home/bin/crsctl query crs softwareversion
Oracle Clusterware version on node [node1] is [11.2.0.1.0]
The new OCR version should be version 3 using Oracle Clusterware 11gR2.
$GRID_Home/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
After the successful upgrade to 11gR2 all the cluster resources should be ONLINE. In order to see the
current status '$GRID_Home/bin/crsctl status resource -t' should run. The 11gR1 crs_stat command is obsolete in 11gR2.
Rolling upgrade Database software
If you use logical standby databases, you can use Oracle Data Guard SQL Apply to perform rolling upgrades to upgrade Oracle Database 10g release 1 (10.1.0.3) to release 1 (10.1.0.4). During a rolling upgrade,you can run different releases of the Oracle software on the primary and logical standby databases while you upgrade them one at a time. If you do this, you will incur minimal downtime on the primary database. For information and instructions on rolling upgrades with logical standby databases, review document Note:300479.1
Rolling upgrade broken on Linux Itanium
Due to Bug 6706111 - Bug 6835261 - and others the CRS patch for Linux Itanium 10.2.0.3 cannot support rolling upgrade
The same seems is happening on 10.2.0.4 as well, see Bug 8434285.
Downgrade Oracle Clusterware
In order to downgrade Oracle Clusterware read the specific section from each versions Clusterware Installation Guide, like for 11.2 downgrade section "F.7 Downgrading Oracle Clusterware After an Upgrade" from the Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2).
Rolling upgrade to 11.2.0.2/11.2.0.3/11.2.0.4
Refer to the following note for more details:
note 1363369.1 - Things to Consider Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM
note 1312225.1 - Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure
Rolling upgrade to 12c
Refer to:
note 1579762.1 - How to Upgrade to Oracle Grid Infrastructure 12c Release 1 and Known Issues
REFERENCES
NOTE:1363369.1 - Things to Consider Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM
NOTE:300479.1 - 10g Rolling Upgrades with Logical Standby
NOTE:466181.1 - Oracle 10g Upgrade Companion
NOTE:601807.1 - Oracle 11gR1 Upgrade Companion
NOTE:785351.1 - Oracle 11gR2 Upgrade Companion
BUG:6835261 - AFTER APPLYING PATH 6706110 IN ONE NODE OF 4 NODE RAC, THE CRS DOES NOT COME UP
BUG:8434285 - ROOT102.SH HANG DURING CRS ROLLING UPGRADE FROM 10.2.0.3 TO 10.2.0.4
NOTE:1312225.1 - Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure/ASM