MOS：Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructur

Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure

RAC Best Practice and Starter Kit Information

The goal of the Oracle Real Application Clusters (RAC) series of Best Practice and Starter Kit notes is to provide the latest information on generic and platform specific best practices for implementing an Oracle cluster using Oracle Clusterware or Oracle Grid Infrastructure (11gR2) and Oracle Real Application Clusters. These documents are compiled and maintained based on Oracle Support's ongoing experience with its global RAC customer base. The process used to install, configure, and create an Oracle Cluster and an Oracle Real Application Clusters (RAC) database shares much in common across all supported operating system platforms. Despite these commonalities, many aspects of the deployment process are O/S specific. As such, there is a Generic Starter Kit, as well as a separate Starter Kit for each supported platform. These Starter Kits provide the latest information in terms of installation help, as well as best practices for ongoing/existing implementations.

• Document 810394.1 RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)
• Document 811306.1 RAC Assurance Support Team RAC Starter Kit (Linux) 
• Document 811280.1 RAC Assurance Support Team RAC Starter Kit (Solaris) 
• Document 811271.1 RAC Assurance Support Team RAC Starter Kit (Windows) 
• Document 811293.1 RAC Assurance Support Team RAC Starter Kit (AIX) 
• Document 811303.1 RAC Assurance Support Team RAC Starter Kit (HP-UX)

The Generic Starter Kit has attached documents with sample test plan outlines for system testing and basic load-testing to validate the infrastructure is setup correctly, as well as a sample project plan outline for your RAC implementation. In addition, best practices for Storage and networking and other areas are included. The Platform-specific Starter Kit notes complement this with Step-by-Step Installation Guides, for each particular platform, as well as best practices for the specific platform in question.

RACcheck - RAC Configuration Audit Tool

RACcheck is a RAC Configuration Audit tool designed to audit various important configuration settings within Real Application Clusters (RAC), Oracle Clusterware (CRS), Automatic Storage Management (ASM) and Grid Infrastructure environments. This utility is to be used to validate the Best Pracices and Success Factors defined in the series of Oracle Real Application Clusters (RAC) Best Practice and Starter Kit notes which are maintained by the RAC Assurance development and support teams. At present RACcheck supports Linux (x86 and x86_64), Solais (SPARC and x86_64), HP-UX (Itanium and PA-RISC with the bash shell) and AIX (with the bash shell) platforms. Those customers running RAC on the RACcheck supported platorms are strongly encouraged to utilize this tool identify potential configuration issues that could impact the stability of the cluster.

• Document 1268927.1 RACcheck - RAC Configuration Audit Tool

Note:  Oracle is constantly generating and maintaining Best Practices and Success Factors from the global customer base.  As a result the RACcheck utility is frequently updated with this information.  That said, it is recommended that you ensure you are using the version of RACcheck prior to execution.

Additional Notes on RAC and Clusterware Installation and Frequently asked Questions

While the Starter Kits referenced above will contain much of what you need to get started, additional notes and resources that are important to take heed of, particularly for a new implementation, can be found here in the following FAQ’s and other references:

• Document 220970.1 RAC: Frequently Asked Questions
• Document 316817.1 CLUSTER VERIFICATION UTILITY FAQ 
• Document 332257.1 Using Oracle Clusterware with Vendor Clusterware FAQ 
• Document 397460.1 Oracle's Policy for Supporting RAC 10g with Symantec SFRAC 
• Document 391771.1 OCFS2 - Frequently Asked Questions 
• Document 787420.1 Cluster Interconnect in Oracle 10g and 11g
• Document 1328466.1 Cluster Health Monitor (CHM) FAQ

Key notes related to OS Installation requirements for a RAC or Oracle Clusterware environment

• Document 359515.1 Mount Options for Oracle files when used with NAS devices
• Document 401132.1 How to install Oracle Clusterware with shared storage on block devices 
• Document 357472.1 Configuring device-mapper for CRS/ASM 
• Document 169706.1 Oracle Database on AIX,HP-UX,Linux,MacOSX,Solaris,Tru64 Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.1) 

Key notes introducing differences in Oracle 11gR2

• Document 1053147.1 11gR2 Clusterware and Grid Home - What You Need to Know
• Document 948456.1 Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment 
• Document 1050908.1 How to Troubleshoot Grid Infrastructure Startup Issues
• Document 887522.1 11gR2 Grid Infrastructure Single Client Access Name (SCAN)Explained

RAC and Clusterware Monitoring, Troubleshooting and Data Collection Notes

To effectively manage your RAC cluster, it is important to know how to monitor, troubleshoot and collect data, both for your own diagnosis, and also to provide supporting information when logging a service request. The following Notes will give you information on how to do both: 


Tools for Monitoring RAC and Clusterware Environments

• Document 301137.1 OS Watcher User Guide
• Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle and CRS 
• Document 736752.1 Introducing Cluster Health Monitor 
• Document 811151.1 How to install Oracle Cluster Health Monitor (former IPD/OS) on Windows 
• Document 314422.1 Remote Diagnostics Agent (RDA) 
• Document 422893.1 11g Understanding Automatic Diagnostic Repository 
• Document 559365.1 Using Diagwait as a Diagnostic.
• Document 1268927.1 RACcheck - RAC Configuration Audit Tool
• Document 135714.1  Script to Collect RAC Diagnostic Information (racdiag.sql)

 

References needed for Troubleshooting and Data Collection

• Document 289690.1 Data Gathering for Troubleshooting CRS issues 
• Document 330358.1 CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide 
• Document 557204.1 Crsd Ocssd Evmd Log Rotation Policy 
• Document 563566.1 Troubleshooting gc block lost and Poor Network Performance in a RAC Environment
• Document 219361.1 Troubleshoot ORA-29740 errors in a RAC Environment. 
• Document 745960.1 11g How to Unpack a Package in to ADR 
• Document 443529.1 11g Quick Steps to Package and Send Critical Error Diagnostic Information to Support 
• Document 726446.1 What Is The Difference Between RDA and Configuration Manager 
• Document 77483.1 External Support FTP site: Information Sheet 
• Document 783456.1 CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage
• Document 1050693.1 Troubleshooting 11.2 Clusterware Node Evictions (Reboots)
• Document 265769.1 Troubleshooting 10g and 11.1 Clusterware Reboots
• Document 1367153.1 Top 5 Issues That Cause Node Reboots or Unexpected Recycle of CRS
• Document 1485042.1  11gR2 GI/ASM/RAC Universal Collection Guide

 

RAC and Clusterware Testing and Performance Tuning

Much of RAC performance tuning is no different from single instance database€ tuning. However, there are some additional areas to check in a RAC environment when tuning or troubleshooting performance related issues. The following notes will make you aware of some of the areas where performance tuning and diagnosis has a specific RAC flavor:

• Document 390374.1 Oracle Performance Diagnostic Guide (OPDG)
• Document 181489.1 Tuning Inter-Instance Performance in RAC and OPS 
• Document 395314.1 RAC Hangs due to small cache size on SYS.AUDSES$ 
• Document 563566.1 gc lost blocks diagnostics 
• Document 361323.1 HugePages on Linux: What It Is... and What It Is Not... 
• Document 280939.1 Checklist for Performance Problems with Parallel Execution 
• Document 759565.1 Turning NUMA on can cause database hangs

 

RAC and Clusterware Certification, Maintenance, and Patching References

Maintenance of a RAC environment is similar in many respects to maintaining a single-instance environment. However, with the Clusterware and Grid Infrastructure stacks, there are some additional tasks to be aware of regarding ongoing maintenance, as well as maintenance that may be brought on by changes to your environment. Below are some references speaking to the areas of general maintenance that may be needed or affect RAC environments in a particular way, as well as references to patching, which is a key part of an ongoing maintenance strategy.

Certification References for RAC and Clusterware Environments

• Document 184875.1 How To Check The Certification Matrix for Real Application Clusters
• Document 1065024.1 Database 11g Release 2 Certification Highlights
• Document 337737.1 Oracle Clusterware - ASM - Database Version Compatibility

General Maintenance References for RAC and Clusterware Environments

• Document 428681.1 How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk
• Document 276434.1 Modifying the VIP or VIP Hostname of a 10g or 11g Oracle Clusterware Node. 
• Document 283684.1 How to Change Interconnect/Public Interface IP or Subnet in Oracle Clusterware 
• Document 754594.1 Potential disk corruption when adding a node to a cluster
• Document 403743.1 VIP Failover Take Long Time After Network Cable Pulled 
• Document 563905.1 Implementing LIBUMEM for CRS on Solaris 64 with 3rd Party Clusterware 
• Document 92602.1 How to Password Protect the Listener 
• Document 1373500.1  Top 5 Database and/or Instance Performance Issues in RAC Environment

Patching Best Practices and References for RAC and Oracle Clusterware 
As part of an overall maintenance strategy, it is critical that customers have a formal strategy to stay in front of known issues and bugs. To make it easier for customers to obtain and deploy fixes for known critical issues please refer to the following list of references.

• Document 209768.1 Database, FMW, Em Grid Control and OCS Software Error Correction Support Policy.
• Document 316900.1 Oracle 10g Release 2 (10.2) Support Status and Alerts 
• Document 454507.1 Oracle 11g Release 1 (11.1) Support Status and Alerts 
• Document 742060.1 Release Schedule of Current Database Releases. 
• Document 756671.1 Oracle Recommended Patches -- Oracle Database. 
• Document 854428.1 Introduction to Patch Set Updates (PSUs) 
• Document 850471.1 Oracle Announces First Patch Set Update For Oracle Database Release 10.2 
• Document 405820.1 10.2.0.X CRS Bundle Patch Information. 
• Document 810663.1 11.1.0.X CRS Bundle Patch Information. 
• Document 1082394.1 11.2.0.X Grid Infrastructure Bundle Patch Information. 
• Document 363254.1 Applying one-off Oracle Clusterware patches in a mixed version home environment 
• Document 761111.1 Online Patches (Hot Patching). 
• Document 438314.1 Critical Patch Update - Introduction to Database n-Apply CPUs. 
• Document 1082394.1 11.2.0.X Grid Infrastructure PSU Known Issues

 

Upgrade and Platform Migration for RAC and Oracle Clusterware

When migrating a RAC cluster to a new platform or upgrading a RAC cluster from an older version of Oracle to the current version of Oracle, the following references will be helpful:

• Document 466181.1 Oracle 10gR2 Upgrade Companion.
• Document 601807.1 Oracle 11gR1 Upgrade Companion. 
• Document 785351.1 Oracle 11gR2 Upgrade Companion. 
• Document 429825.1 Complete Checklist for Manual Upgrades to 11gR1. 
• Document 747457.1 How to Convert 10g Single-Instance database to 10g RAC using Manual Conversion procedure. 
• Document 338706.1 Cluster Ready Services (CRS) rolling upgrade

References for running RAC with Applications

• Document 790189.1 Oracle Clusterware and Application Failover Management

RAC and Oracle E-Business

• Document 362135.1 Configuring Oracle Applications Release 11i with Oracle10g Release 2 Real Application Clusters and Automatic Storage Management.
• Document 380489.1 Using Load-Balancers with Oracle E-Business Suite Release 12 
• Document 727171.1 Implementing Load Balancing On Oracle E-Business Suite - Documentation For Specific Load Balancer Hardware 
• Document 217368.1 Advanced Configurations and Topologies for Enterprise Deployments of E-Business Suite 11i 
• Document 403347.1 MAA Roadmap for the E-Business Suite 
• Document 388577.1 Using Oracle 10gR2 RAC and Automatic Storage Management with Oracle E-Business Suite Release 12

RAC ASM Documentation

Automatic Storage Management (ASM) is an evolution in file system and volume management functionality for Oracle database files. ASM further enhances automation and simplicity in storage management that is critical to the success of the Oracle grid architecture. ASM also improves file system scalability and performance, manageability and database availability for RAC environments. Use of ASM with RAC is an Oracle Best Practice.

• Document 1187723.1 Master Note for Automatic Storage Management (ASM)
• Document 1187674.1 Master Note for Oracle Database Machine and Exadata Storage Server

 

MAA/Standby Documentation

Data Guard provides the management, monitoring, and automation software infrastructure to create and maintain one or more standby databases to protect Oracle data from failures, disasters, errors, and data corruptions. As users commit transactions at a primary database, Oracle generates redo records and writes them to a local online log file.

• Document 1101938.1 Master Note for Data Guard

 

Using My Oracle Support Effectively

• Document 166650.1 Working Effectively with Global Customer Support
• Document 747242.5 My Oracle Support FAQ
• Document 868955.1 My Oracle Support Health Checks Catalog
• Document 199389.1 Escalating Service Requests with Oracle Support Services

REFERENCES

NOTE:1054902.1  - How to Validate Network and Name Resolution Setup for the Clusterware and RAC
NOTE:747457.1  - How to Convert 10g Single-Instance database to 10g RAC using Manual Conversion procedure
NOTE:754594.1  - Potential disk corruption when adding a node to a cluster
NOTE:756671.1  - Oracle Recommended Patches -- Oracle Database
NOTE:283684.1  - How to Modify Private Network Information in Oracle Clusterware
NOTE:438314.1  - Critical Patch Update - Introduction to Database n-Apply CPUs
NOTE:850471.1  - Oracle Announces First Patch Set Update For Oracle Database Release 10.2
NOTE:854428.1  - Patch Set Updates for Oracle Products
NOTE:811293.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (AIX)
NOTE:169706.1  - Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
NOTE:742060.1  - Release Schedule of Current Database Releases
NOTE:745960.1  - 11g How to Unpack a Package in to ADR
NOTE:289690.1  - Data Collecting for Troubleshooting Oracle Clusterware (CRS or GI) And Real Application Cluster (RAC) Issues
NOTE:811271.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (Windows)
NOTE:301137.1  - OSWatcher Black Box (Includes: [Video])
NOTE:359515.1  - Mount Options for Oracle files when used with NFS on NAS devices
NOTE:361323.1  - HugePages on Linux: What It Is... and What It Is Not...
NOTE:761111.1  - RDBMS Online Patching Aka Hot Patching
NOTE:454507.1  - ALERT: Oracle 11g Release 1 (11.1) Support Status and Alerts
NOTE:459694.1  - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
NOTE:466181.1  - Oracle 10g Upgrade Companion
NOTE:557204.1  - Oracle Clusterware CRSD OCSSD EVMD Log Rotation Policy
NOTE:265769.1  - Troubleshooting 10g and 11.1 Clusterware Reboots
NOTE:184875.1  - How To Check The Certification Matrix for Real Application Clusters
NOTE:276434.1  - How to Modify Public Network Information including VIP in Oracle Clusterware
NOTE:811303.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (HP-UX)
NOTE:759565.1  - Oracle NUMA Usage Recommendation
NOTE:280939.1  - Checklist for Performance Problems with Parallel Execution
NOTE:811306.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (Linux)
NOTE:220970.1  - RAC: Frequently Asked Questions
NOTE:209768.1  - Database, FMW, EM Grid Control, and OCS Software Error Correction Support Policy
NOTE:726446.1  - What Is The Difference Between RDA and Oracle Configuration Manager?
NOTE:727171.1  - Implementing Load Balancing On Oracle E-Business Suite - Documentation For Specific Load Balancer Hardware
NOTE:736752.1  - Introducing Cluster Health Monitor (IPD/OS)
NOTE:1101938.1  - Master Note for Data Guard
NOTE:1082394.1  - 11.2.0.1.X Grid Infrastructure PSU Known Issues
NOTE:868955.1  - Get Proactive - Oracle Health Checks - Installation, troubleshooting, catalog and more.
NOTE:887522.1  - Grid Infrastructure Single Client Access Name (SCAN) Explained
NOTE:92602.1  - pre-10gR1: How to Password Protect the Listener
NOTE:948456.1  - Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment
NOTE:1050693.1  - Troubleshooting 11.2 Clusterware Node Evictions (Reboots)
NOTE:1050908.1  - Troubleshoot Grid Infrastructure Startup Issues
NOTE:363254.1  - Applying one-off Oracle Clusterware patches in a mixed version home environment
NOTE:380489.1  - Using Load-Balancers with Oracle E-Business Suite Release 12
NOTE:395314.1  - RAC Hangs due to small cache size on SYS.AUDSES$
NOTE:397460.1  - Oracle's Policy for Supporting Oracle RAC with Symantec SFRAC
NOTE:388577.1  - Using Oracle 10g Release 2 Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12
NOTE:390374.1  - Oracle Performance Diagnostic Guide (OPDG)
NOTE:391771.1  - OCFS2 1.2 - FREQUENTLY ASKED QUESTIONS
NOTE:401132.1  - How to Install Oracle Clusterware with Shared Storage on Block Devices
NOTE:403347.1  - MAA Roadmap for the E-Business Suite
NOTE:785351.1  - Oracle 11gR2 Upgrade Companion
NOTE:787420.1  - Cluster Interconnect in Oracle 10g and 11gR1 RAC
NOTE:790189.1  - Oracle Clusterware and Application Failover Management
NOTE:810394.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)
NOTE:783456.1  - CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage
NOTE:428681.1  - OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)
NOTE:810663.1  - 11.1.0.X CRS Bundle Patch Information

NOTE:559365.1  - Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions
NOTE:811151.1  - How to Install Oracle Cluster Health Monitor (former IPD/OS) on Windows
NOTE:429825.1  - Complete Checklist for Manual Upgrades to 11gR1
NOTE:77483.1  - External Support FTP site: Information Sheet
NOTE:1268927.1  - RACcheck - RAC Configuration Audit Tool
NOTE:563566.1  - Troubleshooting gc block lost and Poor Network Performance in a RAC Environment
NOTE:563905.1  - Implementing LIBUMEM for CRS on Solaris 64
NOTE:601807.1  - Oracle 11gR1 Upgrade Companion
NOTE:316817.1  - Cluster Verification Utility (CLUVFY) FAQ
NOTE:316900.1  - ALERT: Oracle 10g Release 2 (10.2) Support Status and Alerts
NOTE:330358.1  - CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide
NOTE:337737.1  - Oracle Clusterware - ASM - Database Version Compatibility
NOTE:338706.1  - Oracle Clusterware (CRS or GI) Rolling Upgrades
NOTE:1187674.1  - Master Note for Oracle Database Machine and Exadata Storage Server
NOTE:1187723.1  - Master Note for Automatic Storage Management (ASM)
NOTE:403743.1  - Pre-11.2: VIP Failover Take Long Time After Network Cable Pulled
NOTE:332257.1  - Using Oracle Clusterware with Vendor Clusterware FAQ
NOTE:1065024.1  - Database 11g Release 2 Certification Highlights
NOTE:181489.1  - Tuning Inter-Instance Performance in RAC and OPS
NOTE:405820.1  - 10.2.0.X CRS Bundle Patch Information
NOTE:422893.1  - 11g Understanding Automatic Diagnostic Repository.
NOTE:1328466.1  - Cluster Health Monitor (CHM) FAQ
NOTE:443529.1  - Database 11g: Quick Steps to Package and Send Critical Error Diagnostic Information to Support [Video]
NOTE:357472.1  - Configuring device-mapper for CRS/ASM
NOTE:362135.1  - Configuring Oracle Applications Release 11i with Oracle10g Release 2 Real Application Clusters and Automatic Storage Management
NOTE:217368.1  - Advanced Configurations and Topologies for Enterprise Deployments of E-Business Suite 11i
NOTE:219361.1  - Troubleshooting ORA-29740 in a RAC Environment
NOTE:1367153.1  - Top 5 Issues That Cause Node Reboots or Evictions or Unexpected Recycle of CRS
NOTE:166650.1  - Working Effectively With Support Best Practices
NOTE:1053147.1  - 11gR2 Clusterware and Grid Home - What You Need to Know
NOTE:1452965.2  - Information Center: Oracle Scalability Grid Infrastructure / Clusterware and Real Application Clusters (RAC)
NOTE:811280.1  - RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)
NOTE:314422.1  - Remote Diagnostic Agent (RDA) - Getting Started