oracle11gr2_rac_添加节点最佳实践

  oracle11gr2_rac_添加节点最佳实践 

分类： Linux

oracle 11gr2 添加节点最佳实践
OS信息:
[grid@11grac1 ~]$ uname -a
Linux 11grac1 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:22:40 EST 2012 i686 i686 i386 GNU/Linux
DB信息:
SQL> select * from v$version where rownum<5;

BANNER
------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for Linux: Version 11.2.0.1.0 - Production

（一）设置网络
（二）添加组、目录和用户
（三）修改相关OS参数
（四）修改用户环境变量
（五）设置存储
（六）配置节点互信
（七）安装RPM包
注：1-7步略(其中的修改需要参照其他节点的配置)

（八）安装GI前检测
[grid@11grac1 bin]$ pwd
/u01/11.2.0/grid/bin
[grid@11grac1 bin]$ ./cluvfy stage -pre nodeadd -n 11grac2 -fixup -verbose

Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "11grac1"
  Destination Node                      Reachable?              
  ------------------------------------  ------------------------
  11grac2                               yes                     
Result: Node reachability check passed from node "11grac1"

Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Comment                 
  ------------------------------------  ------------------------
  11grac2                               passed                  
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       passed                                            
  11grac1       passed                                           

Verification of the hosts config file successful

Interface information for node "11grac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.161 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.163 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.200 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth1   192.168.136.161 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:C4 1500 

Interface information for node "11grac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.162 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth1   192.168.136.162 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:13 1500 

Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
Result: Node connectivity passed for interface "eth0"

Result: Node connectivity check passed

Checking CRS integrity...
The Oracle clusterware is healthy on node "11grac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
The location "/u01/11.2.0/grid" is not shared but is present/creatable on all nodes

Result: Shared resources check for node addition passed

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       passed                                            
  11grac1       passed                                           

Verification of the hosts config file successful

Interface information for node "11grac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.161 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.163 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.200 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth1   192.168.136.161 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:C4 1500 

Interface information for node "11grac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.162 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth1   192.168.136.162 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:13 1500 

Check: Node connectivity of subnet "192.168.137.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
  11grac1:eth0                    11grac2:eth0                    yes             
Result: Node connectivity passed for subnet "192.168.137.0" with node(s) 11grac1,11grac2

Check: TCP connectivity of subnet "192.168.137.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:192.168.137.161         11grac1:192.168.137.163         passed          
  11grac1:192.168.137.161         11grac1:192.168.137.200         passed          
  11grac1:192.168.137.161         11grac2:192.168.137.162         passed          
Result: TCP connectivity check passed for subnet "192.168.137.0"

Check: Node connectivity of subnet "192.168.136.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:eth1                    11grac2:eth1                    yes             
Result: Node connectivity passed for subnet "192.168.136.0" with node(s) 11grac1,11grac2

Check: TCP connectivity of subnet "192.168.136.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:192.168.136.161         11grac2:192.168.136.162         passed          
Result: TCP connectivity check passed for subnet "192.168.136.0"

Interfaces found on subnet "192.168.137.0" that are likely candidates for VIP are:
11grac1 eth0:192.168.137.161 eth0:192.168.137.163 eth0:192.168.137.200
11grac2 eth0:192.168.137.162

Interfaces found on subnet "192.168.136.0" that are likely candidates for a private interconnect are:
11grac1 eth1:192.168.136.161
11grac2 eth1:192.168.136.162

Result: Node connectivity check passed

Check: Total memory 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       1.97GB (2061864.0KB)      1.5GB (1572864.0KB)       passed    
Result: Total memory check passed

Check: Available memory 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       1.88GB (1970204.0KB)      50MB (51200.0KB)          passed    
Result: Available memory check passed

Check: Swap space 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       4GB (4192956.0KB)         2.95GB (3092796.0KB)      passed    
Result: Swap space check passed

Check: Free disk space for "11grac2:/tmp" 
  Path              Node Name     Mount point   Available     Required      Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              11grac2       /             15.54GB       1GB           passed      
Result: Free disk space check passed for "11grac2:/tmp"

Check: User existence for "grid" 
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       exists                    passed                  
Result: User existence check passed for "grid"

Check: Run level 
  Node Name     run level                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       3                         3,5                       passed    
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors" 
  Node Name         Type          Available     Required      Comment         
  ----------------  ------------  ------------  ------------  ----------------
  11grac2           hard          65536         65536         passed          
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors" 
  Node Name         Type          Available     Required      Comment         
  ----------------  ------------  ------------  ------------  ----------------
  11grac2           soft          1024          1024          passed          
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes" 
  Node Name         Type          Available     Required      Comment         
  ----------------  ------------  ------------  ------------  ----------------
  11grac2           hard          16384         16384         passed          
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes" 
  Node Name         Type          Available     Required      Comment         
  ----------------  ------------  ------------  ------------  ----------------
  11grac2           soft          2047          2047          passed          
Result: Soft limits check passed for "maximum user processes"

Check: System architecture 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       i686                      x86                       passed    
Result: System architecture check passed

Check: Kernel version 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       2.6.32-300.10.1.el5uek    2.6.18                    passed    
Result: Kernel version check passed

Check: Kernel parameter for "semmsl" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       250                       250                       passed    
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       32000                     32000                     passed    
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       100                       100                       passed    
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       128                       128                       passed    
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       1073741824                536870912                 passed    
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       4096                      4096                      passed    
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       8388608                   2097152                   passed    
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       6815744                   6815744                   passed    
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       between 9000 & 65500      between 9000 & 65500      passed    
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       262144                    262144                    passed    
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       4194304                   4194304                   passed    
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       262144                    262144                    passed    
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       1048586                   1048576                   passed    
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr" 
  Node Name     Configured                Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       1048576                   1048576                   passed    
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.81" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       make-3.81-3.el5           make-3.81                 passed    
Result: Package existence check passed for "make-3.81"

Check: Package existence for "binutils-2.17.50.0.6" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       binutils-2.17.50.0.6-20.el5  binutils-2.17.50.0.6      passed    
Result: Package existence check passed for "binutils-2.17.50.0.6"

Check: Package existence for "gcc-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       gcc-4.1.2-52.el5          gcc-4.1.2                 passed    
Result: Package existence check passed for "gcc-4.1.2"

Check: Package existence for "gcc-c++-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       gcc-c++-4.1.2-52.el5      gcc-c++-4.1.2             passed    
Result: Package existence check passed for "gcc-c++-4.1.2"

Check: Package existence for "libgomp-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libgomp-4.4.6-3.el5.1     libgomp-4.1.2             passed    
Result: Package existence check passed for "libgomp-4.1.2"

Check: Package existence for "libaio-0.3.106" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libaio-0.3.106-5          libaio-0.3.106            passed    
Result: Package existence check passed for "libaio-0.3.106"

Check: Package existence for "glibc-2.5-24" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       glibc-2.5-81              glibc-2.5-24              passed    
Result: Package existence check passed for "glibc-2.5-24"

Check: Package existence for "compat-libstdc++-33-3.2.3" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       compat-libstdc++-33-3.2.3-61  compat-libstdc++-33-3.2.3  passed    
Result: Package existence check passed for "compat-libstdc++-33-3.2.3"

Check: Package existence for "elfutils-libelf-0.125" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       elfutils-libelf-0.137-3.el5  elfutils-libelf-0.125     passed    
Result: Package existence check passed for "elfutils-libelf-0.125"

Check: Package existence for "elfutils-libelf-devel-0.125" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed    
Result: Package existence check passed for "elfutils-libelf-devel-0.125"

Check: Package existence for "glibc-common-2.5" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       glibc-common-2.5-81       glibc-common-2.5          passed    
Result: Package existence check passed for "glibc-common-2.5"

Check: Package existence for "glibc-devel-2.5" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       glibc-devel-2.5-81        glibc-devel-2.5           passed    
Result: Package existence check passed for "glibc-devel-2.5"

Check: Package existence for "glibc-headers-2.5" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       glibc-headers-2.5-81      glibc-headers-2.5         passed    
Result: Package existence check passed for "glibc-headers-2.5"

Check: Package existence for "libaio-devel-0.3.106" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libaio-devel-0.3.106-5    libaio-devel-0.3.106      passed    
Result: Package existence check passed for "libaio-devel-0.3.106"

Check: Package existence for "libgcc-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libgcc-4.1.2-52.el5       libgcc-4.1.2              passed    
Result: Package existence check passed for "libgcc-4.1.2"

Check: Package existence for "libstdc++-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libstdc++-4.1.2-52.el5    libstdc++-4.1.2           passed    
Result: Package existence check passed for "libstdc++-4.1.2"

Check: Package existence for "libstdc++-devel-4.1.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       libstdc++-devel-4.1.2-52.el5  libstdc++-devel-4.1.2     passed    
Result: Package existence check passed for "libstdc++-devel-4.1.2"

Check: Package existence for "sysstat-7.0.2" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       sysstat-7.0.2-11.el5      sysstat-7.0.2             passed    
Result: Package existence check passed for "sysstat-7.0.2"

Check: Package existence for "unixODBC-2.2.11" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       unixODBC-2.2.11-10.el5    unixODBC-2.2.11           passed    
Result: Package existence check passed for "unixODBC-2.2.11"

Check: Package existence for "unixODBC-devel-2.2.11" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       unixODBC-devel-2.2.11-10.el5  unixODBC-devel-2.2.11     passed    
Result: Package existence check passed for "unixODBC-devel-2.2.11"

Check: Package existence for "ksh-20060214" 
  Node Name     Available                 Required                  Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       ksh-20100621-5.el5        ksh-20060214              passed    
Result: Package existence check passed for "ksh-20060214"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       does not exist            passed                  
Result: User "grid" is not part of "root" group. Check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Pre-check for node addition was successful. 
[grid@11grac1 bin]$

（九）扩展新节点到集群中（期间传文件可能会花段时间）
[grid@11grac1 bin]$ pwd
/u01/11.2.0/grid/oui/bin
[grid@11grac1 bin]$ ./addNode.sh -silent "cluster_new_nodes={11grac2}" "cluster_new_virtual_hostnames={11grac2-vip}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.

Performing tests to see whether nodes 11grac2 are available
............................................................... 100% Done.

...
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/11.2.0/grid
   New Nodes
Space Requirements
   New Nodes
      11grac2
         /: Required 3.46GB : Available 14.46GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.1.0 
      Sun JDK 1.5.0.17.0 
      Installer SDK Component 11.2.0.1.0 
      Oracle One-Off Patch Installer 11.2.0.0.2 
      Oracle Universal Installer 11.2.0.1.0 
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0 
      Enterprise Manager Common Core Files 10.2.0.4.2 
      Oracle DBCA Deconfiguration 11.2.0.1.0 
      Oracle RAC Deconfiguration 11.2.0.1.0 
      Oracle Quality of Service Management (Server) 11.2.0.1.0 
      Installation Plugin Files 11.2.0.1.0 
      Universal Storage Manager Files 11.2.0.1.0 
      Oracle Text Required Support Files 11.2.0.1.0 
      Automatic Storage Management Assistant 11.2.0.1.0 
      Oracle Database 11g Multimedia Files 11.2.0.1.0 
      Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 
      Oracle Globalization Support 11.2.0.1.0 
      Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 
      Oracle Core Required Support Files 11.2.0.1.0 
      Bali Share 1.1.18.0.0 
      Oracle Database Deconfiguration 11.2.0.1.0 
      Oracle Quality of Service Management (Client) 11.2.0.1.0 
      Expat libraries 2.0.1.0.1 
      Oracle Containers for Java 11.2.0.1.0 
      Perl Modules 5.10.0.0.1 
      Secure Socket Layer 11.2.0.1.0 
      Oracle JDBC/OCI Instant Client 11.2.0.1.0 
      Oracle Multimedia Client Option 11.2.0.1.0 
      LDAP Required Support Files 11.2.0.1.0 
      Character Set Migration Utility 11.2.0.1.0 
      Perl Interpreter 5.10.0.0.1 
      PL/SQL Embedded Gateway 11.2.0.1.0 
      OLAP SQL Scripts 11.2.0.1.0 
      Database SQL Scripts 11.2.0.1.0 
      Oracle Extended Windowing Toolkit 3.4.47.0.0 
      SSL Required Support Files for InstantClient 11.2.0.1.0 
      SQL*Plus Files for Instant Client 11.2.0.1.0 
      Oracle Net Required Support Files 11.2.0.1.0 
      Oracle Database User Interface 2.2.13.0.0 
      RDBMS Required Support Files for Instant Client 11.2.0.1.0 
      Enterprise Manager Minimal Integration 11.2.0.1.0 
      XML Parser for Java 11.2.0.1.0 
      Oracle Security Developer Tools 11.2.0.1.0 
      Oracle Wallet Manager 11.2.0.1.0 
      Enterprise Manager plugin Common Files 11.2.0.1.0 
      Platform. Required Support Files 11.2.0.1.0 
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 
      RDBMS Required Support Files 11.2.0.1.0 
      Oracle Ice Browser 5.2.3.6.0 
      Oracle Help For Java 4.2.9.0.0 
      Enterprise Manager Common Files 10.2.0.4.2 
      Deinstallation Tool 11.2.0.1.0 
      Oracle Java Client 11.2.0.1.0 
      Cluster Verification Utility Files 11.2.0.1.0 
      Oracle Notification Service (eONS) 11.2.0.1.0 
      Oracle LDAP administration 11.2.0.1.0 
      Cluster Verification Utility Common Files 11.2.0.1.0 
      Oracle Clusterware RDBMS Files 11.2.0.1.0 
      Oracle Locale Builder 11.2.0.1.0 
      Oracle Globalization Support 11.2.0.1.0 
      Buildtools Common Files 11.2.0.1.0 
      Oracle RAC Required Support Files-HAS 11.2.0.1.0 
      SQL*Plus Required Support Files 11.2.0.1.0 
      XDK Required Support Files 11.2.0.1.0 
      Agent Required Support Files 10.2.0.4.2 
      Parser Generator Required Support Files 11.2.0.1.0 
      Precompiler Required Support Files 11.2.0.1.0 
      Installation Common Files 11.2.0.1.0 
      Required Support Files 11.2.0.1.0 
      Oracle JDBC/THIN Interfaces 11.2.0.1.0 
      Oracle Multimedia Locator 11.2.0.1.0 
      Oracle Multimedia 11.2.0.1.0 
      HAS Common Files 11.2.0.1.0 
      Assistant Common Files 11.2.0.1.0 
      PL/SQL 11.2.0.1.0 
      HAS Files for DB 11.2.0.1.0 
      Oracle Recovery Manager 11.2.0.1.0 
      Oracle Database Utilities 11.2.0.1.0 
      Oracle Notification Service 11.2.0.0.0 
      SQL*Plus 11.2.0.1.0 
      Oracle Netca Client 11.2.0.1.0 
      Oracle Net 11.2.0.1.0 
      Oracle JVM 11.2.0.1.0 
      Oracle Internet Directory Client 11.2.0.1.0 
      Oracle Net Listener 11.2.0.1.0 
      Cluster Ready Services Files 11.2.0.1.0 
      Oracle Database 11g 11.2.0.1.0 
-----------------------------------------------------------------------------

Instantiating scripts for add node (Wednesday, August 29, 2012 1:04:40 PM CST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Wednesday, August 29, 2012 1:04:51 PM CST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Wednesday, August 29, 2012 1:19:31 PM CST)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system. 
To register the new inventory please run the script. at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes '11grac2'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oraInventory/orainstRoot.sh #On nodes 11grac2
/u01/11.2.0/grid/root.sh #On nodes 11grac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node
    
The Cluster Node Addition of /u01/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

（十）按说明执行脚本
[root@11grac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script. is complete.
[root@11grac2 ~]# /u01/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: y
Creating y directory...
   Copying dbhome to y ...
   Copying oraenv to y ...
   Copying coraenv to y ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-08-29 13:23:07: Parsing the host name
2012-08-29 13:23:07: Checking for super user privileges
2012-08-29 13:23:07: User has super user privileges
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE 
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-5-8.0.2

 

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11grac1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on '11grac2'
CRS-2677: Stop of 'ora.cssdmonitor' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on '11grac2'
CRS-2677: Stop of 'ora.gpnpd' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '11grac2'
CRS-2677: Stop of 'ora.gipcd' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on '11grac2'
CRS-2677: Stop of 'ora.mdnsd' on '11grac2' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on '11grac2'
CRS-2676: Start of 'ora.mdnsd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on '11grac2'
CRS-2676: Start of 'ora.gipcd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on '11grac2'
CRS-2676: Start of 'ora.gpnpd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on '11grac2'
CRS-2676: Start of 'ora.cssdmonitor' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '11grac2'
CRS-2672: Attempting to start 'ora.diskmon' on '11grac2'
CRS-2676: Start of 'ora.diskmon' on '11grac2' succeeded
CRS-2676: Start of 'ora.cssd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '11grac2'
CRS-2676: Start of 'ora.ctssd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on '11grac2'
CRS-2676: Start of 'ora.asm' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '11grac2'
CRS-2676: Start of 'ora.crsd' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on '11grac2'
CRS-2676: Start of 'ora.evmd' on '11grac2' succeeded
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
PRCC-1015 : LISTENER was already running on 11grac2

start listener on node=11grac2 ... failed
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... failed
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

（十一）检查扩展结果
[grid@11grac1 bin]$ pwd
/u01/11.2.0/grid/oui/bin
[grid@11grac1 bin]$ cluvfy stage -post nodeadd -n 11grac2 -verbose

Performing post-checks for node addition

Checking node reachability...

Check: Node reachability from node "11grac1"
  Destination Node                      Reachable?              
  ------------------------------------  ------------------------
  11grac2                               yes                     
Result: Node reachability check passed from node "11grac1"

Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Comment                 
  ------------------------------------  ------------------------
  11grac2                               passed                  
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       passed                                            
  11grac1       passed                                           

Verification of the hosts config file successful

Interface information for node "11grac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.162 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth0   192.168.137.164 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth1   192.168.136.162 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:13 1500 

Interface information for node "11grac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.161 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.163 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.200 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth1   192.168.136.161 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:C4 1500 

Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac2:eth0                    11grac2:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
Result: Node connectivity passed for interface "eth0"

Result: Node connectivity check passed

Checking cluster integrity...

  Node Name                           
  ------------------------------------
  11grac1                             
  11grac2                            

Cluster integrity check passed

Checking CRS integrity...
The Oracle clusterware is healthy on node "11grac2"
The Oracle clusterware is healthy on node "11grac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
The location "/u01/11.2.0/grid" is not shared but is present/creatable on all nodes

Result: Shared resources check for node addition passed

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       passed                                            
  11grac1       passed                                           

Verification of the hosts config file successful

Interface information for node "11grac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.162 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth0   192.168.137.164 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:09 1500  
 eth1   192.168.136.162 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:D6:B6:13 1500 

Interface information for node "11grac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.137.161 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.163 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth0   192.168.137.200 192.168.137.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:BA 1500  
 eth1   192.168.136.161 192.168.136.0   0.0.0.0         192.168.137.1   00:0C:29:31:BE:C4 1500 

Check: Node connectivity of subnet "192.168.137.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac2:eth0                    11grac2:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac2:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
  11grac1:eth0                    11grac1:eth0                    yes             
Result: Node connectivity passed for subnet "192.168.137.0" with node(s) 11grac2,11grac1

Check: TCP connectivity of subnet "192.168.137.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:192.168.137.161         11grac2:192.168.137.162         passed          
  11grac1:192.168.137.161         11grac2:192.168.137.164         passed          
  11grac1:192.168.137.161         11grac1:192.168.137.163         passed          
  11grac1:192.168.137.161         11grac1:192.168.137.200         passed          
Result: TCP connectivity check passed for subnet "192.168.137.0"

Check: Node connectivity of subnet "192.168.136.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac2:eth1                    11grac1:eth1                    yes             
Result: Node connectivity passed for subnet "192.168.136.0" with node(s) 11grac2,11grac1

Check: TCP connectivity of subnet "192.168.136.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  11grac1:192.168.136.161         11grac2:192.168.136.162         passed          
Result: TCP connectivity check passed for subnet "192.168.136.0"

Interfaces found on subnet "192.168.137.0" that are likely candidates for VIP are:
11grac2 eth0:192.168.137.162 eth0:192.168.137.164
11grac1 eth0:192.168.137.161 eth0:192.168.137.163 eth0:192.168.137.200

Interfaces found on subnet "192.168.136.0" that are likely candidates for a private interconnect are:
11grac2 eth1:192.168.136.162
11grac1 eth1:192.168.136.161

Result: Node connectivity check passed

Checking node application existence...

Checking existence of VIP node application 
  Node Name     Required                  Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       yes                       online                    passed    
  11grac1       yes                       online                    passed    
Result: Check passed.

Checking existence of ONS node application 
  Node Name     Required                  Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       no                        online                    passed    
  11grac1       no                        online                    passed    
Result: Check passed.

Checking existence of GSD node application 
  Node Name     Required                  Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       no                        online                    passed    
  11grac1       no                        online                    passed    
Result: Check passed.

Checking existence of EONS node application 
  Node Name     Required                  Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       no                        online                    passed    
  11grac1       no                        online                    passed    
Result: Check passed.

Checking existence of NETWORK node application 
  Node Name     Required                  Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  11grac2       no                        online                    passed    
  11grac1       no                        online                    passed    
Result: Check passed.

Checking Single Client Access Name (SCAN)...
  SCAN VIP name     Node          Running?      ListenerName  Port          Running?    
  ----------------  ------------  ------------  ------------  ------------  ------------
  racscan           11grac1       true          LISTENER      1521          true       

Checking name resolution setup for "racscan"...

ERROR: 
PRVF-4664 : Found inconsistent name resolution entries for SCAN name "racscan"
  SCAN Name     IP Address                Status                    Comment   
  ------------  ------------------------  ------------------------  ----------
  racscan       192.168.137.200           failed                    NIS Entry

ERROR: 
PRVF-4657 : Name resolution setup check for "racscan" (IP address: 192.168.137.200) failed

ERROR: 
PRVF-4664 : Found inconsistent name resolution entries for SCAN name "racscan"

Verification of SCAN VIP and Listener setup failed

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  11grac2       does not exist            passed                  
Result: User "grid" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status                  
  ------------------------------------  ------------------------
  11grac2                               passed                  
Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State                   
  ------------------------------------  ------------------------
  11grac2                               Active                  
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status                  
  ------------  ------------------------  ------------------------
  11grac2       0.0                       passed                 

Time offset is within the specified limits on the following set of nodes: 
"[11grac2]" 
Result: Check of clock time offsets passed

Oracle Cluster Time Synchronization Services check passed

Post-check for node addition was unsuccessful on all the nodes. 
[grid@11grac1 bin]$

（十二）扩展oracle数据库（期间传文件可能会花段时间）
[oracle@11grac1 bin]$ pwd
/u01/app/oracle/product/11.2.0/db_1/oui/bin
[oracle@11grac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={11grac2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4056 MB    Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.

Performing tests to see whether nodes 11grac2 are available
............................................................... 100% Done.

..
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0/db_1
   New Nodes
Space Requirements
   New Nodes
      11grac2
         /: Required 3.74GB : Available 11.27GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.1.0 
      Sun JDK 1.5.0.17.0 
      Installer SDK Component 11.2.0.1.0 
      Oracle One-Off Patch Installer 11.2.0.0.2 
      Oracle Universal Installer 11.2.0.1.0 
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0 
      Oracle DBCA Deconfiguration 11.2.0.1.0 
      Oracle RAC Deconfiguration 11.2.0.1.0 
      Oracle Database Deconfiguration 11.2.0.1.0 
      Oracle Configuration Manager 10.3.1.1.0 
      Oracle ODBC Driverfor Instant Client 11.2.0.1.0 
      LDAP Required Support Files 11.2.0.1.0 
      SSL Required Support Files for InstantClient 11.2.0.1.0 
      Bali Share 1.1.18.0.0 
      Oracle Extended Windowing Toolkit 3.4.47.0.0 
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 
      Oracle Real Application Testing 11.2.0.1.0 
      Oracle Database Vault J2EE Application 11.2.0.1.0 
      Oracle Label Security 11.2.0.1.0 
      Oracle Data Mining RDBMS Files 11.2.0.1.0 
      Oracle OLAP RDBMS Files 11.2.0.1.0 
      Oracle OLAP API 11.2.0.1.0 
      Platform. Required Support Files 11.2.0.1.0 
      Oracle Database Vault option 11.2.0.1.0 
      Oracle RAC Required Support Files-HAS 11.2.0.1.0 
      SQL*Plus Required Support Files 11.2.0.1.0 
      Oracle Display Fonts 9.0.2.0.0 
      Oracle Ice Browser 5.2.3.6.0 
      Oracle JDBC Server Support Package 11.2.0.1.0 
      Oracle SQL Developer 11.2.0.1.0 
      Oracle Application Express 11.2.0.1.0 
      XDK Required Support Files 11.2.0.1.0 
      RDBMS Required Support Files for Instant Client 11.2.0.1.0 
      SQLJ Runtime 11.2.0.1.0 
      Database Workspace Manager 11.2.0.1.0 
      RDBMS Required Support Files 11.2.0.1.0 
      Oracle Globalization Support 11.2.0.1.0 
      Exadata Storage Server 11.2.0.1.0 
      Provisioning Advisor Framework 10.2.0.4.2 
      Enterprise Manager Database Plugin -- Repository Support 11.2.0.1.0 
      Enterprise Manager Repository Core Files 10.2.0.4.2 
      Enterprise Manager Database Plugin -- Agent Support 11.2.0.1.0 
      Enterprise Manager Grid Control Core Files 10.2.0.4.2 
      Enterprise Manager Common Core Files 10.2.0.4.2 
      Enterprise Manager Agent Core Files 10.2.0.4.2 
      Agent Required Support Files 10.2.0.4.2 
      regexp 2.1.9.0.0 
      Parser Generator Required Support Files 11.2.0.1.0 
      Oracle 11g Warehouse Builder Required Files 11.2.0.1.0 
      Oracle Notification Service (eONS) 11.2.0.1.0 
      Oracle Text Required Support Files 11.2.0.1.0 
      Precompiler Required Support Files 11.2.0.1.0 
      Oracle Database 11g Multimedia Files 11.2.0.1.0 
      Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 
      Oracle Multimedia Annotator 11.2.0.1.0 
      Oracle JDBC/OCI Instant Client 11.2.0.1.0 
      Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 
      Oracle Core Required Support Files 11.2.0.1.0 
      Oracle Help For Java 4.2.9.0.0 
      Sample Schema Data 11.2.0.1.0 
      Oracle Starter Database 11.2.0.1.0 
      Oracle Message Gateway Common Files 11.2.0.1.0 
      Oracle XML Query 11.2.0.1.0 
      XML Parser for Oracle JVM 11.2.0.1.0 
      Expat libraries 2.0.1.0.1 
      Installation Plugin Files 11.2.0.1.0 
      Enterprise Manager Common Files 10.2.0.4.2 
      Perl Modules 5.10.0.0.1 
      Deinstallation Tool 11.2.0.1.0 
      Oracle Quality of Service Management (Client) 11.2.0.1.0 
      Perl Interpreter 5.10.0.0.1 
      JAccelerator (COMPANION) 11.2.0.1.0 
      Oracle Containers for Java 11.2.0.1.0 
      Oracle Code Editor 1.2.1.0.0I 
      Oracle Net Required Support Files 11.2.0.1.0 
      Secure Socket Layer 11.2.0.1.0 
      Oracle Universal Connection Pool 11.2.0.1.0 
      Oracle JDBC/THIN Interfaces 11.2.0.1.0 
      Oracle Multimedia Client Option 11.2.0.1.0 
      Oracle Java Client 11.2.0.1.0 
      Character Set Migration Utility 11.2.0.1.0 
      Oracle Locale Builder 11.2.0.1.0 
      PL/SQL Embedded Gateway 11.2.0.1.0 
      OLAP SQL Scripts 11.2.0.1.0 
      Database SQL Scripts 11.2.0.1.0 
      Oracle Globalization Support 11.2.0.1.0 
      Required Support Files 11.2.0.1.0 
      SQL*Plus Files for Instant Client 11.2.0.1.0 
      Oracle ODBC Driver 11.2.0.1.0 
      Oracle Database User Interface 2.2.13.0.0 
      Oracle Notification Service 11.2.0.0.0 
      Enterprise Manager Minimal Integration 11.2.0.1.0 
      XML Parser for Java 11.2.0.1.0 
      Oracle Security Developer Tools 11.2.0.1.0 
      Oracle Wallet Manager 11.2.0.1.0 
      Cluster Verification Utility Common Files 11.2.0.1.0 
      Oracle Clusterware RDBMS Files 11.2.0.1.0 
      Oracle UIX 2.2.24.5.0 
      Enterprise Manager plugin Common Files 11.2.0.1.0 
      HAS Common Files 11.2.0.1.0 
      Precompiler Common Files 11.2.0.1.0 
      Installation Common Files 11.2.0.1.0 
      Oracle Help for the  Web 2.0.14.0.0 
      Oracle LDAP administration 11.2.0.1.0 
      Buildtools Common Files 11.2.0.1.0 
      Assistant Common Files 11.2.0.1.0 
      Oracle Recovery Manager 11.2.0.1.0 
      PL/SQL 11.2.0.1.0 
      Generic Connectivity Common Files 11.2.0.1.0 
      Oracle Database Gateway for ODBC 11.2.0.1.0 
      Oracle Programmer 11.2.0.1.0 
      Oracle Database Utilities 11.2.0.1.0 
      Enterprise Manager Agent 10.2.0.4.2 
      Oracle Netca Client 11.2.0.1.0 
      SQL*Plus 11.2.0.1.0 
      Oracle Call Interface (OCI) 11.2.0.1.0 
      Oracle Multimedia Locator 11.2.0.1.0 
      Oracle Multimedia 11.2.0.1.0 
      Oracle Net 11.2.0.1.0 
      Database Configuration and Upgrade Assistants 11.2.0.1.0 
      Oracle XML Development Kit 11.2.0.1.0 
      Oracle JVM 11.2.0.1.0 
      Oracle Advanced Security 11.2.0.1.0 
      Oracle Internet Directory Client 11.2.0.1.0 
      HAS Files for DB 11.2.0.1.0 
      Oracle Enterprise Manager Console DB 11.2.0.1.0 
      Oracle Net Listener 11.2.0.1.0 
      Oracle Text 11.2.0.1.0 
      Oracle Net Services 11.2.0.1.0 
      Oracle Database 11g 11.2.0.1.0 
      Oracle OLAP 11.2.0.1.0 
      Oracle Spatial 11.2.0.1.0 
      Oracle Partitioning 11.2.0.1.0 
      Enterprise Edition Options 11.2.0.1.0 
-----------------------------------------------------------------------------

Instantiating scripts for add node (Wednesday, August 29, 2012 1:35:23 PM CST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Wednesday, August 29, 2012 1:35:42 PM CST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Wednesday, August 29, 2012 2:44:50 PM CST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes 11grac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node
    
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@11grac1 bin]$

（十三）按提示执行脚本
[root@11grac2 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@11grac2 ~]#

（十四）创建数据库实例
可以使用dbca进行，也可用下边的命令进行
[oracle@11grac1 dbs]$ export DISPLAY=192.168.137.1:0.0
[oracle@11grac1 dbs]$ dbca

[oracle@11grac1 bin]$ pwd
/u01/app/oracle/product/11.2.0/db_1/bin
[oracle@11grac1 bin]$ dbca -silent -addInstance -nodeList 11grac2 -gdbName racdb -instanceName racdb2 -sysDBAUserName sys -sysDBAPassword "oracle"
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/racdb.log" for further details.
[oracle@11grac1 bin]$

（十五）验证
[grid@11grac2 ~]$ crsctl check cluster -all
**************************************************************
11grac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
11grac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@11grac2 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    11grac1     
ora....C1.lsnr application    ONLINE    ONLINE    11grac1     
ora....ac1.gsd application    ONLINE    ONLINE    11grac1     
ora....ac1.ons application    ONLINE    ONLINE    11grac1     
ora....ac1.vip ora....t1.type ONLINE    ONLINE    11grac1     
ora....SM2.asm application    ONLINE    ONLINE    11grac2     
ora....C2.lsnr application    ONLINE    ONLINE    11grac2     
ora....ac2.gsd application    ONLINE    ONLINE    11grac2     
ora....ac2.ons application    ONLINE    ONLINE    11grac2     
ora....ac2.vip ora....t1.type ONLINE    ONLINE    11grac2     
ora.DATA.dg    ora....up.type ONLINE    ONLINE    11grac1     
ora....ER.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora....N1.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora.asm        ora.asm.type   ONLINE    ONLINE    11grac1     
ora.eons       ora.eons.type  ONLINE    ONLINE    11grac1     
ora.gsd        ora.gsd.type   ONLINE    ONLINE    11grac1     
ora....network ora....rk.type ONLINE    ONLINE    11grac1     
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    11grac1     
ora.ons        ora.ons.type   ONLINE    ONLINE    11grac1     
ora.racdb.db   ora....se.type ONLINE    ONLINE    11grac1     
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    11grac1     
[grid@11grac2 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node 11grac1
Instance racdb2 is running on node 11grac2
[oracle@11grac2 ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 29 15:30:52 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

SQL> conn /as sysdba
Connected.
SQL> select * from test;

        ID NAME
---------- ----------
         1 yallonking

至此，ok！