oracle vm上安装oracle11g r2 rac（11.2.0.4）

oracle 11g R2 rac安装（11.2.0.4）

环境：
2个节点rac部署
hostname:rac01  
操作系统：CentOS release 6.5 (Final)  
内存：1837496 kB
磁盘：30G
hostname:rac02  
操作系统：CentOS release 6.5 (Final)  
内存：1837496 kB
磁盘：3G
数据库软件：
p13390677_112040_Linux-x86-64_1of7
p13390677_112040_Linux-x86-64_2of7
p13390677_112040_Linux-x86-64_3of7

安装虚拟机安装操作系统(省略)

1.主机环境检查
内存
[root@rac01 ~]# grep MemTotal /proc/meminfo
MemTotal:        1837496 kB
swap
[root@rac01 ~]# grep SwapTotal /proc/meminfo
SwapTotal:       3071992 kB
oracle要求swap大小：
Between 2.5 GB and 32 GB Equal to the size of RAM
More than 32 GB           32 GB of RAM
磁盘
[root@rac01 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        20G  2.9G   16G  16% /
tmpfs           898M  300K  897M   1% /dev/shm
/dev/sda1       194M   30M  155M  16% /boot
/dev/sda3       5.8G  140M  5.4G   3% /home
/dev/sda6       2.0G   35M  1.8G   2% /tmp
[root@rac01 ~]# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       2.0G   35M  1.8G   2% /tmp
2.关闭防火墙
[root@rac01 ~]# service iptables status
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 
Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 
Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
[root@rac01 ~]#  service iptables stop 
[root@rac01 ~]# chkconfig iptables off
[root@rac01 ~]#  service iptables status
3.关闭selinux
查看selinux状态
[root@rac01 ~]#  /usr/sbin/sestatus -v
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted
Process contexts:
Current context:                unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Init context:                   system_u:system_r:init_t:s0
/sbin/mingetty                  system_u:system_r:getty_t:s0
/usr/sbin/sshd                  system_u:system_r:sshd_t:s0-s0:c0.c1023
File contexts:
Controlling term:               unconfined_u:object_r:user_devpts_t:s0
/etc/passwd                     system_u:object_r:etc_t:s0
/etc/shadow                     system_u:object_r:shadow_t:s0
/bin/bash                       system_u:object_r:shell_exec_t:s0
/bin/login                      system_u:object_r:login_exec_t:s0
/bin/sh                         system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
/sbin/agetty                    system_u:object_r:getty_exec_t:s0
/sbin/init                      system_u:object_r:init_exec_t:s0
/sbin/mingetty                  system_u:object_r:getty_exec_t:s0
/usr/sbin/sshd                  system_u:object_r:sshd_exec_t:s0
查看selinux的模式
[root@rac01 ~]# getenforce
Enforcing
[root@rac01 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 
关闭selinux
[root@rac01 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@rac01 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 
临时性关闭
[root@rac01 ~]# setenforce 0


4.网络ip规划：
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.38.1.182     rac01
10.38.1.85      rac02
#private ip
192.168.56.102  rac01-priv
192.168.56.101  rac02-priv
#public ip
10.38.1.56      rac01-vip
10.38.1.59      rac02-vip
#scan ip
10.38.1.188     scanip

5.创建用户（oracle，grid）创建用户组（oinstall,asmadmin,asmdba,asmoper）以及设置用户环境变量：
#!/bin/bash
#Purpose:Create 6 groups named 'oinstall','dba','asmadmin','asmdba','asmoper','oper', plus 2 users named 'oracle','grid'.
#Also setting the Environment
#variable for oracle user.
#variable for grid user.
#Usage:Log on as the superuser('root'),and then execute the command:#./1preusers.sh
#Author:Asher Huang
echo "Now create 6 groups named 'oinstall','dba','asmadmin','asmdba','asmoper','oper'"
echo "Plus 2 users named 'oracle','grid',Also setting the Environment"
groupadd -g 1000 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "grid Infrastructure Owner" grid
echo "grid" | passwd --stdin grid
echo 'export PS1="`/bin/hostname -s`-> "'>> /home/grid/.bash_profile
echo "export TMP=/tmp">> /home/grid/.bash_profile
echo 'export TMPDIR=$TMP'>>/home/grid/.bash_profile
echo "export ORACLE_SID=+ASM1">> /home/grid/.bash_profile
echo "export ORACLE_BASE=/u01/app/grid">> /home/grid/.bash_profile
echo "export ORACLE_HOME=/u01/app/11.2.0/grid">> /home/grid/.bash_profile
echo "export NLS_DATE_FORMAT='yyyy/mm/dd hh24:mi:ss'" >> /home/grid/.bash_profile
echo 'export TNS_ADMIN=$ORACLE_HOME/network/admin' >> /home/grid/.bash_profile
echo 'export PATH=/usr/sbin:$PATH'>> /home/grid/.bash_profile
echo 'export PATH=$ORACLE_HOME/bin:$PATH'>> /home/grid/.bash_profile
echo 'export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib'>> /home/grid/.bash_profile
echo 'export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib'>> /home/grid/.bash_profile
echo "export EDITOR=vi" >> /home/grid/.bash_profile
echo "export LANG=en_US" >> /home/grid/.bash_profile
echo "export NLS_LANG=american_america.AL32UTF8" >> /home/grid/.bash_profile
echo "umask 022">> /home/grid/.bash_profile
groupadd -g 1300 dba
groupadd -g 1301 oper
useradd -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
echo "oracle" | passwd --stdin oracle
echo 'export PS1="`/bin/hostname -s`-> "'>> /home/oracle/.bash_profile
echo "export TMP=/tmp">> /home/oracle/.bash_profile
echo 'export TMPDIR=$TMP'>>/home/oracle/.bash_profile
echo "export ORACLE_HOSTNAME=node2">> /home/oracle/.bash_profile
echo "export ORACLE_SID=rac01">> /home/oracle/.bash_profile
echo "export ORACLE_BASE=/u01/app/oracle">> /home/oracle/.bash_profile
echo 'export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1'>> /home/oracle/.bash_profile
echo 'export TNS_ADMIN=$ORACLE_HOME/network/admin' >> /home/oracle/.bash_profile
echo 'export PATH=/usr/sbin:$PATH'>> /home/oracle/.bash_profile
echo 'export PATH=$ORACLE_HOME/bin:$PATH'>> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib'>> /home/oracle/.bash_profile
echo 'export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib' >> /home/oracle/.bash_profile
echo "export EDITOR=vi" >> /home/oracle/.bash_profile
echo "export LANG=en_US" >> /home/oracle/.bash_profile
echo "export NLS_LANG=american_america.AL32UTF8" >> /home/oracle/.bash_profile
echo "export NLS_DATE_FORMAT='yyyy/mm/dd hh24:mi:ss'" >> /home/oracle/.bash_profile
echo "umask 022">> /home/oracle/.bash_profile
echo "The Groups and users has been created"
echo "The Environment for grid,oracle also has been set successfully"

6.创建目录grid_base,grid_home,oracle_base
#!/bin/bash
#Purpose:Create the necessary directory for oracle,grid users and change the authention to oracle,grid users.
#Usage:Log on as the superuser('root'),and then execute the command:#./2predir.sh
#Author:Asher Huang
echo "Now create the necessary directory for oracle,grid users and change the authention to oracle,grid users..."
mkdir -p /u01/app/grid
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01
chown -R grid:oinstall /u01/app/grid
chown -R grid:oinstall /u01/app/11.2.0
chmod -R 775 /u01
echo "The necessary directory for oracle,grid users and change theauthention to oracle,grid users has been finished"

7.调整资源限制limts.conf
#!/bin/bash
#Purpose:Change the /etc/security/limits.conf.
#Usage:Log on as the superuser('root'),and then execute the command:#./3prelimits.sh
#Author:Asher Huang
echo "Now modify the /etc/security/limits.conf,but backup it named /etc/security/limits.conf.bak before"
cp /etc/security/limits.conf /etc/security/limits.conf.bak
echo "oracle soft nproc 2047" >>/etc/security/limits.conf
echo "oracle hard nproc 16384" >>/etc/security/limits.conf
echo "oracle soft nofile 1024" >>/etc/security/limits.conf
echo "oracle hard nofile 65536" >>/etc/security/limits.conf
echo "grid soft nproc 2047" >>/etc/security/limits.conf
echo "grid hard nproc 16384" >>/etc/security/limits.conf
echo "grid soft nofile 1024" >>/etc/security/limits.conf
echo "grid hard nofile 65536" >>/etc/security/limits.conf
echo "Modifing the /etc/security/limits.conf has been succeed."

8.设置/etc/pam.d/login.
#!/bin/bash
#Purpose:Modify the /etc/pam.d/login.
#Usage:Log on as the superuser('root'),and then execute the command:#./4prelimits.sh
#Author:Asher Huang
echo "Now modify the /etc/pam.d/login,but with a backup named /etc/pam.d/login.bak"
cp /etc/pam.d/login /etc/pam.d/login.bak
echo "session required /lib/security/pam_limits.so" >>/etc/pam.d/login
echo "session required pam_limits.so" >>/etc/pam.d/login
echo "Modifing the /etc/pam.d/login has been succeed."

9.设置/etc/profile
#!/bin/bash
#Purpose:Modify the /etc/profile.
#Usage:Log on as the superuser('root'),and then execute the command:#./5preprofile.sh
#Author:Asher Huang
echo "Now modify the /etc/profile,but with a backup named /etc/profile.bak"
cp /etc/profile /etc/profile.bak
echo 'if [ $USER = "oracle" ]||[ $USER = "grid" ]; then' >> /etc/profile
echo 'if [ $SHELL = "/bin/ksh" ]; then' >> /etc/profile
echo 'ulimit -p 16384' >> /etc/profile
echo 'ulimit -n 65536' >> /etc/profile
echo 'else' >> /etc/profile
echo 'ulimit -u 16384 -n 65536' >> /etc/profile
echo 'fi' >> /etc/profile
echo 'fi' >> /etc/profile
echo "Modifing the /etc/profile has been succeed."

10.设置内核参数：
#!/bin/bash
#Purpose:Modify the /etc/sysctl.conf.
#Usage:Log on as the superuser('root'),and then execute the command:#./6presysctl.sh
#Author:Asher Huang
echo "Now modify the /etc/sysctl.conf,but with a backup named /etc/sysctl.bak"
cp /etc/sysctl.conf /etc/sysctl.conf.bak
echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf
echo "fs.file-max = 6815744" >> /etc/sysctl.conf
echo "kernel.shmall = 2097152" >> /etc/sysctl.conf
echo "kernel.shmmax = 1054472192" >> /etc/sysctl.conf
echo "kernel.shmmni = 4096" >> /etc/sysctl.conf
echo "kernel.sem = 250 32000 100 128" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 9000 65500" >> /etc/sysctl.conf
echo "net.core.rmem_default = 262144" >> /etc/sysctl.conf
echo "net.core.rmem_max = 4194304" >> /etc/sysctl.conf
echo "net.core.wmem_default = 262144" >> /etc/sysctl.conf
echo "net.core.wmem_max = 1048586" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem = 262144 262144 262144" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 4194304 4194304 4194304" >> /etc/sysctl.conf
echo "Modifing the /etc/sysctl.conf has been succeed."
echo "Now make the changes take effect....."
sysctl –p

11.安装操作系统相应包
Linux x86-64 Oracle Grid Infrastructure and Oracle RAC Package Requirements
The following packages (or later versions) must be installed:
binutils-2.20.51.0.2-5.11.el6 (x86_64)
compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6.i686
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6.i686
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6.i686
libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6.i686
libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6.i686
make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)

Checking the Software Requirements
[root@rac01 home]# cat /proc/version
Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22 03:15:09 UTC 2013
[root@rac01 home]# uname -r
2.6.32-431.el6.x86_64
[root@rac01 home]# rpm -q binutils compat-libstdc++ elfutils gcc glibc libaio libgcc libstdc++ compat-libcap1-1 gcc-4 gcc-c++  glibc-devel libstdc++-devel libaio-devel make sysstat unixodbc
binutils-2.20.51.0.2-5.36.el6.x86_64
package compat-libstdc++ is not installed
elfutils-0.152-1.el6.x86_64
package gcc is not installed
glibc-2.12-1.132.el6.x86_64
libaio-0.3.107-10.el6.x86_64
libgcc-4.4.7-4.el6.x86_64
libstdc++-4.4.7-4.el6.x86_64
package compat-libcap1-1 is not installed
package gcc-4 is not installed
package gcc-c++ is not installed
glibc-devel-2.12-1.132.el6.x86_64
package libstdc++-devel is not installed
package libaio-devel is not installed
make-3.81-20.el6.x86_64
sysstat-9.0.4-22.el6.x86_64
package unixodbc is not installed
[root@rac01 home]#

12.克隆虚拟机 第二个rac节点
Oracle VM VirtualBOX下克隆虚拟机镜像
Windows环境下在Oracle VM VirtualBOX下克隆虚拟机镜像：
1.通过命令操作
VBoxManage.exe clonevdi F:\mysqlvm\mysqledi.vdi F:\mysqlvm\mysqlwms.vdi
F:\Program Files\Oracle\VirtualBox>VBoxManage.exe clonevdi F:\mysqlvm\mysqledi.v
di F:\mysqlvm\mysqlwms.vdi
VBoxManage.exe: error: Failed to lock source media 'F:\mysqlvm\mysqledi.vdi'
VBoxManage.exe: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), c
omponent MediumWrap, interface IMedium, callee IUnknown
VBoxManage.exe: error: Context: "CloneTo(pDstMedium, ComSafeArrayAsInParam(l_var
iants), NULL, pProgress.asOutParam())" at line 950 of file VBoxManageDisk.cpp
上述报错是因为被克隆的虚拟机没有停止
停止虚拟机再次执行克隆命令
F:\Program Files\Oracle\VirtualBox>VBoxManage.exe clonevdi F:\mysqlvm\mysqledi.v
di F:\mysqlvm\mysqlwms.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 4b3399f7-9466-4d42-9314-2654db46c833
2.在VirtualBOX新建虚拟机（导入克隆.vdi文件）：
“新建虚拟机”->“虚拟硬盘”->“使用已有的虚拟硬盘文件(U)”,选择：名称

修改新克隆出来的系统的hostname 为rac02
[root@rac01 ~]# 
[root@rac01 ~]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=rac02
"/etc/sysconfig/network" 2L, 30C written
[root@rac01 ~]# hostname rac02
[root@rac01 ~]#
修改克隆虚拟机的网卡名字
[root@rac02 ~]# ifconfig
eth2      Link encap:Ethernet  HWaddr 08:00:27:E6:71:80  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fee6:7180/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2660 (2.5 KiB)  TX bytes:4644 (4.5 KiB)

eth3      Link encap:Ethernet  HWaddr 08:00:27:5B:E9:D5  
          inet addr:10.38.1.85  Bcast:10.38.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe5b:e9d5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2242 errors:0 dropped:0 overruns:0 frame:0
          TX packets:145 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:202350 (197.6 KiB)  TX bytes:20123 (19.6 KiB)


[root@rac02 ~]# vi /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100e (e1000)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:30:97:a1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100e (e1000)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:e7:9d:4a", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:e6:71:80", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:5b:e9:d5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
~

修改完成后，重启系统reboot

[root@rac02 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:5B:E9:D5  
          inet addr:10.38.1.85  Bcast:11.255.255.255  Mask:254.0.0.0
          inet6 addr: fe80::a00:27ff:fe5b:e9d5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9631 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1225 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:736935 (719.6 KiB)  TX bytes:86099 (84.0 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:E6:71:80  
          inet addr:10.0.2.16  Bcast:11.255.255.255  Mask:254.0.0.0
          inet6 addr: fe80::a00:27ff:fee6:7180/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:816 (816.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:114 errors:0 dropped:0 overruns:0 frame:0
          TX packets:114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9974 (9.7 KiB)  TX bytes:9974 (9.7 KiB)

13.修改配置rac01,rac02两台机器的hosts文件
rac01-> cat /etc/hosts
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.38.1.182     rac01
10.38.1.85      rac02
#private ip
192.168.56.102  rac01-priv
192.168.56.101  rac02-priv
#public ip
10.38.1.56      rac01-vip
10.38.1.59      rac02-vip
#scan ip
10.38.1.188     scanip

rac02-> cat /etc/hosts
#127.0.0.1      lcalhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 localhost.localdomain localhost
10.38.1.182     rac01
10.38.1.85      rac02
#private ip
192.168.56.102       rac01-priv
192.168.56.101       rac02-priv
#public ip
10.38.1.56     rac01-vip
10.38.1.59     rac02-vip
#scan ip
10.38.1.188     scanip

14.虚拟机添加共享磁盘
VBoxManage createhd --filename E:\Oracleracvm\share.vdi --size 30000 --format VDI --variant Fixed
VBoxManage createhd --filename E:\Oracleracvm\share01.vdi --size 1000 --format VDI --variant Fixed
VBoxManage createhd --filename E:\Oracleracvm\share02.vdi --size 2000 --format VDI --variant Fixed
添加共享磁盘到rac01,rac02节点
参加：
http://blog.csdn.net/xcl168/article/details/17539663
https://jingyan.baidu.com/article/0a52e3f4154b11bf62ed72ad.html --图形界面添加共享磁盘

15.UDEV绑定共享存储
[root@rac02]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdb
1ATA_VBOX_HARDDISK_VB931df84b-31600e36
[root@rac02]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdc
1ATA_VBOX_HARDDISK_VBc84e4363-ce0b3691
[root@rac02]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdd
1ATA_VBOX_HARDDISK_VB0c618945-fab402cb
[root@rac02 ~]# cd /etc/udev/rules.d/
[root@rac02 rules.d]# ls
60-fprint-autosuspend.rules  70-persistent-cd.rules   90-hal.rules
60-pcmcia.rules              70-persistent-net.rules  98-kexec.rules
60-raw.rules                 90-alsa.rules            99-fuse.rules
[root@rac02 rules.d]# vi 99-oracle-asmdevices.rules
KERNEL=="sdb", BUS=="scsi", PROGRAM=="/sbin/scsi_id -gud /dev/$name" RESULT=="1ATA_VBOX_HARDDISK_VB931df84b-31600e36", SYMLINK+="asmdisk/asm-data",  OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdc", BUS=="scsi", PROGRAM=="/sbin/scsi_id -gud /dev/$name" RESULT=="1ATA_VBOX_HARDDISK_VBc84e4363-ce0b3691", SYMLINK+="asmdisk/asm-crs",  OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdd", BUS=="scsi", PROGRAM=="/sbin/scsi_id -gud /dev/$name" RESULT=="1ATA_VBOX_HARDDISK_VB0c618945-fab402cb", SYMLINK+="asmdisk/asm-voting",  OWNER="grid", GROUP="asmadmin", MODE="0660"

重启UDEV:
[root@cl6-11gr2-rac2 ~]# /sbin/start_udev 
查看绑定后的共享存储:
[root@cl6-11gr2-rac2 ~]# ls -ltr /dev/asm*
brw-rw----. 1 grid asmadmin 8, 32 8月  17 15:27 /dev/asm-diskc
brw-rw----. 1 grid asmadmin 8, 16 8月  17 15:27 /dev/asm-diskb
brw-rw----. 1 grid asmadmin 8, 48 8月  17 15:27 /dev/asm-diskd
参考：
https://www.cnblogs.com/lanston/p/3917936.html
http://blog.sina.com.cn/s/blog_8317516b0101agp5.html

16.安装grid
解压p13390677_112040_Linux-x86-64_3of7
以grid用户登录执行./runInstaller,启动安装界面
（用xmanager中xstart来设置启动安装界面）参见 http://blog.csdn.net/weiwangsisoftstone/article/details/51782739
图形界面一步一步安装最后执行脚本
[root@rac01 tmp]# /u01/app/11.2.0/grid/root.sh 
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
[root@rac01 tmp]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac01'
CRS-2676: Start of 'ora.mdnsd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac01'
CRS-2676: Start of 'ora.gpnpd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac01'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac01'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac01'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac01'
CRS-2676: Start of 'ora.diskmon' on 'rac01' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac01' succeeded
宸叉寤哄苟邋?ASM?
宸叉寤虹ｇ缁
            OCR?
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 29b6c5afb5064fb8bfbec20c120aa02b.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   29b6c5afb5064fb8bfbec20c120aa02b (/dev/asmdisk/asm-crs) [OCR]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac01'
CRS-2676: Start of 'ora.OCR.dg' on 'rac01' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac01 tmp]# 

17.安装oracle
解压
p13390677_112040_Linux-x86-64_1of7
p13390677_112040_Linux-x86-64_2of7
以oracle用户登录执行./runInstaller
最后安装执行脚本root.sh

18.创建数据库
以oracle用户登录到任何一个节点上执行dbca创建数据库

至此按照完毕
rac资源状态
[root@rac01 oraInventory]# su - grid
rac01-> crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac01                                        
               ONLINE  ONLINE       rac02                                        
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac01                                        
               ONLINE  ONLINE       rac02                                        
ora.OCR.dg
               ONLINE  ONLINE       rac01                                        
               ONLINE  ONLINE       rac02                                        
ora.asm
               ONLINE  ONLINE       rac01                    Started             
               ONLINE  ONLINE       rac02                    Started             
ora.gsd
               OFFLINE OFFLINE      rac01                                        
               OFFLINE OFFLINE      rac02                                        
ora.net1.network
               ONLINE  ONLINE       rac01                                        
               ONLINE  ONLINE       rac02                                        
ora.ons
               ONLINE  ONLINE       rac01                                        
               ONLINE  ONLINE       rac02                                        
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac01                                        
ora.cvu
      1        ONLINE  ONLINE       rac01                                        
ora.oc4j
      1        ONLINE  ONLINE       rac01                                        
ora.rac01.vip
      1        ONLINE  ONLINE       rac01                                        
ora.rac02.vip
      1        ONLINE  ONLINE       rac02                                        
ora.rac11g.db
      1        ONLINE  ONLINE       rac01                    Open                
      2        ONLINE  ONLINE       rac02                    Open                
ora.scan1.vip
      1        ONLINE  ONLINE       rac01                                        
rac01-> 


安装遇到问题以及解决：

执行/u01/app/11.2.0/grid/root.sh 脚本报错
[root@rac01 tmp]# /u01/app/11.2.0/grid/root.sh 
Performing root user operation for Oracle 11g 
The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
Failed to create keys in the OLR, rc = 127, Message:
  /u01/app/11.2.0/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory 

Failed to create keys in the OLR at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 7660.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
Failed to create keys in the OLR, rc = 127, Message:
/opt/app/11.2.0/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory

执行如下操作，就可以在此执行root.sh了
[root@rac1 ~]# cd /lib64
[root@rac1 lib64]# ls libcap*
libcap-ng.so.0 libcap-ng.so.0.0.0 libcap.so.2 libcap.so.2.16
[root@rac1 lib64]# ln -s libcap.so.2.16 libcap.so.1
[root@rac2 lib64]# ln -s libcap.so.2.16 libcap.so.1

2.节点的vip被占用导致节点vip资源offline无法启动

修改节点hosts文件和集群vip
rac02->  srvctl config vip -n rac02
VIP exists: /rac02-vip/10.38.1.57/10.38.1.0/255.255.255.0/eth0, hosting node rac02
rac02->  srvctl config vip -n rac01
VIP exists: /rac01-vip/10.38.1.56/10.38.1.0/255.255.255.0/eth0, hosting node rac01
srvctl modify nodeapps -n rac02 -A 10.38.1.59/255.255.255.0/eth0
重启vip资源问题解决

3.ONS服务offline 启动报错
rac02-> onsctl start
CRS-2672: Attempting to start 'ora.ons' on 'rac02'
CRS-5016: Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/rac02/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-2674: Start of 'ora.ons' on 'rac02' failed
CRS-4000: Command Start failed, or completed with errors.


解决方法：
CRS-2799: Failed to start or stop resource 'ora.ons' on 'node1' due to incorrect hosts configuration (CRS-2674: Start of 'ora.ons' on '<host>' failed) (文档 ID 1926115.1) 转到底部转到底部
In this Document
Symptoms
Cause
Solution
References
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
SYMPTOMS
Unable to start or stop 11.2 GRID Clusterware on one node of a two node cluster.
Errors on starting:
CRS-2674: Start of 'ora.ons' on '<host>' failed
Errors on stopping:
CRS-5014: Agent "/u01/grid/oracle/product/11.2.0.4/grid_source/bin/oraagent.bin" timed out starting process "/u01/grid/oracle/product/11.2.0.4/grid_source/opmn/bin/onsctli" for action "clean": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0.4/grid_source/log/node1/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5017: The resource action "ora.ons clean" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00106:)" in "/u01/grid/oracle/product/11.2.0.4/grid_source/log/node1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-5014: Agent "/u01/grid/oracle/product/11.2.0.4/grid_source/bin/oraagent.bin" timed out starting process "/u01/grid/oracle/product/11.2.0.4/grid_source/opmn/bin/onsctli" for action "check": details at "(:CLSN00009:)" in "/u01/grid/oracle/product/11.2.0.4/grid_source/log/node1/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5017: The resource action "ora.ons check" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00109:)" in "/u01/grid/oracle/product/11.2.0.4/grid_source/log/node1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2680: Clean of 'ora.ons' on 'node1' failed
CRS-2799: Failed to shut down resource 'ora.node1.ons' on 'ctdb1'
CRS-2799: Failed to shut down resource 'ora.ons' on 'node1'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'node1' has failed
ons.log.node1
2014-04-11T16:39:25-05:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2014-04-11T16:39:25-05:00] [ons] [ERROR:1] [17] [ons-listener] 10.x.x.x,6100: BIND (Can't assign requested address)
[2014-04-11T16:39:48-05:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2014-04-11T16:39:48-05:00] [ons] [ERROR:1] [17] [ons-listener] 10.x.x.x,6100: BIND (Can't assign requested address)
[2014-04-11T16:41:20-05:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2014-04-11T16:41:20-05:00] [ons] [ERROR:1] [17] [ons-listener] 10.x.x.x,6100: BIND (Can't assign requested address)
 
CAUSE
ping localhost.<domain> was returning the non-existence IP address. DNS was picking that IP resulting ONS start/stop failure.
node1# ping localhost.<domain>
PING <domain>: 64 byte packets
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 1/2/5
node1# ping localhost                        
PING cern-mt015672.northamerica.cerner.net: 64 byte packets
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms
64 bytes from 10.x.x.x: icmp_seq=0. time=5. ms


SOLUTION
1.  Update DNS to remove the reference of non-existent ip address of localhost.<domain> in DNS. This fixed the issue of ONS stop/start.

2.  Alternatively or perhaps short team,  host name resolution may be performed using local /etc/hosts rather than DNS.
Example for AIX platform:
Edit /etc/netsvc.conf, add:
hosts = local, bind
Example for Linux platform:
Edit /etc/nsswitch.conf:
hosts: files dns
Ensure the following is set in /etc/hosts:
127.0.0.1 localhost.localdomain localhost
Finally, ping localhost to verify 127.0.0.1 is returned.  After this ora.ons should start and stop without error.

4.grid安装报[INS-20802] Oracle Cluster Verification Utility failed.

导致这个错误的原因是在/etc/hosts中配置了SCAN的地址，尝试ping这个地址信息，如果可以成功，则这个错误可以忽略。
我尝试ping scan ip可以ping通，所以暂时也就忽略了这个错误。