Oracle11g RAC for Linux 实施手册

1.1 网络分配：
主机名 公网(public) 专网(private) 虚拟网(vip)
Rac1 IP IP IP
Rac2 IP IP IP
1.2 硬件资源
Cpu ：Intel(R) Xeon(TM) MP CPU 3.16GHz*2
内存：1G
硬盘： 20G
共享磁盘：6G
1.3 操作系统安装
1.3.1 操作系统版本：Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
1.3.2 系统分区
分区信息 分区类型 分区大小
/boot ext3 100M
/ ext3 10240M
Swap swap 2048
/oracle ext3 10240M
/opt ext3 5120M
/tmp ext3 1024M
1.3.3 系统语言
简体中文，ENGLISH（USA）
1.3.4 安装模式
完全安装
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
2
二 数据安装资源
2.1 数据库安装包
2.1.1 ORACLE 11G 软件包：
下载地址：www.oracle.com
Oracle Database 11g Release 1 (11.1.0.6.0) for Linux x86
Oracle Database 11g Release 1 Client (11.1.0.6.0) for Linux x86
Oracle Database 11g Examples (formerly Companion)
Oracle Clusterware Release 1 (11.1.0.6.0) for Linux x86
Oracle Database Gateways 11g Release 1 (11.1.0.6.0) for Linux x86
2.1.2 ORACLE 11G-ASMlib 工具包：
下载地址：http://www.oracle.com/technology/software/tech/linux/asmlib/rhel4.html
oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm
oracleasm-support-2.0.3-1.i386.rpm
oracleasmlib-2.0.2-1.i386.rpm
2.1.3 系统补丁的下载
下载地址：http://oss.oracle.com/projects/compat-oracle/files/RedHat/
compat-libcwait-2.1-1.i386.rpm
compat-oracle-rhel4-1.0-5.i386.rpm
三 系统安装配置
3.1 系统参数配置
3.1.1 关闭没必要的服务
chkconfig --level 35 autofs off
chkconfig --level 35 acpid off
chkconfig –level 35 sendmail off
chkconfig –level 35 cups-config-daemon off
chkconfig –level 35 cpus off
chkconfig –level 35 xfs off
chkconfig –level 35 lm_sensors off
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
3
chkconfig –level gpm off
chkconfig –level openibd off
chkconfig –level iiim off
chkconfig –level pcmcia off
chkconfig –level cpuspeed off
chkconfig –level nfslock off
chkconfig –level ip6tables off
chkconfig –level rpcidmapd off
chkconfig –level apmd off
chkconfig –level sendmail off
chkconfig –level arptables_jf off
chkconifg –level microcode_ctl off
chkconfig –level rpcgssd off
3.1.2 设置主机表
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
IP rac1
IP racp1
IP racv1
IP rac2
IP racp2
IP racv2
3.1.3 修改内核参数
vi /etc/sysctl.conf
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 4194304
net.core.wmem_max = 4194304
kernel.panic = 60
3.1.4 设置 oracle 登录使用资源数
vi /etc/security/limits.conf
oracle soft nofile 65536
oracle hard nofile 65536
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
4
oracle soft nproc 16384
oracle hard nproc 16384
vi /etc/pam.d/login –末行加入以下
session required pam_limits.so multiple open
3.1.5 建立 hangcheck 模块加载
检测是否安装了 hangcheck 模块
[root@rac1]# find /lib/modules -name "hangcheck-timer.ko"
/lib/modules/2.6.9-42.ELsmp/kernel/drivers/char/hangcheck-timer.ko
设置模块为自动加载
su –
echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf
或者
echo " /sbin/modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/rc.local
检测模块是否运行
su -
modprobe hangcheck-timer
grep Hangcheck /var/log/messages | tail -2
3.1.6 禁用 selinux
getsebool getsebool: SELinux is disabled
vi /etc/selinux/config
SELINUX=disabled
3.1.7 添加 oracle 用户和用户组
mkdir –p /oracle/oracle
groupadd –g 502 dba
groupadd –g 501 oinstall
useradd –g 501 –G 502 –d /oracle/oracle oracle
passwd oracle
chown -R oracle.oinstall /oracle/oracle
3.1.8 修改主机名
vi /etc/sysconfig/network
HOSTNAME=rac1
# hostname rac1
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
5
Rac 2 上同样执行
3.2 设置 oracle 用户
vi .bash_profile
umask 022
export ORACLE_BASE=/oracle/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
#export ORACLE_SID=rac2
export ORACLE_SID=rac1
export NLS_LANG=AMERICAN_AMERICA.zhs16gbk
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_ASSUME_KERNEL=2.4.19 # for RHEL AS 4
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
6
export TMPDIR=/tmp
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
3.3 配置 ASM 服务
3.3.1 安装 asm 包
rpm -Uvh oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm oracleasmlib-2.0.2-1.i386.rpm oracleasm-support-2.0.3-1.i386.rpm
安装 asm 包一定要和系统的内核相对应
3.3.2 配置 asm 服务
cd /etc/init.d/
[root@localhost init.d]# ./oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Creating /dev/oracleasm mount point: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
7
3.4 建立ssh 等效性（在oracle 用户操作）
3.4.1 建立相关目录生成密钥和公钥
su - oracle
mkdir -p ~/.ssh
chmod 755 ~/.ssh
[oracle@rac1 ~]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/oracle/oracle//.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /oracle/oracle//.ssh/id_rsa.
Your public key has been saved in /oracle/oracle//.ssh/id_rsa.pub.
The key fingerprint is:
16:2b:45:27:24:ee:97:46:5e:72:43:f1:9f:df:4b:dd oracle@rac1
[oracle@rac1 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/oracle/oracle//.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /oracle/oracle//.ssh/id_dsa.
Your public key has been saved in /oracle/oracle//.ssh/id_dsa.pub.
The key fingerprint is:
0a:d5:e7:6a:a1:11:de:e4:72:74:fc:5f:b2:63:5b:6b oracle@rac1
同等在RAC2 上建立公钥和私钥
3.4.2 建立等效密钥
在第一个主机上，以 oracle 用户身份登录
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/authorized_keys ~/.ssh/rac1.authorized_keys
第二台主机上，以oracle 用户登录
cat ~/.ssh/id_ >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/authorized_keys ~/.ssh/rac2.authorized_keys
相互交换公钥
scp ~/.ssh/rac1.authorized_keys oracle@IP:
scp ~/.ssh/rac2.authorized_keys oracle@IP:
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
8
RAC2 上执行：
cat rac1.authorized_keys >> ~/.ssh/authorized_keys
RAC1 上执行：
cat rac2.authorized_keys >> ~/.ssh/authorized_keys
建立等效性（每台机器都执行）
ssh rac1
ssh rac2
ssh racp1
ssh racp2
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add
四 配置共享磁盘
4.1 建立 ISCSI 的共享磁盘
4.1.1 编译 iscsi-target 的源包
mkdir –p /usr/src/iscsi-target
wget http://www.cryptoforge.net/iscsi/RPMS/iscsitarget-0.4.12-6.src.rpm
rpmbuild --rebuild iscsitarget-0.4.12-6.src.rpm
rpm -Uvh iscsitarget-kernel-smp-0.4.12-6_2.6.9_42.EL.i386.rpm iscsitarget-0.4.12-6.i386.rpm
4.1.2 建立分区
fdisk /dev/sda
生成：/dev/sda8
fdisk /dev/sdb
生成：/dev/sdb1
4.1.3 配置共享盘
ISCSI 服务端配置：
vi /etc/ietd.conf
注释掉所有的其他
Target iqn.2000-12.com.digicola:storage.lun1
IncomingUser gfs secretsecret
OutgoingUser
Lun 0 Path=/dev/sdb1,Type=fileio
Alias iDISK0
#MaxConnections 6
Target iqn.2000-12.com.digicola:storage.lun2
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
9
IncomingUser gfs secretsecret
OutgoingUser
Lun 0 Path=/dev/sda8,Type=fileio
Alias iDISK1
#MaxConnections 6
启动服务：service iscsi-target restart
加入开机启动：chkconfig –level 2345 iscsi-target on
Iscsi 客户端配置：
vi /etc/iscsi.conf
DiscoveryAddress=192.168.1.156
OutgoingUserName=gfs
OutgoingPassword=secretsecret
LoginTimeout=15
DiscoveryAddress=192.168.1.15
启动服务：service iscis restart
加入开机启动：chkconfig –level 2345 iscsi on
检查：iscsi 共享磁盘：
[root@rac1 ~]# iscsi-ls
*******************************************************************************
SFNet iSCSI Driver Version ...4:0.1.11-3(02-May-2006)
*******************************************************************************
TARGET NAME : iqn.2000-12.com.digicola:storage.lun2
TARGET ALIAS : HOST ID : 1
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.1.156:3260,1
SESSION STATUS : ESTABLISHED AT Thu Oct 25 23:14:52 CST 2007
SESSION ID : ISID 00023d000001 TSIH 400
注：在从启系统时要注意 iscsi-target 服务和 iscsi 服务的启动顺序。先启动 iscsi-target 在启动 iscsi ，在
/etc/rc3.d 下修改 S 后的数值，值约小启动级别越高
4.2 建立共享分区列表(一个节点操作即可)
分区信息 大小
/dev/sdc1 200M
/dev/sdc2 200M
/dev/sdc3 200M
/dev/sdc5 200M
/dev/sdc6 200M
/dev/sdc7 12000M
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
10
/dev/sdd1 2000M
/dev/sdd2 1000M
/dev/sdd3 1000M
/dev/sdd5 2600M
4.3 建立 raw 设备
vi /etc/sysconfig/rawdevices
/dev/raw/raw1 /dev/sdc1
/dev/raw/raw2 /dev/sdc2
/dev/raw/raw3 /dev/sdc3
/dev/raw/raw4 /dev/sdc5
/dev/raw/raw5 /dev/sdc6
启动服务：sevice rawdevices restart
开机启动：chkconfig –level 2345 rawdevices on 4.4 开机授权
vi /etc/rc.local
chown –R oracle.oinstall /dev/raw/raw*
chmod +755 /dev/raw/raw*
4.5 建立 asm 磁盘卷组
4.5.1 建立 asm 卷组（单机执行）
service oracleasm createdisk FLASH /dev/sdd1
service oracleasm createdisk ARCH1 /dev/sdd2
service oracleasm createdisk ARCH2 /dev/sdd3
service oracleasm createdisk DATA /dev/sdd5
4.5.2 扫描 asm 卷组
service oracleasm scandisks
4.5.3 显示 asm 卷组
service oracleasm listdisks
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
11
五安装 Cluster 软件
注： 所有数据库软件都存放在/opt/oracle11g
5.1 cluster 硬件检测
./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
5.1.1 解压 cluster 软件包
unzip linux_11gR1_clusterware.zip
安装补丁包
cd /opt/oracle11g/cluster/rpm
rpm -Uvh cvuqdisk-1.0.1-1.rpm
安装系统补丁包
rpm -Uvh compat-libcwait-2.1-1.i386.rpm compat-oracle-rhel4-1.0-5.i386.rpm
第一步运行 $ ./runInstaller
su – oracle
cd /opt/oracle11g/cluster
./runInstaller
启动界面
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
12
指定 crs 主目录
检测系统资源
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
13
添加节点
设定网络使用（public、private）
指定 OCR 安装位置
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
14
指定 Vo 磁盘
指定安装内容开始安装
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
15
安装完毕
Root 在 rac1、rac2 上依次运行
cd /oracle/oracle/
./orainstRoot.sh
和
cd /oracle/oracle/product/11.1.0.6.0/crs
./root.sh
安装完毕
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
16
六安装数据库软件
6.1 安装数据库的软件
Su – oracle
Cd /opt/oracle11g/database
./runInstaller
安装界面
选择安装内容
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
17
安装目录的选择
集群模式安装
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
18
选择安装组建
选择操作组（默认）
安装软件
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
19
是否需求 oracle 注册管理
安装软件确定
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
20
开始安装
安装完毕
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
21
6.2 配置监听
su – oracle
netca
配置监听名（不能变）
七 建立实例、安装数据库、配置 asm
su – oracle
dbca
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
22
7.1 建立 asm 磁盘组
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
23
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
24
7.2 建立数据库实例
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
25
选择密码
集群文件系统的选择
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
26
选择 asm 卷组
数据区的选择
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
27
选择安装组建
指定 SGA、字符集、语言、进程数、连接模式
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
28
开始建立数据实例
建立实例模板
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
29
建立实例
数据库建立完成指名 emo 位置
八.参数调整
8.1 集群命令的使用
8.1.1 crsctl
Usage: crsctl check crs - checks the viability of the Oracle Clusterware
crsctl check cssd - checks the viability of Cluster Synchronization Services
crsctl check crsd - checks the viability of Cluster Ready Services
crsctl check evmd - checks the viability of Event Manager
crsctl check cluster [-node <nodename>] - checks the viability of CSS across nodes
http://www.Database8.com
Oracle11g RAC for Linux 实施手册
30
crsctl set css <parameter> <value> - sets a parameter override
crsctl get css <parameter> - sets the value of a Cluster Synchronization Services parameter
crsctl unset css <parameter> - sets the Cluster Synchronization Services parameter to
its default
crsctl query css votedisk - lists the voting disks used by Cluster Synchronization
Services
crsctl add css votedisk <path> - adds a new voting disk
crsctl delete css votedisk <path> - removes a voting disk
crsctl enable crs - enables startup for all Oracle Clusterware daemons
crsctl disable crs - disables startup for all Oracle Clusterware daemons
crsctl start crs [-wait] - starts all Oracle Clusterware daemons
crsctl stop crs [-wait] - stops all Oracle Clusterware daemons. Stops Oracle Clusterware
managed resources in case of cluster.
crsctl start resources - starts Oracle Clusterware managed resources
crsctl stop resources - stops Oracle Clusterware managed resources
crsctl debug statedump css - dumps state info for Cluster Synchronization Services
objects
crsctl debug statedump crs - dumps state info for Cluster Ready Services objects
crsctl debug statedump evm - dumps state info for Event Manager objects
crsctl debug log css [module:level] {,module:level} ... - turns on debugging for Cluster
Synchronization Services
crsctl debug log crs [module:level] {,module:level} ... - turns on debugging for Cluster
Ready Services
crsctl debug log evm [module:level] {,module:level} ... - turns on debugging for Event
Manager
crsctl debug log res [resname:level] ... - turns on debugging for Event Manager
crsctl debug trace css [module:level] {,module:level} ... - turns on debugging for Cluster Synchronization Services
crsctl debug trace crs [module:level] {,module:level} ... - turns on debugging for Cluster Ready Services
crsctl debug trace evm [module:level] {,module:level} ... - turns on debugging for Event
Manager
crsctl query crs softwareversion [<nodename>] - lists the version of Oracle Clusterware
software installed
crsctl query crs activeversion - lists the Oracle Clusterware operating version
crsctl lsmodules css - lists the Cluster Synchronization Services modules that can be
used for debugging
crsctl lsmodules crs - lists the Cluster Ready Services modules that can be used for
debugging
crsctl lsmodules evm - lists the Event Manager modules that can be used for debugging