Oracle_11g_RAC安装与配置for_Linux（2）

1. 确认软件要求
   1. LINUX发行版要求

Linux Distribution	Requirements
Asianux Distributions	Asianux Server 3, Service Pack 2 (SP2)
Oracle Linux Distributions	Oracle Linux 4 Update 7, kernel 2.6.9 or later Oracle Linux 5 Update 2, kernel 2.6.18 or later (compatible with Red Hat Enterprise kernel) Oracle Linux 5 Update 5, with the Unbreakable Enterprise Kernel for Linux, kernel 2.6.32-100.0.19 or later
Red Hat Enterprise Linux Distributions	Red Hat Enterprise Linux 4 Update 7, kernel 2.6.9 or later Red Hat Enterprise Linux 5 Update 2, kernel 2.6.18 or later Red Hat Enterprise Linux 5, Update 5, with the Unbreakable Enterprise Kernel for Linux, 2.6.32 or later
SUSE Enterprise Linux Distributions	SUSE 10, kernel 2.6.16.21 or later SUSE 11, kernel 2.6.27.19 or later

1. 1. Grid Infrastructure与RAC必须的安装包

Oracle Linux 5 Update 5 with Unbreakable Enterprise Kernel (2.6.32-100.0.19 or later)

Subscribe to the Oracle Linux 5 channel on the Unbreakable Linux Network, and then install the Oracle Validated RPM. This installs the Oracle Unbreakable Enterprise Kernel for Linux, and all required kernel packages for Oracle Grid Infrastructure and Oracle Database installations.

Asianux Server 3, Oracle Linux 5, Update 2, and Red Hat Enterprise Linux 5 Update 2

The following packages (or later versions) must be installed: Note: Starting with Oracle Grid Infrastructure 11g release 2 patchset 1 (11.2.0.2), all the 32-bit packages listed in the following table, except for gcc-32bit-4.3, are no longer required for installation. Only the 64-bit packages are required. However, for Oracle 11g release 2 (11.2.0.1), both the 32-bit and 64-bit packages listed in the following table are required. binutils-2.17.50.0.6 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.125 elfutils-libelf-devel-0.125 gcc-4.1.2 gcc-c++-4.1.2 glibc-2.5-24 glibc-2.5-24 (32 bit) glibc-common-2.5 glibc-devel-2.5 glibc-devel-2.5 (32 bit) glibc-headers-2.5 ksh-20060214 libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libaio-devel-0.3.106 (32 bit) libgcc-4.1.2 libgcc-4.1.2 (32 bit) libstdc++-4.1.2 libstdc++-4.1.2 (32 bit) libstdc++-devel 4.1.2 make-3.81 sysstat-7.0.2 unixODBC-2.2.11 unixODBC-2.2.11 (32 bit) unixODBC-devel-2.2.11 unixODBC-devel-2.2.11 (32 bit)

1. 1. Oracle数据库所需包

LDAP package	If you did not perform a default Linux installation, you intend to use LDAP, and you want to use the scripts odisrvreg, oidca, or schemasync, then install the Korn shell RPM for your Linux distribution.
Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer's Kit (XDK)	Intel C/C++ Compiler 10.1 or later and the version of GNU C and C++ compilers listed previously for the distribution are supported for use with these products. Note: Intel C/C++ Compiler v10.1 can be used only with the standard template libraries in the supported gcc versions to build Oracle C++ Call Interface (OCCI) applications. Oracle XML Developer's Kit is supported with the same compilers as OCCI.
Pluggable Authentication Modules for Linux (Linux PAM)	Install the latest Linux PAM (Pluggable Authentication Modules for Linux) library for your Linux distribution. PAM provides greater flexibility for system administrators to choose how applications authenticate users. On Linux, external scheduler jobs require PAM.
Oracle ODBC Drivers	If you intend to use Open Database Connectivity (ODBC), then you should install the most recent ODBC Driver Manager for Linux. You should obtain the most current ODBC driver from your operating system vendor. You can read about ODBC at the following URL: unixODBC The ODBC RPMs are only needed if you plan on using ODBC. If you do not plan to use ODBC, then you do not need to install the ODBC RPMs for Oracle Clusterware, Oracle ASM, or Oracle RAC. To use ODBC, you must also install the following additional 32-bit ODBC RPMs, depending on your operating system: Oracle Linux 4, and Red Hat Enterprise Linux 4: unixODBC-2.2.11 (32 bit) or later unixODBC-devel-2.2.11 (32 bit) or later Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5: unixODBC-2.2.11 (32 bit) or later unixODBC-devel-2.2.11 (32 bit) or later
Oracle JDBC/OCI Drivers	You can use the following optional JDK version with the Oracle JDBC/OCI drivers; however, it is not required for the installation: JDK 6 Update 10 (Java SE Development Kit 1.6.0_21) JDK 1.5.0-24 (JDK 5.0) with the JNDI extension Note: By default, JDK 1.5 is installed with this release.
Oracle Real Application Clusters	For a cluster file system, use the following option: OCFS2 Version 1.2.9 or later To check your OCFS version, enter the following command: modinfo ocfs2 To check the version of OCFS2 tools and OCFS2 console, enter the following command: rpm -qa| grep ocfs2 Ensure that ocfs2console and ocfs2-tools are at least version 1.2.7 For information about OCFS2, refer to the following Web site: Oracle Open Source For OCFS2 certification status, refer to the Certify page on My Oracle Support. Oracle Open Source For OCFS2 certification status, refer to the Certify page on My Oracle Support.

1. 检查软件需求
   1. 确认Linux发行版

cat /proc/version

1. 1. 确认内核版本

uname -r

1. 1. 查询安装包

rpm -q binutils

rpm -q compat-libstdc++

rpm -1 elfutils

rpm -q gcc

rpm -q glibc

rpm -q ksh

rpm -q libaio

rpm -q libgcc

rpm -q libstdc++

rpm -q make

rpm -q sysstat

rpm -q unixODBC

1. 1. 查看OCFS是否安装

/sbin/modinfo ocfs2  如果没有则下载安装一个

http://oss.oracle.com/projects/ocfs2/

1.  校验和设置UDP和TCP内核参数
   1. 查看参数

cat /proc/sys/net/ipv4/ip_local_port_range

1. 1. 配置/etc/sysctl.conf

net.ipv4.ip_local_port_range = 9000 65500

1. 1. 重启网络

/etc/rc.d/init.d/network restart

1.  NTP配置
   1. 可以使用群集同步时间服务代替NTP，当群集发NTP协议没有启用时，群集会自动使用时间同步。停止时间同步服务步骤如下：

# /sbin/service ntpd stop

# chkconfig ntpd off

# mv /etc/ntp.conf  /etc/ntp.conf.org.

1. 1. 如果使用群集时间同步，执行如下命令检查

crsctl check ctss

• 1. 如果使用NTP，那么增加-x参数，防止时间回退
• 编辑文件/etc/sysconfig/ntpd

# Drop root to id 'ntp:ntp' by default.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

# Set to 'yes' to sync hw clock after successful ntpdate

SYNC_HWCLOCK=no

# Additional options for ntpdate

NTPDATE_OPTIONS=""

• 重新启动 NTP service.

 # /sbin/service ntp restart

• 1. 配置NTP
     1. linux硬件时间设置
• 默认情况下Linux运行使用的是CPU tick，不会获取BIOS里面存的系统时间，在系统开机时会自动从BIOS里面获取。
• 系统时间可以用date设置：date –set “月/日/年时:分:秒”
• 硬件时间使用clock和hwclock设置：

hwclock –show   显示硬件时间

hwclock –set –date=“月/日/年时:分:秒”

• 将系统时间写入硬件时间

hwclock –systohc

hwclock -w

• 将硬件时间写入系统时间

hwclock –hctosys

hwclock –s

1. 1. 1. 不同机器间时间同步，在配置NTP服务前首先使用ntpdate同步机器间的时间，因为时间差异过大时NTP不会同步

ntpdate hostname

1. 1. 1. 配置NTP
         1. 配置/etc/ntp.conf文件

配置权限

restrict default nomodify notrap noquery # 关闭所有的 NTP 要求封包

restrict 127.0.0.1    #这是允许本级查询

restrict 192.168.0.0 mask 255.255.255.0 nomodify

#在192.168.0.0/24网段内的服务器就可以通过这台NTP Server进行时间同步了

                            server 0.pool.ntp.org

server 1.pool.ntp.org

server 2.pool.ntp.org

server  127.127.1.0  # 本地时间，在NTP服务端配置

fudge   127.127.1.0 stratum 10 # 本地时间，在NTP服务端配置

server  192.168.0.1  # 远程主机，在NTP客户端配置

fudge  192.168.0.1 stratum 10 # 远程主机，在NTP客户端配置

driftfile /var/lib/ntp/drift  #用于保存漂移时间

1. 1. 1. 1. 启动NTP服务

service ntpd start 或 /etc/init.d/ntpd start

1. 1. 1. 1. 查看端口确认是否启动NTP

netstat –in | grep 123

1. 1. 1. 1. 查看NTP与上层主机同步情况：ntpstat
         2. 查看NTP与上层主机的关系：ntptrace –n 127.0.0.1
         3. 查看NTP与上层主机的状态ntpq -p

1.  安装cvuqdisk包

cvuqdisk用于CVU查找共享磁盘，在grid_home/grid目录下

1. 1. 查看是否安装cvuqdisk

rpm -qi cvuqdisk

1. 1. 如果安装，卸载cvuqdisk

rpm -e cvuqdisk

1. 1. 配置环境变量

CVUQDISK_GRP=oinstall; export CVUQDISK_GRP

1. 1. 安装包

rpm -iv package

1.  使能Intelligent Platform Management Interface (IPMI)

IPMI提供监视系统硬件和固件的接口

安装见Oracle Clusterware Administration and Deployment Guide

1.  在安装期间自动配置SSH

如果没有配置其它认证方式，在安装期间Oracle可以自动配置SSH

1.  配置Grid Infrastructure Software Owner用户环境
   1.  Grid Infrastructure Software Owner环境要求

设置software owner用户（grid，oracle）的umask为022，确保建立文件的权限是644。

设置software owner用户（grid，oracle）处理器权限为ulimit

设置DISPLAY环境变量

1. 1.  配置Grid Infrastructure Software Owner用户环境
      1. 设置主机显示权限

xhost + 主机名

1. 1. 1.  登录安装目录确认默认shell

su – grid

eche $SHELL

1. 1. 1.  编辑shell启动文件

Bash shell (bash):

 $ vi .bash_profile

Bourne shell (sh) or Korn shell (ksh):

 $ vi .profile

C shell (csh or tcsh):

 % vi .login

1. 1. 1.  在启动文件中增加 umask 022
      2.  如果不是在本地安装增加DISPLAY变量

export DISPLAY=curstom_name:0.0

1. 1. 1.  如果/tmp空间小于1GB，配置TEMP和TMPDIR环境变量替换默认临时空间

$ TEMP=/mount_point/tmp

$ TMPDIR=/mount_point/tmp

$ export TEMP TMPDIR

1. 1.  校验环境

umask

env | more

1. 1.  检查资源限制，确保显示值在推荐范围内
      1. 文件句柄限制

$ ulimit -Sn

1024

$ ulimit -Hn

65536

1. 1. 1. 进程限制

$ ulimit -Su

2047

$ ulimit -Hu

16384

1. 1. 1. 堆栈限制

$ ulimit -Ss

10240

$ ulimit -Hs

32768

1. 1. 设置X11转发

为了防止X11转发导致的安装失败，增加并配置~/.ssh/config文件

Host *

      ForwardX11 no

1. 1. 消除终端输出命令导致的错误

Bourne, Bash, or Korn shell:

 if [ -t 0 ]; then

   stty intr ^C

fi

C shell:

 test -t 0

if ($status == 0) then

   stty intr ^C

endif

1.  建立必须的目录

# mkdir -p  /u01/app/11.2.0/grid

# chown grid:oinstall /u01/app/11.2.0/grid

# chmod -R 775 /u01/app/11.2.0/grid

# mkdir -p  /u01/app/oracle

# chown -R oracle:oinstall /u01/app/oracle

# chmod -R 775 /u01/app/oracle

1.  群集名要求

群集名必须少于15字符

• 为Cluster and Oracle Real Application Clusters (Oracle RAC)安装Grid Infrastructure配置存储

1. Oracle Clusterware and Oracle RAC存储选项综述

有两种方式支持Oracle群集文件：Oracle Clusterware files (Oracle Cluster Registry and voting disk files)能存储在Oracle ASM上。也能存放在一个共享文件系统上，如NFS。对于IBM linux或system z，能够使用块设备，但是不能使用ACFS和OCFS

1. 了解ACFS

Oracle Automatic Storage Management Cluster File System (Oracle ACFS)用于存储Oracle执行文件，但是不能用于存储数据文件和群集文件。

1. 支持的存储选项

Storage Option	OCR and Voting Disk Files	Oracle Clusterware binaries	Oracle RAC binaries	Oracle Database Files	Oracle Recovery Files
Oracle Automatic Storage Management (Oracle ASM) Note: Loopback devices are not supported for use with Oracle ASM	Yes	No	No	Yes	Yes
Oracle Automatic Storage Management Cluster File System (Oracle ACFS)	No	No	Yes	No	No
Local file system	No	Yes	Yes	No	No
NFS file system on a certified NAS filer Note: Direct NFS does not support Oracle Clusterware files.	Yes	Yes	Yes	Yes	Yes
Shared disk partitions (block devices or raw devices)	No 注一	No	No	No 注一.	No

注一：Not supported by OUI or ASMCA, but supported by the software. They can be added or removed after installation

1. 共享文件系统配置
   1. 群集文件文件系统所需尺寸

File Types Stored	Number of Volumes	Volume Size
Voting disks with external redundancy	3	At least 300 MB for each voting disk volume.
Oracle Cluster Registry (OCR) with external redundancy	1	At least 300 MB for each OCR volume
Oracle Clusterware files (OCR and voting disks) with redundancy provided by Oracle software.	1	At least 300 MB for each OCR volume At least 300 MB for each voting disk volume

1. 1. Oracle RAC共享文件系统所需尺寸

File Types Stored	Number of Volumes	Volume Size
Oracle Database files	1	At least 1.5 GB for each volume
Recovery files Note: Recovery files must be on a different volume than database files	1	At least 2 GB for each volume

• 1. Oracle ASM存储配置
     1. 使用ASMLIB配置Oracle ASM
        1. 安装配置ASM Library Driver软件
• 检查Linux内核版本：uname –rm
• 下载ASMLIB软件

http://www.oracle.com/technetwork/server-storage/linux/downloads/index-088143.html

oracleasm-support-version.arch.rpm

oracleasm-kernel-version.arch.rpm

oracleasmlib-version.arch.rpm

1. 1. 1. 1. 安装ASMLIB包

rpm -Uvh oracleasm-support-2.0.1.x86_64.rpm \

           oracleasmlib-2.0.1.x86_64.rpm \

           oracleasm-2.6.9-11.EL-2.0.1.x86_64.rpm

1. 1. 1. 1. 执行命令初始化OracleASM配置，此脚本将创建

/etc/sysconfig/oracleasm配置文件，并创建/dev/oracleasm安装点，安装ASMLIB文件系统

# /usr/sbin/oracleasm configure -i

1. 1. 1. 1. 载入oracleasm内核模块

# /usr/sbin/oracleasm init

1. 1. 1. 1. 使用ASM library配置磁盘设备

在linux平台使用fdisk创建磁盘分区

fdisk /dev/sdb

1. 1. 1. 1. 表示磁盘作为ASM磁盘

# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

1. 1. 1. 1. 在其它节点识别磁盘

# /usr/sbin/oracleasm scandisks

1. 1. 1. 1. 管理ASMLIB驱动和磁盘

Option	Description
configure	Use the configure option to reconfigure the Oracle Automatic Storage Management library driver, if necessary: # /usr/sbin/oracleasm configure -i To see command options, enter oracleasm configure without the -i flag.
enable disable	Use the disable and enable options to change the actions of the Oracle Automatic Storage Management library driver when the system starts. The enable option causes the Oracle Automatic Storage Management library driver to load when the system starts: # /usr/sbin/oracleasm enable
start stop restart	Use the start, stop, and restart options to load or unload the Oracle Automatic Storage Management library driver without restarting the system: # /usr/sbin/oracleasm restart
createdisk	Use the createdisk option to mark a disk device for use with the Oracle Automatic Storage Management library driver and give it a name: # /usr/sbin/oracleasm createdisk DISKNAME devicename
deletedisk	Use the deletedisk option to unmark a named disk device: # /usr/sbin/oracleasm deletedisk DISKNAME Caution: Do not use this command to unmark disks that are being used by an Oracle Automatic Storage Management disk group. You must delete the disk from the Oracle Automatic Storage Management disk group before you unmark it.
querydisk	Use the querydisk option to determine if a disk device or disk name is being used by the Oracle Automatic Storage Management library driver: # /usr/sbin/oracleasm querydisk {DISKNAME | devicename}
listdisks	Use the listdisks option to list the disk names of marked Oracle Automatic Storage Management library driver disks: # /usr/sbin/oracleasm listdisks
scandisks	Use the scandisks option to enable cluster nodes to identify which shared disks have been marked as Oracle Automatic Storage Management library driver disks on another node: # /usr/sbin/oracleasm scandisks

1. 1. 1. 手动配置Oracle ASM
         1. 查询unique device identifiers

Linux 5.x

# /sbin/scsi_id -g -s /block/sdb/sdb1

Linux 6.x

scsi_id --whitelisted --replace-whitespace --device=/dev/sdc

1. 1. 1. 1. 增加或修改/etc/scsi_id.config文件

Linux 5.x

     options=-g

Linux 6.x

     options=--whitelisted --replace-whitespace

1. 1. 1. 1. 编辑文件/etc/udev/rules.d/99-oracle-asmdevices.rules建立UDEV规则

Linux 5.x

KERNEL==”sd?1″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”SATA_VBOX_HARDDISK_VBce8c63bb-ac67a172_”, NAME=”asm-disk3″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

Linux 6.x

KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name",

RESULT=="1ATA_VBOX_HARDDISK_VB36a9e548-1838194a",

NAME="asm-disk1",  OWNER="grid",  GROUP="asmadmin", MODE="0660"

1. 1. 1. 1. 复制rule文件到其它节点

scp 99-oracle-asmdevices.rules node:/etc/udev/rules.d

1. 1. 1. 1. 使用udevadm进行测试

udevadm test /sys/block/sdc

udevadm info --query=all --path=/sys/block/sdc

udevadm info --query=all --name=asm-disk1

1. 1. 1. 1. 加载配置文件

Linux 5.x

# /sbin/udevcontrol reload_rules

Linux 6.x

# udevadm control reload-rules

1. 1. 1. 1. 启动udev设备

# /sbin/start_udev

1. 1. 1. 1. 检查设备是否正确绑定

ls -l /dev/asm*