在vmware rac上添加新节点

之前在vmware中装了一个双节点的rac，在这里，我们给rac增加一个新节点。当前的rac情况是：

rac1：
IP：192.168.1.131
Private-IP：10.10.10.31
VIP：192.168.1.31

rac2：
IP：192.168.1.132
Private-IP：10.10.10.32
VIP：192.168.1.32

需要新加的rac3：
rac3：
IP：192.168.1.133
Private-IP：10.10.10.33
VIP：192.168.1.33

[@more@]

一、系统的准备：
1、停止双节点rac后，复制其中一个节点。然后只启动这个节点。启动后，create UUID的时候，选择create

2、在启动的时候，create configure IP和网关可随意写，等进入系统后再修改。

3、启动后修改IP、为/etc/hosts增加节点3的信息、修改hostname为rac3，最后重新激活网卡。

4、这时可以启动rac1和rac2，并修改2个节点的/etc/hosts文件，增加上rac3的信息。3个节点的hosts文件形如：

一定要有127.0.0.1 localhost这一行，否则后面会出问题。

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost
192.168.1.131 rac1.mycorpdomain.com rac1
192.168.1.31 rac1-vip.mycorpdomain.com rac1-vip
10.10.10.31 rac1-priv.mycorpdomain.com rac1-priv
192.168.1.132 rac2.mycorpdomain.com rac2
192.168.1.32 rac2-vip.mycorpdomain.com rac2-vip
10.10.10.32 rac2-priv.mycorpdomain.com rac2-priv
192.168.1.133 rac3.mycorpdomain.com rac3
192.168.1.33 rac3-vip.mycorpdomain.com rac3-vip
10.10.10.33 rac3-priv.mycorpdomain.com rac3-priv

# ntp server
192.168.1.189 HEJIANMIN

其中192.168.1.189 HEJIANMIN是ntp服务器的设置，ntp服务器是我的windows本机。关于ntp服务器的架设，可以见这里。

5、修改ocfs的配置，增加第三个的节点。3个机器上的配置文件都需要修改。修改/etc/ocfs2/cluster.conf，形如：

注意增加的number=2,也就是第三个节点，并将node_count改成3.

node:
ip_port = 7777
ip_address = 192.168.1.131
number = 0
name = rac1
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.1.132
number = 1
name = rac2
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.1.133
number = 2
name = rac3
cluster = ocfs2

cluster:
node_count =3
name = ocfs2

6、在rac3：

/etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK

7、重新配置rac3的o2cb

[root@rac3 ocfs]# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting < ENTER > without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [61]: 61
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
[root@rac3 ocfs]#

8、检查rac3上的ocfs服务：

[root@rac3 ocfs]# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Checking O2CB heartbeat: Not active

9、重启3个节点，使得ocfs在3个节点上生效。
其实也可以这样：
节点1，umount ; o2cb stop; start;mount
节点2，umount ; o2cb stop; start;mount
这样避免rac的对外服务全部停止。

所以在实际的环境中应该可以做到在不停服务的情况下增加节点。

10、配置用户信任。

之后两个节点rac时，只有rsa的密钥，三个节点的发现用rsa的不行，最后将dsa的密钥也增加了互信，OK.

在rac3上：

rac3-> cd
rac3-> cd .ssh
rac3-> rm *
rac3-> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_rsa.
Your public key has been saved in /export/home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
c8:86:12:ed:96:57:33:84:4f:47:00:95:ad:25:52:dd oracle@rac3.mycorpdomain.com
rac3-> ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_dsa.
Your public key has been saved in /export/home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
ff:9a:1d:f1:9a:40:48:ce:2c:b2:31:01:79:6e:a3:ac oracle@rac3.mycorpdomain.com
rac3->
rac3->
rac3-> ll
total 32
-rw------- 1 oracle oinstall 668 Jun 28 16:35 id_dsa
-rw-r--r-- 1 oracle oinstall 618 Jun 28 16:35 id_dsa.pub
-rw------- 1 oracle oinstall 883 Jun 28 16:35 id_rsa
-rw-r--r-- 1 oracle oinstall 238 Jun 28 16:35 id_rsa.pub
rac3->
rac3->
rac3->
rac3-> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
rac3-> cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
rac3-> ll
total 40
-rw-r--r-- 1 oracle oinstall 856 Jun 28 16:36 authorized_keys
-rw------- 1 oracle oinstall 668 Jun 28 16:35 id_dsa
-rw-r--r-- 1 oracle oinstall 618 Jun 28 16:35 id_dsa.pub
-rw------- 1 oracle oinstall 883 Jun 28 16:35 id_rsa
-rw-r--r-- 1 oracle oinstall 238 Jun 28 16:35 id_rsa.pub
rac3->
rac3->
rac3-> ssh rac2 cat ~/.ssh/id_rsa.pub
The authenticity of host 'rac2 (192.168.1.132)' can't be established.
RSA key fingerprint is aa:e4:aa:8d:bc:8b:59:da:a5:f8:4f:de:b9:4b:6a:30.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2,192.168.1.132' (RSA) to the list of known hosts.
oracle@rac2's password:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEApj3pcBEu8xCAzBSW9ZD1AQQTICAGJb5EkVM/HmgkvkrXZS16oQ7APk4abUNFknR93/takk61P0/zapF74R3YP0ycTLGQj6T6215Hp3yaN7vsMphn2Wn0LC4jtkEeS7bKazpxeuFwk6baHfNQ5RuTLMVs97rklF8eLjigLlj/so8= oracle@rac2.mycorpdomain.com
rac3->
rac3->
rac3->
rac3->
rac3-> ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
oracle@rac2's password:
rac3->
rac3-> ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
oracle@rac2's password:
rac3-> ssh rac1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'rac1 (192.168.1.131)' can't be established.
RSA key fingerprint is aa:e4:aa:8d:bc:8b:59:da:a5:f8:4f:de:b9:4b:6a:30.
Are you sure you want to continue connecting (yes/no)? ys
Please type 'yes' or 'no': yes
Warning: Permanently added 'rac1,192.168.1.131' (RSA) to the list of known hosts.
oracle@rac1's password:
rac3->
rac3->
rac3-> ssh rac1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
oracle@rac1's password:
rac3->
rac3->
rac3->
rac3->
rac3-> scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys
oracle@rac2's password:
authorized_keys 100% 2568 2.5KB/s 00:00
rac3->
rac3-> scp ~/.ssh/authorized_keys rac1:~/.ssh/authorized_keys
oracle@rac1's password:
authorized_keys 100% 2568 2.5KB/s 00:00
rac3->

并且在3个节点上运行以下语句，看是否提示需要密码。如果一次不行，请重复执行一次：

ssh rac1 date
ssh rac2 date
ssh rac3 date
ssh rac1-priv date
ssh rac2-priv date
ssh rac3-priv date
ssh rac1.mycorpdomain.com date
ssh rac2.mycorpdomain.com date
ssh rac3.mycorpdomain.com date
ssh rac1-priv.mycorpdomain.com date
ssh rac2-priv.mycorpdomain.com date
ssh rac3-priv.mycorpdomain.com date

11、修改oracle用户的环境变量为devdb3。

12、由于我们只需一份操作系统，我们进行到此步骤后，可以删除/u01/app和/u01/oradata.

(注：系统的准备工作在这里已经完成，如果条件允许，在这里可以关闭vmware的3个机器，做个备份。)

二、CRS的安装：
1、启动xmanager，在rac1上export DISPLAY之后，用oracle用户运行addnode.sh
cd /u01/app/oracle/product/10.2.0/crs_1/oui/bin
./addNode.sh
后面的操作步骤，看图说话：

这里按照提示，在rac1用root权限执行

[root@rac1 ~]# sh /u01/app/oracle/product/10.2.0/crs_1/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node < nodenumber > : < nodename > < private interconnect name > < hostname >
node 3: rac3 rac3-priv rac3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/u01/app/oracle/product/10.2.0/crs_1/bin/srvctl add nodeapps -n rac3 -A rac3-vip/255.255.255.0/eth0 -o /u01/app/oracle/product/10.2.0/crs_1
[root@rac1 ~]#

在rac3用root执行：

[root@rac3 10.2.0]# sh /u01/app/oracle/product/10.2.0/crs_1/root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
OCR LOCATIONS = /ocfs/clusterware/ocr
OCR backup directory '/u01/app/oracle/product/10.2.0/crs_1/cdata/crs' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node < nodenumber > : < nodename > < private interconnect name > < hostname >
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
rac2
rac3
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
IP address "rac1-vip" has already been used. Enter an unused IP address.
[root@rac3 10.2.0]#

2、执行完在rac1上的rootaddnode.sh 和rac3上的root.sh之后，在rac3上执行
$ORA_CRS_HOME/bin/racgons add_conf rac3:4948 将本地ons服务注册。

3、检查安装情况：

/u01/app/oracle/product/10.2.0/crs_1/bin/cluvfy stage -post crsinst -n rac1,rac2,rac3
Performing post-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "rac3".


Checking user equivalence...
User equivalence check passed for user "oracle".

Checking Cluster manager integrity...


Checking CSS daemon...
Daemon status check passed for "CSS daemon".

Cluster manager integrity check passed.

Checking cluster integrity...


Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...
OCR of correct Version "2" exists.

Checking data integrity of OCR...
Data integrity check for OCR passed.

OCR integrity check passed.

Checking CRS integrity...

Checking daemon liveness...
Liveness check passed for "CRS daemon".

Checking daemon liveness...
Liveness check passed for "CSS daemon".

Checking daemon liveness...
Liveness check passed for "EVM daemon".

Checking CRS health...
CRS health check passed.

CRS integrity check passed.

Checking node application existence...


Checking existence of VIP node application (required)
Check passed.

Checking existence of ONS node application (optional)
Check passed.

Checking existence of GSD node application (optional)
Check passed.


Post-check for cluster services setup was successful.
rac3->

三、oracle软件的安装：
1、在rac1上用xmanager先export DISPLAY，再执行$ORACLE_HOME/oui/bin/addNode.sh
继续看图说话：）

[root@rac3 rpm]# sh /u01/app/oracle/product/10.2.0/db_1/root.sh
Running Oracle10 root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.

[root@rac3 rpm]#

2、在rac3上，用root执行/u01/app/oracle/product/10.2.0/crs_1/bin/vipca -nodelist rac1,rac2,rac3 检查vip配置是否正确

3、配置rac3的侦听
在rac3上用oracle用户运行netca:

四、在rac3上，instance的创建：
1、检查资源组状态，开始在rac3上建实例：

rac2-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE ONLINE rac2
ora....b1.inst application ONLINE ONLINE rac1
ora....b2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE ONLINE rac3
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
rac2->

2、在rac1上运行dbca：

点YES

点NO，完成安装。

至此，已经全部安装完成，在crs_stat -t能看到各个资源已经online，且在数据库里面查询gv$的视图可以看到各个实例的情况。

另外，由于rac3节点是从rac1复制过来的，rac3的环境就没有再重新设定一遍，包括安装asm,ocfs2包，oracle用户及目录，节点时间的同步，系统环境变量等。

我测试的结果如下：

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.devdb.db application ONLINE ONLINE rac3
ora....b1.inst application ONLINE ONLINE rac1
ora....b2.inst application ONLINE ONLINE rac2
ora....b3.inst application ONLINE ONLINE rac3
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
ora....SM3.asm application ONLINE ONLINE rac3
ora....C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application ONLINE ONLINE rac3
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip application ONLINE ONLINE rac3
[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jul 26 09:55:48 2012

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> select instance_name from gv$instance;

INSTANCE_NAME
----------------
devdb2
devdb1
devdb3

SQL>

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/271283/viewspace-1059013/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/271283/viewspace-1059013/