RHEL6.5_Oracle_11.2.0.4 RAC 删除和添加节点

一、RAC中删除和保留节点信息

保留节点： 节点1
删除的节点： 节点2
重新添加的节点： 节点2

二、 删除节点
1、 保留节点备份OCR

ocrconfig -showbackup;
ocrconfig -manualbackup;
ocrconfig -local -manualbackup;
ocrconfig -export "/u01/app/11.2.0.4/grid/cdata/aaa/lgc_$(date +%Y-%m-%d_%H:%M:%S).ocr"

2、 保留节点备份CRS网格资料

cp /u01/app/11.2.0.4/grid/gpnp/aaa-node1/profiles/peer/profile.xml /u01/app/11.2.0.4/grid/gpnp/aaa-node1/profiles/peer/profile.xml_bak

3、保留节点备份数据库

对RAC环境执行重大操作前，应备份数据库

run{
allocate channel c1 device type disk;
allocate channel c2 device type disk;
backup as compressed backupset incremental level 0 database;
sql 'alter system archive log current';
sql 'alter system archive log current';
sql 'alter system archive log current';
crosscheck archivelog all;
delete noprompt expired archivelog all;
backup as compressed backupset archivelog all;
backup current controlfile;
release channel c1;
release channel c2;
}

4、删除实例

–删除实例前需要注意的事

Note: Before deleting an instance from an Oracle RAC database, use
either SRVCTL or Oracle Enterprise Manager to do the following:
■ If you have services configured, relocate the services
■ Modify the services so that each service can run on one of the
remaining instances
■ Set "not used" for each service running on the instance that is to be
deleted

如果是正常删除节点，那么在要删除的节点上执行操作

sqlplus / as sysdba
shutdown immediate

如果是节点损坏，则没有必要进行该操作

1）、数据库禁用节点2线程

在保留节点上执行：

alter database disable thread 2;

SYS@DGPRI1>select thread#,status,enabled from v$thread;

   THREAD# STATUS	      ENABLED
---------- ------------------ ------------------------
	 1 OPEN 	      PUBLIC
	 2 CLOSED	      PUBLIC

SYS@DGPRI1>alter database disable thread 2;

Database altered.

SYS@DGPRI1>select thread#,status,enabled from v$thread;

   THREAD# STATUS	      ENABLED
---------- ------------------ ------------------------
	 1 OPEN 	      PUBLIC
	 2 CLOSED	      DISABLED

2）、查看集群节点信息

在存活节点查看节点信息是否为unpin状态，如果不是使用下面命令，将节点设置为unpin状态

[oracle@aaa-node1 ~]$ olsnodes -n -s -t
aaa-node1	1	Active	Unpinned
aaa-node2	2	Inactive	Unpinned

crsctl unpin css -n aaa-node2

如果要删除的节点ASM实例长时间不是集群的一部分，移除ASM

If this node had an ASM instance and the node will no longer be a part of the cluster, you must remove the ASM instance by issuing the following commands:

srvctl stop asm -n node_name
srvctl remove asm -n node_name

Verify that ASM has been removed by issuing the following command:

srvctl config asm -n node_name

[oracle@aaa-node1 dbca]$ srvctl config asm -n aaa-node1
Warning:-n option has been deprecated and will be ignored.
ASM home: /u01/app/11.2.0.4/grid
ASM listener: LISTENER
[oracle@aaa-node1 dbca]$ srvctl config asm -n aaa-node2
PRKO-2006 : Invalid node name: aaa-node2

3）、保留节点上执行删除实例命令

执行前确保sys密码正确

sqlplus sys/oracle@192.168.59.110:1521/xxxdgpri as sysdba;

dbca -silent -deleteInstance \
 -nodeList aaa-node2 \
 -gdbName XXXDGPRI \
 -instanceName DGPRI2 \
 -sysDBAUserName sys \
 -sysDBAPassword oracle \

-gdbName  db_unique_name(srvctl -d 中指定的参数)

–检查CRS中DB信息的改变

[oracle@aaa-node1 ~]$ srvctl config db -d xxxdgpri;
Database unique name: XXXDGPRI
Database name: XXXDG
Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_1
Oracle user: oracle
Spfile: +DATA/XXXDG/spfileXXXDG.ora
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: XXXDGPRI
Database instances: DGPRI1
Disk Groups: DATA,FRA
Mount point paths: 
Services: 
Type: RAC
Database is administrator managed

4）、查询活动的实例

SYS@DGPRI1>col instance format a10;
SYS@DGPRI1>select thread#,status,enabled,instance from v$thread;

   THREAD# STATUS	      ENABLED		       INSTANCE
---------- ------------------ ------------------------ ----------
	 1 OPEN 	      PUBLIC		       DGPRI1

5、 RAC层面删除节点(删除Oracle软件)

Delete Nodes from the Cluster

1）Stop the node applications on the node you are deleting.

As the root user, run the srvctl stop nodeapps command to stop and remove the nodeapps on the node you are deleting. For example:

# srvctl stop nodeapps -n nodename

2）Remove the listener from the node.

If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NETCA to remove this listener. If necessary, re-create this listener in another home. Invoke NETCA and proceed as follows:

Choose Cluster Configuration.

Select only the node you are removing and click Next.

Choose Listener Configuration and click Next.

Choose Delete and delete any listeners configured on the node you are removing.

1)、保留节点上删除节点2的NodeApps

srvctl stop nodeapps -n aaa-node2 –f

2）、保留节点上停用节点2监听

srvctl status listener;
srvctl stop listener -n aaa-node2;
srvctl disable listener -n aaa-node2;

如果是正常删除节点，需要在被删除节点上执行以下操作，损坏则不需要

			
a）删除节点更新中央目录

$ORACLE_HOME/oui/bin/runInstaller -updateNodeList \
 ORACLE_HOME=$ORACLE_HOME \
 "CLUSTER_NODES=aaa-node2" \
 -local \

--CLUSTER_NODES 删除的节点名

b）删除节点卸载ORACLE软件

su – oracle
$ORACLE_HOME/deinstall/deinstall –local

或者

$ORACLE_HOME/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={Oracle_home}" -local

注意：卸载ORACLE软件要指定-local，不然的话将卸载整个集群的ORACLE软件

3）、保留节点移除nodeapps

srvctl remove nodeapps -n aaa-node2;

4）、保留节点上更新中央目录

su - oracle

$ORACLE_HOME/oui/bin/runInstaller -updateNodeList \
 ORACLE_HOME=$ORACLE_HOME \
 "CLUSTER_NODES=aaa-node1" \

相关帮助

$ORACLE_HOME/oui/bin/runInstaller -help

-updateNodeList  
	For updating node list for this home in the OUI inventory.

更新完中央目录的信息为：

[oracle@aaa-node1 ContentsXML]$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0.4/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="aaa-node1"/>
      <NODE NAME="aaa-node2"/>
   </NODE_LIST>
</HOME>

--更新了这个一行，如果节点2是损坏的.. 没有相关文件则手动删除
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0.4/dbhome_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="aaa-node1"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraHome1" LOC="/u01/app/ogg_src" TYPE="O" IDX="3"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

6、GI层面删除节点(删除GI软件)

1）、删除节点2 VIP服务

srvctl stop vip -i aaa-node2-vip -f
srvctl remove vip -i aaa-node2-vip -f

如果正常删除节点需执行以下操作，损坏则不需要

a) 禁用集群应用和守护进程

删除节点上执行

su - root
$ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force;

2）、保留节点上删除节点2

[root@aaa-node1 ~]# olsnodes -n -t -s
aaa-node1	1	Active	Unpinned
aaa-node2	2	Inactive	Unpinned
[root@aaa-node1 ~]# crsctl delete node -n aaa-node2
CRS-4661: Node aaa-node2 successfully deleted.
[root@aaa-node1 ~]# olsnodes -n -t -s
aaa-node1	1	Active	Unpinned

如果正常删除节点需在删除节点上执行以下操作，损坏则不需要

a) 被删除节点更新节点信息

su - grid
$ORACLE_HOME/oui/bin/runInstaller -silent \
 -updateNodeList \
 ORACLE_HOME=$ORACLE_HOME \
 "CLUSTER_NODES=aaa-node2" \
 CRS=true \
 -local \

b) 卸载GI

su - grid
$ORACLE_HOME/deinstall/deinstall -local;

root脚本只能在被删除的节点上执行，千万不能在保留节点上执行.

注意：如果不指定-local选项，那么默认将会把所有的集群信息全部删除，这是非常危险的操作
运行此命令过程中需要进行多次手动配置，请注意！！！！

3）、保留节点跟新中央目录

su - grid

$ORACLE_HOME/oui/bin/runInstaller \
 -silent \
 -updateNodeList \
 ORACLE_HOME=$ORACLE_HOME \
 "CLUSTER_NAME=aaa-node1" \
 CRS=true \

更新后的inventory.xml

[grid@aaa-node1 ContentsXML]$ cat inventory.xml 
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>

--好像没删除，是节点2已经损坏不能执行运行卸载脚本，手动删除

<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0.4/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="aaa-node1"/>
      <NODE NAME="aaa-node2"/>--- 删除这一行..
   </NODE_LIST>
</HOME>

<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0.4/dbhome_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="aaa-node1"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraHome1" LOC="/u01/app/ogg_src" TYPE="O" IDX="3"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

7、验证节点2是否被删除

su - grid
cluvfy stage -post nodedel -n aaa-node2;

[grid@aaa-node1 ContentsXML]$ cluvfy stage -post nodedel -n aaa-node2

Performing post-checks for node removal 

Checking CRS integrity...

Clusterware version consistency passed

CRS integrity check passed

Node removal check passed

Post-check for node removal was successful. 

至此，集群节点2删除完毕，删除过程中因为节点2重新安装更新集群信息时会报错，可以手动删除相关信息…

三、增加节点
1、 添加前检查

确保新安装的OS环境和保留节点一致

1、存储，心跳线互联
2、系统内核、依赖包、核心参数、/etc/hosts保持一致
3、用户oracle和用户grid 组、用户、ID(gid,uid)保持一致
4、目录、环境变量保持一致
5、添加节点前检查

cluvfy stage -pre nodeadd -n aaa-node2 -verbose -fixup;
cluvfy stage -post hwos -n aaa-node2 -verbose;

2、 添加GI软件

1）、保留节点上执行

export IGNORE_PREADDNODE_CHECKS=Y

su - grid
$ORACLE_HOME/oui/bin/addNode.sh -silent \
 "CLUSTER_NEW_NODES={aaa-node2}" \
 "CLUSTER_NEW_VIRTUAL_HOSTNAMES={aaa-node2-vip}" \
 "CLUSTER_NEW_PRIVATE_HOSTNAMES={aaa-node2-priv}" \

2）、新增节点上执行

/u01/app/oraInventory/oraInstRoot.sh
/u01/app/11.2.0.4/grid/root.sh

3）、检查是否成功添加GI

cluvfy stage -post nodeadd -n aaa-node2 –verbose

3、添加ORACLE软件

1）、保留节点上执行添加数据库软件命令

$ORACLE_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={aaa-node2}"

2）、新增节点执行root脚本

/u01/app/oracle/product/11.2.0.4/dbhome_1/root.sh

4、 添加实例

1）、执行添加实例命令

dbca -silent -addInstance \
 -nodeList aaa-node2 \
 -gdbName xxx \
 -instanceName xxx2 \
 -sysDBAUserName sys \
 -sysDBAPassword oracle \

2）、检查实例

srvctl config db -d xxx;
select thread#,status,enabled,instance from v$thread;

四、遇到的问题

1、删除实例时遇到以下错误

The nodes "[aaa-node2]" are not part of the cluster. Make sure clusterware is active on these nodes before proceeding.

--加节点2添加到CRS
crsctl unpin css -n aaa-node2

       如果节点2是故障及节点，olsnodes -n -t –s可能会找不到节点2，则可以先删除ORACLE软件和GI软件，然后添加GI软件后(这样olsnodes -n -t -s下就能找到节点2)再删除实例，再删除GI软件，然后重新添加GI软件(没有删除实例前添加GI软件可能会报错)

The instance name DGPRI2 does not exist in the cluster database. Supply a valid instance name and retry the operation.

srvctl中删除了节点2的信息，增加回去

2、添加GI软件时报错

Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
SEVERE:Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.

--执行以下脚本跟新inventory.xml

[grid@aaa-node1 ContentsXML]$ $ORACLE_HOME/oui/bin/detachHome.sh 
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 7874 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
The Oracle home '/u01/app/11.2.0.4/grid' could not be updated as it does not exist.
'DetachHome' failed.
[grid@aaa-node1 ContentsXML]$ $ORACLE_HOME/oui/bin/attachHome.sh
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 7874 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'AttachHome' was successful.

$ORACLE_HOME/oui/bin/runInstaller -silent \
 -updateNodeList \
 ORACLE_HOME=$ORACLE_HOME \
 "CLUSTER_NODES=aaa-node1" \
 CRS=true \
 -local \