Remove a Node from an Existing Oracle RAC 10g R2 Cluster on Linux
Contents
1. Overview
2. Remove the Oracle Instance on the Node to be Deleted
3. ASM Instance Clean-Up Procedures
4. Remove the Listener From the Node to be Deleted
5. Remove the Oracle Database Software From the Node to be Deleted
6. Remove the Node to be Deleted from Oracle Clusterware
8. Remove Access Permissions on Openfiler for the Node to be Deleted
Although not as exciting as building an Oracle RAC or adding a new node and instance to a clustered database; removing a node from a clustered environment is just as important to understand for a DBA managing Oracle RAC 10g. While it is true that most of the attention in a clustered database environment is focused on extending the database tier to support increased demand, the exact opposite is just as likely to be encountered where the DBA needs to remove a node from an existing RAC environment. It may be that a server is being underutilized in the database cluster and could be better served in another business unit. Another scenario is a node failure. In this case, a node can be removed from the cluster while the remaining nodes continue to service ongoing requests.
This document is an extension to two of my articles: "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". Contained in this new article are the steps required to remove a single node from an already running and configured three-node Oracle RAC 10g Release 2 environment on the CentOS 32-bit (x86) platform. The node being removed is the third node I added in the second article. Although this article was written and tested on CentOS 4.5 Linux, it should work unchanged with Red Hat Enterprise Linux 4 Update 5.
When removing the Oracle Cluster and Oracle Database software, you must first remove the Oracle Database software and then remove the Oracle Clusterware from the node that you are deleting. In other words, you remove the software components from the node you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components. It is important that you perform. each step contained this article in the order provided.
This article assumes the following:
|
| The reader has already built and configured a three-node Oracle RAC 10g Release 2 environment using the articles "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". These two articles provide comprehensive instructions for building a two-node RAC cluster and then adding a third node to the cluster named linux3. All Oracle RAC nodes are single processor machines running CentOS 4.5, Oracle RAC 10g Release 2, OCFS2, and ASMLib 2.0. The current three-node RAC environment actually consists of four machines — three named linux1, linux2, and linux3 which each run an Oracle10g instance and a forth node to run the network storage server named openfiler1. Note: The current three-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to version 10.2.0.3.0 by applying the 5337014 patchset (p5337014_10203_LINUX.zip). The patchset was applied to Oracle Clusterware and the Oracle Database software. I also applied the one-off patchset - "BUNDLE Patch for Base Bug 6000740" (MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle Database software. The procedures for installing both patchsets are not included in any of the parent article(s). |
|
| The third node in the current Oracle RAC environment named linux3 (running the orcl3 instance) will be removed from the cluster making it a two-node cluster. |
|
| Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current three-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software. |
|
| The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system will be included. |
|
| Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to removing the ASM instance from the deleted node, we will also be removing ASMLib from this node. |
|
| During the creation of the existing two-node cluster and the addition of the third node, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster — namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that the oracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely removed from the node to be deleted (linux3). This task will need to be performed from a node that is to remain a part of the RAC cluster database environment (linux1 in this example) using ssh and scp without being prompted for a password. |
|
| All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). While removing the node linux3 from the Oracle RAC cluster, we will also be removing access permissions to the Openfiler server from linux3. |
The following is a conceptual look at what the environment will look like after removing the third Oracle RAC node (linux3) from the cluster. Click on the graphic below to enlarge the image:
Figure 1: Removing linux3 from the current Oracle RAC 10g Release 2 Environment
| While this article provides comprehensive instructions for successfully removing a node from an existing Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, and administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com. Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2) |
Remove the Oracle Instance on the Node to be Deleted
|
Overview
Before removing any of the Oracle software from the node to be deleted (linux3), the first step is to remove the Oracle instance (orcl3) from the RAC database. This task can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA.
The DBCA will need to be run from a node other than the node that hosts the instance you want to delete. The suggested node to run the DBCA from is the same node used to install the original Oracle RAC environment from which for this example is linux1.
Verifying Terminal Shell Environment
Before starting the DBCA, you should first verify you are logged onto the server you will be running the DBCA from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster:
Verify Server and Enable X Server Access
# hostname
linux1
# xhost +
access control disabled, clients can connect from any host
Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=:0.0
$ export DISPLAY
Verify Remote Access / User Equivalence
Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password.
When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform. the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1
$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2
$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3
Remove Database Instance
To start the database instance removal process, run the following from linux1:
$ dbca &
| Screen Name | Response |
| Welcome Screen | Select Oracle Real Application Clusters database. |
| Operations | Select Instance Management. |
| Instance Management | Select Delete an instance. |
| List of cluster databases | Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is orcl. Select this clustered database. At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges": Username: sys Password: Click Next to continue. |
| List of cluster | This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on. Select the node:instance to delete (linux3:orcl3) and Click Next to continue. |
| Database Services | If the current clustered database has any database services defined, the next screen allows the DBA to re-configure those database services to account for the instance being removed. In this example, the existing clustered database has one service defined named orcl_taf. With the "orcl_taf" database service selected, change the details to Not used for the instance to be removed (orcl3) and leave the "TAF Policy" set to Basic. Click Finish to remove the instance. |
| Database Configuration Assistant: Summary | After verifying the instance removal options in the summary dialog, Click OK to begin the instance management process. |
| Database Configuration Assistant | A second dialog box is presented which includes the warning: "The Database Configuration Assistance will delete the Oracle instance and its associated OFA directory structure. All information about this instance will be deleted. Do you want to proceed?" Click OK to acknowledge this dialog. |
| Database Configuration Assistant | A progress bar is display while the instance is being deleted. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform. another operation?" Click No to end and exit the DBCA utility. |
When the Oracle Database Configuration Assistant has completed, the orcl3 instance (and associated services) will have been successfully removed from linux3!
Verify Database Instance and Associated Objects Were Removed
Check Cluster Services
$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
----------------------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE linux1
ora....X1.lsnr application ONLINE ONLINE linux1
ora.linux1.gsd application ONLINE ONLINE linux1
ora.linux1.ons application ONLINE ONLINE linux1
ora.linux1.vip application ONLINE ONLINE linux1
ora....SM2.asm application ONLINE ONLINE linux2
ora....X2.lsnr application ONLINE ONLINE linux2
ora.linux2.gsd application ONLINE ONLINE linux2
ora.linux2.ons application ONLINE ONLINE linux2
ora.linux2.vip application ONLINE ONLINE linux2
ora....SM3.asm application ONLINE ONLINE linux3
ora....X3.lsnr application ONLINE ONLINE linux3
ora.linux3.gsd application ONLINE ONLINE linux3
ora.linux3.ons application ONLINE ONLINE linux3
ora.linux3.vip application ONLINE ONLINE linux3
ora.orcl.db application ONLINE ONLINE linux1
ora....l1.inst application ONLINE ONLINE linux1
ora....l2.inst application ONLINE ONLINE linux2
ora...._taf.cs application ONLINE ONLINE linux2
ora....cl1.srv application ONLINE ONLINE linux1
ora....cl2.srv application ONLINE ONLINE linux2
- or -
$ rac_crs_stat
HA Resource Target State
----------- ------ -----
ora.linux1.ASM1.asm ONLINE ONLINE on linux1
ora.linux1.LISTENER_LINUX1.lsnr ONLINE ONLINE on linux1
ora.linux1.gsd ONLINE ONLINE on linux1
ora.linux1.ons ONLINE ONLINE on linux1
ora.linux1.vip ONLINE ONLINE on linux1
ora.linux2.ASM2.asm ONLINE ONLINE on linux2
ora.linux2.LISTENER_LINUX2.lsnr ONLINE ONLINE on linux2
ora.linux2.gsd ONLINE ONLINE on linux2
ora.linux2.ons ONLINE ONLINE on linux2
ora.linux2.vip ONLINE ONLINE on linux2
ora.linux3.ASM3.asm ONLINE ONLINE on linux3
ora.linux3.LISTENER_LINUX3.lsnr ONLINE ONLINE on linux3
ora.linux3.gsd ONLINE ONLINE on linux3
ora.linux3.ons ONLINE ONLINE on linux3
ora.linux3.vip ONLINE ONLINE on linux3
ora.orcl.db ONLINE ONLINE on linux1
ora.orcl.orcl1.inst ONLINE ONLINE on linux1
ora.orcl.orcl2.inst ONLINE ONLINE on linux2
ora.orcl.orcl_taf.cs ONLINE ONLINE on linux2
ora.orcl.orcl_taf.orcl1.srv ONLINE ONLINE on linux1
ora.orcl.orcl_taf.orcl2.srv ONLINE ONLINE on linux2
Verify Instance we Removed
Login to one of the instances and query the gv$instance view:
SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') from gv$instance order by inst_id;
INST_ID INSTANCE_NAME STATUS TO_CHAR(STARTUP_TIME
------------ ----------------------- -------------- ---------------------------------------
1 orcl1 OPEN 27-FEB-2008 01:26:52
2 orcl2 OPEN 27-FEB-2008 01:27:22
Verify Database Files / Tablespaces have been Removed
Check that the UNDO tablespace that was defined for orcl3 (UNDOTBS3) has been removed:
SQL> select tablespace_name from dba_tablespaces order by 1;
TABLESPACE_NAME
------------------
EXAMPLE
SOE
SOEINDEX
SYSAUX
SYSTEM
TEMP
UNDOTBS1
UNDOTBS2
USERS
Check that the online redo log files that were defined for orcl3 have been removed:
SQL> select inst_id, group#, member from gv$logfile order by 1,2,3;
INST_ID GROUP# MEMBER
------- ------ ---------------------------------------------------------
1 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
1 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
1 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
1 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
1 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
1 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
1 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
1 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299
2 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669
2 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667
2 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675
2 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671
2 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297
2 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295
2 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303
2 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299
Update TNSNAMES
Login to all machines that were accessing the instance removed and update the tnsnames.ora file (if necessary).
ASM Instance Clean-Up Procedures
|
Overview
When the clustered database is running on ASM (as is the case in this example), the steps in this section describe how to remove the ASM instance (and ASMLib) from the node to be deleted. If the Oracle RAC database is not running on ASM, this section can be skipped.
The tasks in this section will need to be run from a node other than the node that hosts the ASM instance you want to delete. The suggested node to run these steps from is the same node used to install the original Oracle RAC environment from which for this example is linux1.
Remove the ASM Instance from the Oracle Cluster Registry File
Run the following from linux1 as the oracle user account:
$ srvctl stop asm -n linux3
$ srvctl remove asm -n linux3
Remove ASM Directories from the Node to be Deleted
Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
$ cd $ORACLE_BASE/admin
$ rm -rf +ASM
$ cd $ORACLE_HOME/dbs
$ rm -f *ASM*
|
Remove ASMLib from the Node to be Deleted
The Linux nodes in the current Oracle RAC database make use of ASMLib so that the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls.
In this section we will remove the ASMLib kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
# /etc/init.d/oracleasm stop
Unmounting ASMlib driver filesystem: [ OK ]
Unloading module "oracleasm": [ OK ]
# rpm -qa | grep oracleasm
oracleasm-support-2.0.3-1
oracleasm-2.6.9-55.EL-2.0.3-1
oracleasmlib-2.0.2-1
# rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/rc.d/init.d/oracleasm
# rm -f /etc/rc0.d/*oracleasm*
# rm -f /etc/rc1.d/*oracleasm*
# rm -f /etc/rc2.d/*oracleasm*
# rm -f /etc/rc3.d/*oracleasm*
# rm -f /etc/rc4.d/*oracleasm*
# rm -f /etc/rc5.d/*oracleasm*
# rm -f /etc/rc6.d/*oracleasm*
Remove the Listener From the Node to be Deleted
|
Overview
Each of the nodes in the current Oracle RAC environment use a node-specific listener named LISTENER_nodename which runs out of the local Oracle home.
In this section, you will use the Network Configuration Assistant (NETCA) to delete the TNS listener for the node to be deleted from the existing clustered configuration. The NETCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Although it is possible to re-create this listener in another Oracle home (on a different node), we will simply be removing it from the cluster.
Verifying Terminal Shell Environment
As was performed earlier, the terminal shell environment needs to be configured for remote access and user equivalence to all Oracle RAC nodes before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform. any of the actions described below with regards to setting up remote access and the DISPLAY variable:
Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=:0.0
$ export DISPLAY
Verify Remote Access / User Equivalence
Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password.
When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform. the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
$ ssh linux1 "date;hostname"
Fri Feb 29 00:08:32 EST 2008
linux1
$ ssh linux2 "date;hostname"
Fri Feb 29 00:10:10 EST 2008
linux2
$ ssh linux3 "date;hostname"
Fri Feb 29 00:07:25 EST 2008
linux3
Run the Network Configuration Assistant
To start the NETCA, run the following from linux1:
$ netca &
The following table walks you through the process of reconfiguring the clustered TNS listeners to remove the node-specific listener from the node to be deleted.
| Screen Name | Response |
| Select the Type of Oracle Net Services Configuration | Select Cluster configuration |
| Select the nodes to configure | Only select the Oracle RAC node to remove the listener from: linux3. |
| Type of Configuration | Select Listener configuration. |
| Listener Configuration Next 4 Screens | The following screens are now like any other normal listener configuration when removing a listener: What do you want to do: Delete Select a listener you want to delete: LISTENER Acknowledge warning dialog: Yes Listener deleted [ Next ] Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. |
| Type of Configuration | Click Finish to exit the NETCA. |
Verify TNS Listener Configuration
The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC cluster (linux1 and linux2):
$ hostname
linux1
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX1
$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr
NAME=ora.linux1.LISTENER_LINUX1.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux1
=====================
$ hostname
linux2
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX2
$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr
NAME=ora.linux2.LISTENER_LINUX2.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on linux2
=====================
$ hostname
linux3
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
$ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
Remove the Oracle Database Software From the Node to be Deleted
|
Overview
In this section, you will be removing the node to be deleted (linux3) from the Node List for its Oracle inventory. Next will be to remove the Oracle Database software from the node to be deleted using the Oracle Universal Installer (OUI). Finally, update the Oracle inventory on all remaining Oracle RAC nodes to reflect the nodes which will remain in the cluster.
Update Node List for Oracle Database Software - (Remove linux3)
From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Database software:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES="" -local
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
|
De-install Oracle Database Software
Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Database software. Make certain that you choose the home to be removed and not just the products under that home.
From linux3 as the oracle user account, run the following:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller
| Screen Name | Response |
| Welcome Screen | Click the Installed Products button. |
| Inventory: Contents Tab | Check the Oracle home to deleted (OraDb10g_home1) and click the Remove button. |
| Confirmation | Acknowledge the warning dialog by clicking Yes. |
| Deinstallation Process | A progress bar is displayed while the Oracle Database software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Database software (Oracle home) was successfully removed, Click Close to exit this dialog. |
| Welcome Screen | Click Cancel to exit the OUI. |
Update Node List for Remaining Nodes in the Cluster
Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={linux1,linux2}"
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
|
Remove the Node to be Deleted from Oracle Clusterware
|
Overview
After successfully removing the Oracle Database software from the node to be deleted, the next major step is to remove the node to be deleted from Oracle Clusterware.
Remove Node-Specific Interface Configuration
Run the following commands to remove node-specific interface configurations from the node to be deleted. For this example, run these commands from linux3 as the oracle user account:
$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200
|
$ $ORA_CRS_HOME/bin/oifcfg delif -node linux3
PROC-4: The cluster registry key to be operated on does not exist.
PRIF-11: cluster registry error
|
Disable Oracle Clusterware Applications
From the node you are deleting from the cluster (linux3), run the script. $ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle Clusterware applications that are on the node. This script. should only be run once. Given the Clusterware software install is on local disk (non-shared), make certain to use the nosharedhome argument. The default for this script. is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user account.
Running this script. will stop the CRS stack and delete the ocr.loc file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file sytem.
While logged into linux3 as the root user account, run the following:
$ su
# cd $ORA_CRS_HOME/install
# ./rootdelete.sh local nosharedvar nosharedhome
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script. for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
Delete Node from Cluster and Update OCR
Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script. to delete the node (linux3) from the Oracle cluster and to update the Oracle Cluster Registry (OCR). This script. should be run from a pre-existing / available node in the cluster (linux1) as the root user account:
Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from linux1:
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
linux3 3
From the listing above, the node number for linux3 is 3.
While logged into linux1 as the root user account, run the following using the name linux3 and the node number 3:
$ su
# cd $ORA_CRS_HOME/install
# ./rootdeletenode.sh linux3,3
CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.
CRS-0210: Could not find resource 'ora.linux3.ons'.
CRS-0210: Could not find resource 'ora.linux3.vip'.
CRS-0210: Could not find resource 'ora.linux3.gsd'.
CRS-0210: Could not find resource ora.linux3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'linux3,3' deleted successfully
To verify that the node was successfully removed, use the following as either the oracle or root user:
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
Update Node List for Oracle Clusterware Software - (Remove linux3)
From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Clusterware software:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES="" -local CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
|
De-install Oracle Clusterware Software
Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Clusterware software. Make certain that you choose the home to be removed and not just the products under that home.
From linux3 as the oracle user account, run the following:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller
| Screen Name | Response |
| Welcome Screen | Click the Installed Products button. |
| Inventory: Contents Tab | Check the Oracle home to deleted (OraCrs10g_home) and click the Remove button. |
| Confirmation | Acknowledge the warning dialog by clicking Yes to remove the Oracle Clusterware software and to remove the /u01/app/crs directory. |
| Deinstallation Process | A progress bar is displayed while the Oracle Clusterware software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Clusterware software (Clusterware home) was successfully removed, Click Close to exit this dialog. |
| Welcome Screen | Click Cancel to exit the OUI. |
Update Node List for Remaining Nodes in the Cluster
Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=:0.0; export DISPLAY
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={linux1,linux2}" CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
|
Verify Node to be Deleted is no Longer a Member of the Cluster
Run the following commands to verify that the node to be deleted from the Oracle RAC cluster is no longer a member of the cluster and to verify that the Oracle Clusterware components have been successfully removed from that node.
Run the following commands from linux1 as the oracle user account:
$ srvctl status nodeapps -n linux3
PRKC-1056 : Failed to get the hostname for node linux3
PRKH-1001 : HASContext Internal Error
[OCR Error(Native: getHostName:[21])]
The error above indicates that linux3 is no longer a member of the cluster.
$ $ORA_CRS_HOME/bin/crs_stat | grep -i linux3
You should not see any output from the above command
$ $ORA_CRS_HOME/bin/olsnodes -n
linux1 1
linux2 2
You should see the present node list without the deleted node (that is linux1 and linux2 only).
Remove/Rename any Remaining Oracle Files from Node to be Deleted
From the node to be deleted (linux3), remove/rename any remaining Oracle files while logged in as the root user account:
# mv -f /etc/inittab.no_crs /etc/inittab
# rm -f /etc/inittab.orig
# rm -f /etc/inittab.crs
# rm -rf /etc/oracle
# rm -f /etc/oratab
# rm -f /etc/oraInst.loc
# rm -rf /etc/ORCLcluster
# rm -rf /u01/app/oracle
# rm -rf /u01/app/crs
# rm -f /usr/local/bin/coraenv
# rm -f /usr/local/bin/dbhome
# rm -f /usr/local/bin/oraenv
Finally, remove the Oracle user account and all associated UNIX groups from linux3:
# userdel -r oracle
# groupdel oinstall
# groupdel dba
|
Overview
The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. It is also used to store the shared SPFILE for ASM. This section includes instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system.
Remove OCFS2 from the Node to be Deleted
In this section we will remove the OCFS2 kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
# umount /u02
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# rpm -qa | grep ocfs2
ocfs2-tools-1.2.4-1
ocfs2-2.6.9-55.EL-1.2.5-6
ocfs2console-1.2.4-1
# rpm -ev ocfs2-tools-1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1
warning: /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave
# rm -f /etc/sysconfig/o2cb.rpmsave
# rm -rf /etc/ocfs2
Remove Node to be Deleted from the OCFS2 Cluster
The next step is to remove the Oracle RAC node being deleted (linux3) from the "live" OCFS2 cluster. This entails running the o2cb_ctl command-line utility from the remaining two RAC nodes linux1 and linux2.
As root, run the following from linux1 and then linux2:
[root@linux1 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported
[root@linux2 ~]# o2cb_ctl -D -u -n linux3
o2cb_ctl: Not yet supported
o2cb_ctl parameters:
-D : Delete an object from the existing OCFS2 Cluster Configuration.
-u : Valid only with -D. When deleting something (node or cluster),
it will also remove it from the live cluster (/config). If the
parameter is not specified, then only update the
/etc/ocfs2/cluster.conf.
-n : Object name which is usually the node name or cluster name.
|
Remove OCFS2 Mount Options from /etc/fstab
Finally, remove the following entry from the /etc/fstab from linux3:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0
Remove Access Permissions on Openfiler for the Node to be Deleted
|
Overview
At this point, linux3 has been successfully removed from the Oracle RAC configuration. Remember that in the existing Oracle RAC configuration that all shared disk storage is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). When we added linux3 to the current Oracle RAC configuration in the article "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)", we needed to configure network access in Openfiler so that linux3 had permissions to the shared iSCSI volumes. We also configured the iSCSI initiator on linux3 to automatically discover all available iSCSI volumes on each boot.
In this final section of the article, we will be revoking network access in Openfiler for the node linux3 and lastly, modifying the iSCSI initiator on linux3 to disable automatic discovery of iSCSI volumes from Openfiler. For the purpose of this example, all iSCSI traffic was configured to use the private network interface eth1 which in this article is on the 192.168.2.0 network.
Revoking Openfiler network access permissions from linux3 can be done using the Openfiler Storage Control Center — a browser based tool over an https connection on port 446. For example:
https://openfiler1:446/
From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:
· Username: openfiler
· Password: password
The first page the administrator sees is the [Accounts] / [Authentication] screen. Configuring user accounts and groups is not necessary for this article and will therefore not be discussed.
Revoke Access from linux3 to each iSCSI Volume
The first step in this process is to remove access from linux3 to each individual iSCSI volume within Openfiler.
The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1 which is used for the existing Oracle RAC configuration.
| iSCSI / Logical Volumes in Volume Group rac1 | |||
| Volume Name | Volume Description | Required Space (MB) | Filesystem Type |
| crs | Oracle Clusterware | 2,048 | iSCSI |
| asm1 | Oracle ASM Volume 1 | 118,720 | iSCSI |
| asm2 | Oracle ASM Volume 2 | 118,720 | iSCSI |
| asm3 | Oracle ASM Volume 3 | 118,720 | iSCSI |
| asm4 | Oracle ASM Volume 4 | 118,720 | iSCSI |
To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. There we will see all five logical volumes within the volume group rac1:
Figure 2: Current Logical (iSCSI) Volumes
For each of the five logical volumes, click on the 'Edit' link (under the Properties column). This will bring up the 'Edit properties' screen for that volume. Scroll to the bottom of this screen; change the access for host linux3-priv from 'Allow' to 'Deny' and click the 'Update' button. Perform. this task for all five logical volumes.
Remove linux3 from Openfiler Local Network Configuration
The next step is to modify the local network configuration within Openfiler by removing the node linux3 (which was actually entered as node linux3-priv).
This task can be completed using the Openfiler Storage Control Center by navigating to [General] / [Local Networks]. The Local Networks screen allows an administrator to setup (add/remove) networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we want to remove linux3 from the Openfiler local network configuration.
Simply, check the linux3-priv / 192.168.2.107 row and click the [Update] button at the bottom of the screen to remove linux3 from the Openfiler local network configuration.
The following image shows the step required to remove linux3 and update the local network configuration within Openfiler:
Disable iSCSI Initiator Service on linux3
The final step in this process is to modify the iSCSI initiator service on linux3 so it will not automatically start and therefore will not attempt to discover iSCSI volumes from the Openfiler server.
First, stop the iscsi initiator service on linux3:
# service iscsi stop
Searching for iscsi-based multipath maps
Found 0 maps
Stopping iscsid: [ OK ]
Removing iscsi driver: [ OK ]
Next, edit the file /etc/iscsi.conf on linux3 and comment out (or remove) the DiscoveryAddress entry which specifies the hostname of the Openfiler network storage server. In our example, use a # at the beginning of DiscoveryAddress entry as shown below:
...
# DiscoveryAddress=openfiler1-priv
...
Finally, configure the iSCSI service to not be active across machine reboots on linux3. The Linux command chkconfig can be used to achieve that as follows:
# chkconfig --level 345 iscsi
$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=RAC01,RAC02" -local
$ORA_CRS_HOME/bin/racgons remove_config RAC03:6200
srvctl remove nodeapps -n RAC03
$ORA_CRS_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES=RAC01,RAC02" CRS=TRUE
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/24867586/viewspace-714115/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/24867586/viewspace-714115/
6334
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
