Delete a Node from Oracle RAC 11gR2(11.2.0.3) on Oracle Linux 6
This guide shows how to add a node to an existing 11gR2(11.2.0.3) Oracle RAC cluster. It is assumed that the node in question is available and is not part of a GNS/Grid Plug and Play cluster. In other words, the database is considered to be "Administrator-Managed". Also, the database software is non-shared and uses role separation where "grid" is the clusterware owner and "oracle" owns the database software. This guide uses a 2-node cluster running Oracle Linux 6.3 (x64). There are two pre-existing nodes "node1" and "node2". We will be adding "node3" to the cluster. This guide does not cover node preparation steps/prerequisites. The assumption is that since there is a pre-existing cluster the user knows how to prepare a node – from a prerequisite perspective – for cluster addition.
From an existing node, run "cluvfy" to ensure that "node3" – the cluster node to be added – is ready from a hardware and operating system perspective.
cluvfy stage -post hwos -n node3
The cluster verify utility – "cluvfy" – is used to determine the integrity of the cluster and whether it is ready for a new node. From an existing node, run "cluvfy" to verify the integrity of the cluster.
cluvfy stage -pre nodeadd -n node3 -fixup -verbose
[grid@node1 ~]$ cluvfy stage -pre crsinst -n node1,node2,node3 -fixup -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "node1"
Destination Node Reachable?
------------------------------------ ------------------------
node3 yes
node2 yes
node1 yes
Result: Node reachability check passed from node "node1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
Verification of the hosts config file successful
Interface information for node "node3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.53 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:B5:7E:7A 1500
eth1 10.10.10.53 10.10.0.0 0.0.0.0 192.168.1.254 08:00:27:73:27:02 1500
Interface information for node "node2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.52 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:B1:3A:93 1500
eth0 192.168.1.152 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:B1:3A:93 1500
eth1 10.10.10.52 10.10.0.0 0.0.0.0 192.168.1.254 08:00:27:4C:3F:43 1500
eth1 169.254.95.34 169.254.0.0 0.0.0.0 192.168.1.254 08:00:27:4C:3F:43 1500
Interface information for node "node1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.51 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:7E:5C:93 1500
eth0 192.168.1.151 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:7E:5C:93 1500
eth0 192.168.1.59 192.168.0.0 0.0.0.0 192.168.1.254 08:00:27:7E:5C:93 1500
eth1 10.10.10.51 10.10.0.0 0.0.0.0 192.168.1.254 08:00:27:3C:19:3F 1500
eth1 169.254.72.131 169.254.0.0 0.0.0.0 192.168.1.254 08:00:27:3C:19:3F 1500
Check: Node connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node3[192.168.1.53] node2[192.168.1.52] yes
node3[192.168.1.53] node2[192.168.1.152] yes
node3[192.168.1.53] node1[192.168.1.51] yes
node3[192.168.1.53] node1[192.168.1.151] yes
node3[192.168.1.53] node1[192.168.1.59] yes
node2[192.168.1.52] node2[192.168.1.152] yes
node2[192.168.1.52] node1[192.168.1.51] yes
node2[192.168.1.52] node1[192.168.1.151] yes
node2[192.168.1.52] node1[192.168.1.59] yes
node2[192.168.1.152] node1[192.168.1.51] yes
node2[192.168.1.152] node1[192.168.1.151] yes
node2[192.168.1.152] node1[192.168.1.59] yes
node1[192.168.1.51] node1[192.168.1.151] yes
node1[192.168.1.51] node1[192.168.1.59] yes
node1[192.168.1.151] node1[192.168.1.59] yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) node3,node2,node1
Check: TCP connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node1:192.168.1.51 node3:192.168.1.53 passed
node1:192.168.1.51 node2:192.168.1.52 passed
node1:192.168.1.51 node2:192.168.1.152 passed
node1:192.168.1.51 node1:192.168.1.151 passed
node1:192.168.1.51 node1:192.168.1.59 passed
Result: TCP connectivity check passed for subnet "192.168.0.0"
Check: Node connectivity of subnet "10.10.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node3[10.10.10.53] node2[10.10.10.52] yes
node3[10.10.10.53] node1[10.10.10.51] yes
node2[10.10.10.52] node1[10.10.10.51] yes
Result: Node connectivity passed for subnet "10.10.0.0" with node(s) node3,node2,node1
Check: TCP connectivity of subnet "10.10.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node1:10.10.10.51 node3:10.10.10.53 passed
node1:10.10.10.51 node2:10.10.10.52 passed
Result: TCP connectivity check passed for subnet "10.10.0.0"
Check: Node connectivity of subnet "169.254.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node2[169.254.95.34] node1[169.254.72.131] yes
Result: Node connectivity passed for subnet "169.254.0.0" with node(s) node2,node1
Check: TCP connectivity of subnet "169.254.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node1:169.254.72.131 node2:169.254.95.34 passed
Result: TCP connectivity check passed for subnet "169.254.0.0"
Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
node3 eth0:192.168.1.53
node2 eth0:192.168.1.52 eth0:192.168.1.152
node1 eth0:192.168.1.51 eth0:192.168.1.151 eth0:192.168.1.59
Interfaces found on subnet "10.10.0.0" that are likely candidates for a private interconnect are:
node3 eth1:10.10.10.53
node2 eth1:10.10.10.52
node1 eth1:10.10.10.51
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "10.10.0.0".
Subnet mask consistency check passed for subnet "169.254.0.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "10.10.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.10.0.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Node Name Status
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
Result: Check for ASMLib configuration passed.
Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 1.9599GB (2055112.0KB) 1.5GB (1572864.0KB) passed
node2 1.9599GB (2055112.0KB) 1.5GB (1572864.0KB) passed
node1 1.9599GB (2055112.0KB) 1.5GB (1572864.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 1.7953GB (1882508.0KB) 50MB (51200.0KB) passed
node2 975.3867MB (998796.0KB) 50MB (51200.0KB) passed
node1 940.7617MB (963340.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 4GB (4194300.0KB) 2.9399GB (3082668.0KB) passed
node2 4GB (4194300.0KB) 2.9399GB (3082668.0KB) passed
node1 4GB (4194300.0KB) 2.9399GB (3082668.0KB) passed
Result: Swap space check passed
Check: Free disk space for "node3:/u01/app/11.2.0/grid,node3:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/11.2.0/grid node3 / 71.0596GB 8.5GB passed
/tmp node3 / 71.0596GB 8.5GB passed
Result: Free disk space check passed for "node3:/u01/app/11.2.0/grid,node3:/tmp"
Check: Free disk space for "node2:/u01/app/11.2.0/grid,node2:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/11.2.0/grid node2 / 62.9854GB 8.5GB passed
/tmp node2 / 62.9854GB 8.5GB passed
Result: Free disk space check passed for "node2:/u01/app/11.2.0/grid,node2:/tmp"
Check: Free disk space for "node1:/u01/app/11.2.0/grid,node1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/11.2.0/grid node1 / 56.2256GB 8.5GB passed
/tmp node1 / 56.2256GB 8.5GB passed
Result: Free disk space check passed for "node1:/u01/app/11.2.0/grid,node1:/tmp"
Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
node3 passed exists(1100)
node2 passed exists(1100)
node1 passed exists(1100)
Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "grid"
Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
node3 passed exists
node2 passed exists
node1 passed exists
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
node3 passed exists
node2 passed exists
node1 passed exists
Result: Group existence check passed for "dba"
Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Status
---------------- ------------ ------------ ------------ ------------ ------------
node3 yes yes yes yes passed
node2 yes yes yes yes passed
node1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
Check: Membership of user "grid" in group "dba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
node3 yes yes yes passed
node2 yes yes yes passed
node1 yes yes yes passed
Result: Membership check for user "grid" in group "dba" passed
Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
node3 3 3,5 passed
node2 3 3,5 passed
node1 5 3,5 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node3 hard 65536 65536 passed
node2 hard 65536 65536 passed
node1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node3 soft 1024 1024 passed
node2 soft 1024 1024 passed
node1 soft 1024 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node3 hard 16384 16384 passed
node2 hard 16384 16384 passed
node1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node3 soft 2047 2047 passed
node2 soft 2047 2047 passed
node1 soft 2047 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 x86_64 x86_64 passed
node2 x86_64 x86_64 passed
node1 x86_64 x86_64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 2.6.39-200.24.1.el6uek.x86_64 2.6.32 passed
node2 2.6.39-200.24.1.el6uek.x86_64 2.6.32 passed
node1 2.6.39-200.24.1.el6uek.x86_64 2.6.32 passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 250 250 250 passed
node2 250 250 250 passed
node1 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 32000 32000 32000 passed
node2 32000 32000 32000 passed
node1 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 100 100 100 passed
node2 100 100 100 passed
node1 100 100 100 passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 128 128 128 passed
node2 128 128 128 passed
node1 128 128 128 passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 4398046511104 4398046511104 1052217344 passed
node2 4398046511104 4398046511104 1052217344 passed
node1 4398046511104 4398046511104 1052217344 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 4096 4096 4096 passed
node2 4096 4096 4096 passed
node1 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 1073741824 1073741824 2097152 passed
node2 1073741824 1073741824 2097152 passed
node1 1073741824 1073741824 2097152 passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 6815744 6815744 6815744 passed
node2 6815744 6815744 6815744 passed
node1 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
node2 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
node1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 262144 262144 262144 passed
node2 262144 262144 262144 passed
node1 262144 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 4194304 4194304 4194304 passed
node2 4194304 4194304 4194304 passed
node1 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 262144 262144 262144 passed
node2 262144 262144 262144 passed
node1 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 1048576 1048576 1048576 passed
node2 1048576 1048576 1048576 passed
node1 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node3 1048576 1048576 1048576 passed
node2 1048576 1048576 1048576 passed
node1 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 binutils-2.20.51.0.2-5.34.el6 binutils-2.20.51.0.2 passed
node2 binutils-2.20.51.0.2-5.34.el6 binutils-2.20.51.0.2 passed
node1 binutils-2.20.51.0.2-5.34.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
node2 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
node1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
node2 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
node1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 libgcc(x86_64)-4.4.6-4.el6 libgcc(x86_64)-4.4.4 passed
node2 libgcc(x86_64)-4.4.6-4.el6 libgcc(x86_64)-4.4.4 passed
node1 libgcc(x86_64)-4.4.6-4.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 libstdc++(x86_64)-4.4.6-4.el6 libstdc++(x86_64)-4.4.4 passed
node2 libstdc++(x86_64)-4.4.6-4.el6 libstdc++(x86_64)-4.4.4 passed
node1 libstdc++(x86_64)-4.4.6-4.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 libstdc++-devel(x86_64)-4.4.6-4.el6 libstdc++-devel(x86_64)-4.4.4 passed
node2 libstdc++-devel(x86_64)-4.4.6-4.el6 libstdc++-devel(x86_64)-4.4.4 passed
node1 libstdc++-devel(x86_64)-4.4.6-4.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
node2 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
node1 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 gcc-4.4.6-4.el6 gcc-4.4.4 passed
node2 gcc-4.4.6-4.el6 gcc-4.4.4 passed
node1 gcc-4.4.6-4.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 gcc-c++-4.4.6-4.el6 gcc-c++-4.4.4 passed
node2 gcc-c++-4.4.6-4.el6 gcc-c++-4.4.4 passed
node1 gcc-c++-4.4.6-4.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 ksh-20100621-16.el6 ksh-20100621 passed
node2 ksh-20100621-16.el6 ksh-20100621 passed
node1 ksh-20100621-16.el6 ksh-20100621 passed
Result: Package existence check passed for "ksh"
Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 make-3.81-20.el6 make-3.81 passed
node2 make-3.81-20.el6 make-3.81 passed
node1 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 glibc(x86_64)-2.12-1.80.el6_3.6 glibc(x86_64)-2.12 passed
node2 glibc(x86_64)-2.12-1.80.el6_3.6 glibc(x86_64)-2.12 passed
node1 glibc(x86_64)-2.12-1.80.el6_3.6 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 glibc-devel(x86_64)-2.12-1.80.el6_3.6 glibc-devel(x86_64)-2.12 passed
node2 glibc-devel(x86_64)-2.12-1.80.el6_3.6 glibc-devel(x86_64)-2.12 passed
node1 glibc-devel(x86_64)-2.12-1.80.el6_3.6 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
node2 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
node1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node3 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
node2 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
node1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
node3 passed does not exist
node2 passed does not exist
node1 passed does not exist
Result: User "grid" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
node3 0022 0022 passed
node2 0022 0022 passed
node1 0022 0022 passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Check: Time zone consistency
Result: Time zone consistency check passed
Pre-check for cluster services setup was successful.
Extend Clusterware
The clusterware software will be extended to the new node.
Run "addNode.sh"
From an existing node, run "addNode.sh" to extend the clusterware to the new node "node3":
[grid@node1 ~]$ export IGNORE_PREADDNODE_CHECKS=Y
[grid@node1 ~]$ cd $ORACLE_HOME/oui/bin
[grid@node1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={node3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node3-vip}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={node3-priv}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 5667 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes node2,node3 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/11.2.0/grid
New Nodes
Space Requirements
New Nodes
node3
/: Required 4.31GB : Available 82.72GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.4
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Server) 11.2.0.3.0
Installation Plugin Files 11.2.0.3.0
Universal Storage Manager Files 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Automatic Storage Management Assistant 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.3.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Perl Interpreter 5.10.0.0.2
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Oracle Net Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Enterprise Manager plugin Common Files 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.3.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Cluster Verification Utility Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle LDAP administration 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
PL/SQL 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Cluster Ready Services Files 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Friday, January 4, 2013 11:19:46 AM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Friday, January 4, 2013 11:19:48 AM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Friday, January 4, 2013 11:25:19 AM CST)
. 100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'node3'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oraInventory/orainstRoot.sh #On nodes node3
/u01/app/11.2.0/grid/root.sh #On nodes node3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
[grid@node1 bin]$
Run the "root.sh" commands on the new node as directed
[root@node3 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@node3 ~]#
[root@node3 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
If successful, clusterware daemons, the listener, the ASM instance, etc. should be started by the "root.sh" script
[grid@node3 ~]$ ps -ef|grep grid|grep -v grep
root 1872 1785 0 Jan04 pts/0 00:00:00 su - grid
grid 1873 1872 0 Jan04 pts/0 00:00:00 -bash
root 11348 1 1 09:16 ? 00:00:11 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
grid 11961 1 0 09:18 ? 00:00:02 /u01/app/11.2.0/grid/bin/oraagent.bin
grid 11973 1 0 09:18 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid 11986 1 0 09:18 ? 00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 12003 1 0 09:18 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
grid 12005 1 0 09:18 ? 00:00:02 /u01/app/11.2.0/grid/bin/gipcd.bin
root 12026 1 0 09:18 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
grid 12051 1 1 09:18 ? 00:00:08 /u01/app/11.2.0/grid/bin/ocssd.bin
root 12104 1 0 09:19 ? 00:00:05 /u01/app/11.2.0/grid/bin/orarootagent.bin
root 12116 1 0 09:19 ? 00:00:02 /u01/app/11.2.0/grid/bin/octssd.bin
root 12138 1 1 09:19 ? 00:00:09 /u01/app/11.2.0/grid/bin/osysmond.bin
grid 12232 1 0 09:20 ? 00:00:00 asm_pmon_+ASM3
grid 12234 1 0 09:20 ? 00:00:00 asm_psp0_+ASM3
grid 12236 1 8 09:20 ? 00:00:47 asm_vktm_+ASM3
grid 12240 1 0 09:20 ? 00:00:00 asm_gen0_+ASM3
grid 12242 1 0 09:20 ? 00:00:00 asm_diag_+ASM3
grid 12244 1 0 09:20 ? 00:00:00 asm_ping_+ASM3
grid 12246 1 0 09:20 ? 00:00:02 asm_dia0_+ASM3
grid 12248 1 0 09:20 ? 00:00:02 asm_lmon_+ASM3
grid 12250 1 0 09:20 ? 00:00:01 asm_lmd0_+ASM3
grid 12252 1 0 09:20 ? 00:00:03 asm_lms0_+ASM3
grid 12256 1 0 09:20 ? 00:00:00 asm_lmhb_+ASM3
grid 12258 1 0 09:20 ? 00:00:00 asm_mman_+ASM3
grid 12260 1 0 09:20 ? 00:00:00 asm_dbw0_+ASM3
grid 12262 1 0 09:20 ? 00:00:00 asm_lgwr_+ASM3
grid 12264 1 0 09:20 ? 00:00:00 asm_ckpt_+ASM3
grid 12266 1 0 09:20 ? 00:00:00 asm_smon_+ASM3
grid 12268 1 0 09:20 ? 00:00:00 asm_rbal_+ASM3
grid 12270 1 0 09:20 ? 00:00:00 asm_gmon_+ASM3
grid 12272 1 0 09:20 ? 00:00:00 asm_mmon_+ASM3
grid 12274 1 0 09:20 ? 00:00:00 asm_mmnl_+ASM3
grid 12282 1 0 09:20 ? 00:00:00 asm_lck0_+ASM3
grid 12284 1 0 09:20 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
root 12292 1 1 09:20 ? 00:00:07 /u01/app/11.2.0/grid/bin/crsd.bin reboot
grid 12311 1 0 09:20 ? 00:00:02 /u01/app/11.2.0/grid/bin/evmd.bin
grid 12314 1 0 09:20 ? 00:00:00 oracle+ASM3_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12320 1 0 09:20 ? 00:00:00 asm_asmb_+ASM3
grid 12322 1 0 09:20 ? 00:00:00 oracle+ASM3_asmb_+asm3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12413 12311 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
grid 12456 1 0 09:21 ? 00:00:01 /u01/app/11.2.0/grid/bin/oraagent.bin
root 12460 1 0 09:21 ? 00:00:02 /u01/app/11.2.0/grid/bin/orarootagent.bin
grid 12493 1 0 09:21 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12502 1 0 09:21 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12505 1 0 09:21 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12551 1 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 12552 12551 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 13032 1 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 13288 1873 0 09:29 pts/0 00:00:00 ps -ef
[grid@node3 ~]$ crs_stat -t -v -c node3
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE node3
ora....E3.lsnr application 0/5 0/0 ONLINE ONLINE node3
ora.node3.ons application 0/3 0/0 ONLINE ONLINE node3
ora.node3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE node3
[grid@node3 ~]$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.CRS.dg ora....up.type 0/5 0/ ONLINE ONLINE node1
ora.DATADG.dg ora....up.type 0/5 0/ ONLINE ONLINE node1
ora.FRADG.dg ora....up.type 0/5 0/ ONLINE ONLINE node1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE node1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE node1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE node1
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE node1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE node1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE node1
ora....E1.lsnr application 0/5 0/0 ONLINE ONLINE node1
ora.node1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.node1.ons application 0/3 0/0 ONLINE ONLINE node1
ora.node1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE node1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE node2
ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE node2
ora.node2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.node2.ons application 0/3 0/0 ONLINE ONLINE node2
ora.node2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE node2
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE node3
ora....E3.lsnr application 0/5 0/0 ONLINE ONLINE node3
ora.node3.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.node3.ons application 0/3 0/0 ONLINE ONLINE node3
ora.node3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE node3
ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE node1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE node1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE node1
ora.zhongwc.db ora....se.type 0/2 0/1 ONLINE ONLINE node1
[grid@node3 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.DATADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.FRADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ONLINE ONLINE node3 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
OFFLINE OFFLINE node3
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node1
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.node3.vip
1 ONLINE ONLINE node3
ora.oc4j
1 ONLINE ONLINE node1
ora.scan1.vip
1 ONLINE ONLINE node1
ora.zhongwc.db
1 ONLINE ONLINE node1 Open
2 ONLINE ONLINE node2 Open
Extend Oracle Database Software
From an existing node – as the database software owner – run the following command to extend the Oracle database software to the new node "node3"
[oracle@node1 ~]$ cd $ORACLE_HOME/oui/bin
[oracle@node1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={node3}"
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "node1"
Checking user equivalence...
User equivalence check passed for user "oracle"
WARNING:
Node "node3" already appears to be part of cluster
Pre-check for node addition was successful.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3919 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.
Performing tests to see whether nodes node2,node3 are available
............................................................... 100% Done.
...
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/oracle/product/11.2.0/dbhome_1
New Nodes
Space Requirements
New Nodes
node3
/: Required 4.13GB : Available 63.03GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.3.0
Sun JDK 1.5.0.30.03
Installer SDK Component 11.2.0.3.0
Oracle One-Off Patch Installer 11.2.0.1.7
Oracle Universal Installer 11.2.0.3.0
Oracle USM Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.3.0
Oracle RAC Deconfiguration 11.2.0.3.0
Oracle Database Deconfiguration 11.2.0.3.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.5.0.1
Oracle ODBC Driverfor Instant Client 11.2.0.3.0
LDAP Required Support Files 11.2.0.3.0
SSL Required Support Files for InstantClient 11.2.0.3.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.3.0
Oracle Database Vault J2EE Application 11.2.0.3.0
Oracle Label Security 11.2.0.3.0
Oracle Data Mining RDBMS Files 11.2.0.3.0
Oracle OLAP RDBMS Files 11.2.0.3.0
Oracle OLAP API 11.2.0.3.0
Platform Required Support Files 11.2.0.3.0
Oracle Database Vault option 11.2.0.3.0
Oracle RAC Required Support Files-HAS 11.2.0.3.0
SQL*Plus Required Support Files 11.2.0.3.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.3.0
Oracle SQL Developer 11.2.0.3.0
Oracle Application Express 11.2.0.3.0
XDK Required Support Files 11.2.0.3.0
RDBMS Required Support Files for Instant Client 11.2.0.3.0
SQLJ Runtime 11.2.0.3.0
Database Workspace Manager 11.2.0.3.0
RDBMS Required Support Files Runtime 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
Enterprise Manager Repository Core Files 10.2.0.4.4
Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
Enterprise Manager Grid Control Core Files 10.2.0.4.4
Enterprise Manager Common Core Files 10.2.0.4.4
Enterprise Manager Agent Core Files 10.2.0.4.4
RDBMS Required Support Files 11.2.0.3.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.3.0
Oracle Text Required Support Files 11.2.0.3.0
Parser Generator Required Support Files 11.2.0.3.0
Oracle Database 11g Multimedia Files 11.2.0.3.0
Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
Oracle Multimedia Annotator 11.2.0.3.0
Oracle JDBC/OCI Instant Client 11.2.0.3.0
Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
Precompiler Required Support Files 11.2.0.3.0
Oracle Core Required Support Files 11.2.0.3.0
Sample Schema Data 11.2.0.3.0
Oracle Starter Database 11.2.0.3.0
Oracle Message Gateway Common Files 11.2.0.3.0
Oracle XML Query 11.2.0.3.0
XML Parser for Oracle JVM 11.2.0.3.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.3.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.3.0
Oracle Quality of Service Management (Client) 11.2.0.3.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.3.0
Oracle Containers for Java 11.2.0.3.0
Perl Interpreter 5.10.0.0.2
Oracle Net Required Support Files 11.2.0.3.0
Secure Socket Layer 11.2.0.3.0
Oracle Universal Connection Pool 11.2.0.3.0
Oracle JDBC/THIN Interfaces 11.2.0.3.0
Oracle Multimedia Client Option 11.2.0.3.0
Oracle Java Client 11.2.0.3.0
Character Set Migration Utility 11.2.0.3.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.3.0
OLAP SQL Scripts 11.2.0.3.0
Database SQL Scripts 11.2.0.3.0
Oracle Locale Builder 11.2.0.3.0
Oracle Globalization Support 11.2.0.3.0
SQL*Plus Files for Instant Client 11.2.0.3.0
Required Support Files 11.2.0.3.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.3.0
Oracle Notification Service 11.2.0.3.0
XML Parser for Java 11.2.0.3.0
Oracle Security Developer Tools 11.2.0.3.0
Oracle Wallet Manager 11.2.0.3.0
Cluster Verification Utility Common Files 11.2.0.3.0
Oracle Clusterware RDBMS Files 11.2.0.3.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.3.0
HAS Common Files 11.2.0.3.0
Precompiler Common Files 11.2.0.3.0
Installation Common Files 11.2.0.3.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.3.0
Buildtools Common Files 11.2.0.3.0
Assistant Common Files 11.2.0.3.0
Oracle Recovery Manager 11.2.0.3.0
PL/SQL 11.2.0.3.0
Generic Connectivity Common Files 11.2.0.3.0
Oracle Database Gateway for ODBC 11.2.0.3.0
Oracle Programmer 11.2.0.3.0
Oracle Database Utilities 11.2.0.3.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.3.0
Oracle Netca Client 11.2.0.3.0
Oracle Multimedia Locator 11.2.0.3.0
Oracle Call Interface (OCI) 11.2.0.3.0
Oracle Multimedia 11.2.0.3.0
Oracle Net 11.2.0.3.0
Oracle XML Development Kit 11.2.0.3.0
Database Configuration and Upgrade Assistants 11.2.0.3.0
Oracle JVM 11.2.0.3.0
Oracle Advanced Security 11.2.0.3.0
Oracle Internet Directory Client 11.2.0.3.0
Oracle Enterprise Manager Console DB 11.2.0.3.0
HAS Files for DB 11.2.0.3.0
Oracle Net Listener 11.2.0.3.0
Oracle Text 11.2.0.3.0
Oracle Net Services 11.2.0.3.0
Oracle Database 11g 11.2.0.3.0
Oracle OLAP 11.2.0.3.0
Oracle Spatial 11.2.0.3.0
Oracle Partitioning 11.2.0.3.0
Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Saturday, January 5, 2013 10:02:31 AM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Saturday, January 5, 2013 10:02:41 AM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Saturday, January 5, 2013 10:45:12 AM CST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh #On nodes node3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@node1 bin]$
Run the "root.sh" commands on the new node as directed
[root@node3 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Add Instance to Clustered Database
A database instance will be established on the new node. Specifically, an instance named "zhongwc3" will be added to "zhongwc" – a pre-existing clustered database.
Satisfy Node Instance Dependencies
Satisfy all node instance dependencies, such as passwordfile, init.ora parameters, etc.
From the new node "node3", run the following commands to create the passwordfile, "init.ora" file, and "oratab" entry for the new instance
[oracle@node3 ~]$ cd $ORACLE_HOME/dbs
[oracle@node3 dbs]$ mv initzhongwc1.ora initzhongwc3.ora
[oracle@node3 dbs]$ mv orapwzhongwc1 orapwzhongwc3
[oracle@node3 dbs]$ echo "zhongwc3:$ORACLE_HOME:N" >> /etc/oratab
From a node with an existing instance of "zhongwc" issue the following commands to create the needed public log thread, undo tablespace, and "init.ora" entries for the new instance
[oracle@node1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jan 5 11:06:02 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> alter database add logfile thread 3 group 5 ('+DATADG','+FRADG') size 50M, group 6 ('+DATADG','+FRADG') size 50M;
Database altered.
SQL> alter database enable public thread 3;
Database altered.
SQL> create undo tablespace undotbs3 datafile '+DATADG' size 200M autoextend on;
Tablespace created.
SQL> alter system set undo_tablespace='undotbs3' scope=spfile sid='zhongwc3';
System altered.
SQL> alter system set instance_number=3 scope=spfile sid='zhongwc3';
System altered.
SQL> alter system set cluster_database_instances=3 scope=spfile sid='*';
System altered.
Update Oracle Cluster Registry (OCR)
The OCR will be updated to account for a new instance – "zhongwc3" – being added to the "zhongwc" cluster database .
Add "zhongwc3" instance to the "zhongwc" database and verify
[oracle@node3 ~]$ srvctl add instance -d zhongwc -i zhongwc3 -n node3
[oracle@node3 ~]$ srvctl status database -d zhongwc -v
Instance zhongwc1 is running on node node1. Instance status: Open.
Instance zhongwc2 is running on node node2. Instance status: Open.
Instance zhongwc3 is not running on node node3
[oracle@node3 ~]$
[oracle@node3 ~]$ srvctl config database -d zhongwc
Database unique name: zhongwc
Database name: zhongwc
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATADG/zhongwc/spfilezhongwc.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: zhongwc
Database instances: zhongwc1,zhongwc2,zhongwc3
Disk Groups: DATADG,FRADG
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@node3 ~]$
Start the Instance
Now that all the prerequisites have been satisfied and OCR updated, the "zhongwc3" instance will be started.
Start the newly created instance – "zhongwc3" – and verify
[oracle@node3 ~]$ srvctl start instance -d zhongwc -i zhongwc3
[oracle@node3 ~]$
[oracle@node3 ~]$ crvctl status database -d zhongwc -v
-bash: crvctl: command not found
[oracle@node3 ~]$ srvctl status database -d zhongwc -v
Instance zhongwc1 is running on node node1. Instance status: Open.
Instance zhongwc2 is running on node node2. Instance status: Open.
Instance zhongwc3 is running on node node3. Instance status: Open.
[oracle@node3 ~]$
[oracle@node2 admin]$ pwd
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
[oracle@node2 admin]$ scp tnsnames.ora node3:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
tnsnames.ora 100% 342 0.3KB/s 00:00
[oracle@node2 admin]$
[oracle@node3 ~]$ cat /etc/issue
Oracle Linux Server release 6.3
Kernel \r on an \m
[oracle@node3 ~]$ uname -p
x86_64
[oracle@node3 ~]$
[oracle@node3 ~]$
[oracle@node3 ~]$ hostname
node3
[oracle@node3 ~]$
[oracle@node3 ~]$ sqlplus system@zhongwc
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jan 5 11:29:32 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss';
Session altered.
SQL> col host_name format a11
SQL> set line 300
SQL> select INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS,ACTIVE_STATE,INSTANCE_ROLE,DATABASE_STATUS from gv$INSTANCE;
INSTANCE_NAME HOST_NAME VERSION STARTUP_TIME STATUS ACTIVE_ST INSTANCE_ROLE DATABASE_STATUS
---------------- ----------- ----------------- ------------------- ------------ --------- ------------------ -----------------
zhongwc3 node3 11.2.0.3.0 2013-01-05 11:23:16 OPEN NORMAL PRIMARY_INSTANCE ACTIVE
zhongwc1 node1 11.2.0.3.0 2013-01-05 09:53:24 OPEN NORMAL PRIMARY_INSTANCE ACTIVE
zhongwc2 node2 11.2.0.3.0 2013-01-04 17:34:40 OPEN NORMAL PRIMARY_INSTANCE ACTIVE
SQL> conn hr@zhongwc
Enter password:
Connected.
SQL> select tname from tab;
TNAME
------------------------------
COUNTRIES
DEPARTMENTS
EMPLOYEES
EMP_DETAILS_VIEW
JOBS
JOB_HISTORY
LOCATIONS
REGIONS
8 rows selected.
[grid@node3 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]
[grid@node3 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 88a0a181a53c4fe0bfc583095b1f3c4b (/dev/asm-diskb) [CRS]
Located 1 voting disk(s).
[grid@node3 ~]$
[grid@node3 ~]$
[grid@node3 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.DATADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.FRADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ONLINE ONLINE node3 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
OFFLINE OFFLINE node3
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node2
ora.cvu
1 ONLINE ONLINE node3
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.node3.vip
1 ONLINE ONLINE node3
ora.oc4j
1 ONLINE ONLINE node3
ora.scan1.vip
1 ONLINE ONLINE node2
ora.zhongwc.db
1 ONLINE ONLINE node1 Open
2 ONLINE ONLINE node2 Open
3 ONLINE ONLINE node3 Open
[grid@node3 ~]$ ps -ef|egrep "oracle|grid"
root 1872 1785 0 Jan04 pts/0 00:00:00 su - grid
grid 1873 1872 0 Jan04 pts/0 00:00:00 -bash
root 1896 1807 0 Jan04 pts/1 00:00:00 su - oracle
oracle 1897 1896 0 Jan04 pts/1 00:00:00 -bash
root 11348 1 0 09:16 ? 00:01:09 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
grid 11961 1 0 09:18 ? 00:00:28 /u01/app/11.2.0/grid/bin/oraagent.bin
grid 11973 1 0 09:18 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid 11986 1 0 09:18 ? 00:00:06 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 12003 1 0 09:18 ? 00:00:10 /u01/app/11.2.0/grid/bin/cssdmonitor
grid 12005 1 0 09:18 ? 00:00:23 /u01/app/11.2.0/grid/bin/gipcd.bin
root 12026 1 0 09:18 ? 00:00:10 /u01/app/11.2.0/grid/bin/cssdagent
grid 12051 1 1 09:18 ? 00:01:36 /u01/app/11.2.0/grid/bin/ocssd.bin
root 12104 1 0 09:19 ? 00:00:42 /u01/app/11.2.0/grid/bin/orarootagent.bin
root 12116 1 0 09:19 ? 00:00:32 /u01/app/11.2.0/grid/bin/octssd.bin
root 12138 1 1 09:19 ? 00:02:02 /u01/app/11.2.0/grid/bin/osysmond.bin
grid 12232 1 0 09:20 ? 00:00:01 asm_pmon_+ASM3
grid 12234 1 0 09:20 ? 00:00:03 asm_psp0_+ASM3
grid 12236 1 7 09:20 ? 00:10:39 asm_vktm_+ASM3
grid 12240 1 0 09:20 ? 00:00:00 asm_gen0_+ASM3
grid 12242 1 0 09:20 ? 00:00:10 asm_diag_+ASM3
grid 12244 1 0 09:20 ? 00:00:02 asm_ping_+ASM3
grid 12246 1 0 09:20 ? 00:00:37 asm_dia0_+ASM3
grid 12248 1 0 09:20 ? 00:00:27 asm_lmon_+ASM3
grid 12250 1 0 09:20 ? 00:00:25 asm_lmd0_+ASM3
grid 12252 1 0 09:20 ? 00:00:46 asm_lms0_+ASM3
grid 12256 1 0 09:20 ? 00:00:00 asm_lmhb_+ASM3
grid 12258 1 0 09:20 ? 00:00:00 asm_mman_+ASM3
grid 12260 1 0 09:20 ? 00:00:00 asm_dbw0_+ASM3
grid 12262 1 0 09:20 ? 00:00:00 asm_lgwr_+ASM3
grid 12264 1 0 09:20 ? 00:00:00 asm_ckpt_+ASM3
grid 12266 1 0 09:20 ? 00:00:00 asm_smon_+ASM3
grid 12268 1 0 09:20 ? 00:00:04 asm_rbal_+ASM3
grid 12270 1 0 09:20 ? 00:00:00 asm_gmon_+ASM3
grid 12272 1 0 09:20 ? 00:00:00 asm_mmon_+ASM3
grid 12274 1 0 09:20 ? 00:00:01 asm_mmnl_+ASM3
grid 12282 1 0 09:20 ? 00:00:01 asm_lck0_+ASM3
grid 12284 1 0 09:20 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
root 12292 1 1 09:20 ? 00:01:27 /u01/app/11.2.0/grid/bin/crsd.bin reboot
grid 12311 1 0 09:20 ? 00:00:36 /u01/app/11.2.0/grid/bin/evmd.bin
grid 12314 1 0 09:20 ? 00:00:00 oracle+ASM3_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12320 1 0 09:20 ? 00:00:00 asm_asmb_+ASM3
grid 12322 1 0 09:20 ? 00:00:01 oracle+ASM3_asmb_+asm3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 12413 12311 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
grid 12456 1 0 09:21 ? 00:00:20 /u01/app/11.2.0/grid/bin/oraagent.bin
root 12460 1 0 09:21 ? 00:00:36 /u01/app/11.2.0/grid/bin/orarootagent.bin
grid 12551 1 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 12552 12551 0 09:21 ? 00:00:02 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 13032 1 0 09:21 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 13418 1 0 09:38 ? 00:00:01 /u01/app/11.2.0/grid/bin/scriptagent.bin
grid 13467 1 0 09:41 ? 00:00:22 /u01/app/11.2.0/grid/jdk/jre//bin/java -server -Xcheck:jni -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -Dport.rmi=23792 -jar /u01/app/11.2.0/grid/oc4j/j2ee/home/oc4j.jar -config /u01/app/11.2.0/grid/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -out /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.out -err /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.err
grid 16394 1 0 10:26 ? 00:00:00 oracle+ASM3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 17977 1 0 11:21 ? 00:00:02 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle 18159 1 0 11:23 ? 00:00:00 ora_pmon_zhongwc3
oracle 18161 1 0 11:23 ? 00:00:00 ora_psp0_zhongwc3
oracle 18163 1 4 11:23 ? 00:00:29 ora_vktm_zhongwc3
oracle 18167 1 0 11:23 ? 00:00:00 ora_gen0_zhongwc3
oracle 18169 1 0 11:23 ? 00:00:00 ora_diag_zhongwc3
oracle 18171 1 0 11:23 ? 00:00:00 ora_dbrm_zhongwc3
oracle 18173 1 0 11:23 ? 00:00:00 ora_ping_zhongwc3
oracle 18175 1 0 11:23 ? 00:00:00 ora_acms_zhongwc3
oracle 18177 1 0 11:23 ? 00:00:04 ora_dia0_zhongwc3
oracle 18179 1 0 11:23 ? 00:00:01 ora_lmon_zhongwc3
oracle 18181 1 0 11:23 ? 00:00:02 ora_lmd0_zhongwc3
oracle 18183 1 1 11:23 ? 00:00:08 ora_lms0_zhongwc3
oracle 18187 1 0 11:23 ? 00:00:00 ora_rms0_zhongwc3
oracle 18189 1 0 11:23 ? 00:00:00 ora_lmhb_zhongwc3
oracle 18191 1 0 11:23 ? 00:00:00 ora_mman_zhongwc3
oracle 18193 1 0 11:23 ? 00:00:00 ora_dbw0_zhongwc3
oracle 18195 1 0 11:23 ? 00:00:00 ora_lgwr_zhongwc3
oracle 18197 1 0 11:23 ? 00:00:00 ora_ckpt_zhongwc3
oracle 18199 1 0 11:23 ? 00:00:00 ora_smon_zhongwc3
oracle 18201 1 0 11:23 ? 00:00:00 ora_reco_zhongwc3
oracle 18203 1 0 11:23 ? 00:00:00 ora_rbal_zhongwc3
oracle 18205 1 0 11:23 ? 00:00:00 ora_asmb_zhongwc3
oracle 18207 1 0 11:23 ? 00:00:01 ora_mmon_zhongwc3
grid 18209 1 0 11:23 ? 00:00:00 oracle+ASM3_asmb_zhongwc3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 18211 1 0 11:23 ? 00:00:00 ora_mmnl_zhongwc3
oracle 18213 1 0 11:23 ? 00:00:00 ora_d000_zhongwc3
oracle 18215 1 0 11:23 ? 00:00:00 ora_s000_zhongwc3
oracle 18217 1 0 11:23 ? 00:00:00 ora_mark_zhongwc3
oracle 18238 1 0 11:24 ? 00:00:01 ora_lck0_zhongwc3
oracle 18240 1 0 11:24 ? 00:00:00 ora_rsmn_zhongwc3
oracle 18281 1 0 11:24 ? 00:00:02 oraclezhongwc3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 18326 1 0 11:24 ? 00:00:00 ora_pz99_zhongwc3
oracle 18328 1 0 11:24 ? 00:00:00 ora_gcr0_zhongwc3
oracle 18338 1 0 11:24 ? 00:00:00 ora_arc0_zhongwc3
oracle 18344 1 0 11:24 ? 00:00:00 ora_arc1_zhongwc3
oracle 18348 1 0 11:24 ? 00:00:00 ora_arc2_zhongwc3
oracle 18351 1 0 11:24 ? 00:00:00 ora_arc3_zhongwc3
oracle 18366 1 0 11:25 ? 00:00:00 ora_gtx0_zhongwc3
oracle 18368 1 0 11:25 ? 00:00:00 ora_rcbg_zhongwc3
oracle 18370 1 0 11:25 ? 00:00:00 ora_qmnc_zhongwc3
oracle 18384 1 0 11:25 ? 00:00:00 oraclezhongwc3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 18394 1 0 11:25 ? 00:00:00 ora_q000_zhongwc3
oracle 18396 1 0 11:25 ? 00:00:00 ora_q001_zhongwc3
oracle 18415 1 0 11:25 ? 00:00:01 ora_cjq0_zhongwc3
oracle 18417 1 0 11:25 ? 00:00:01 ora_vkrm_zhongwc3
oracle 18605 1 0 11:30 ? 00:00:00 ora_smco_zhongwc3
oracle 18607 1 0 11:30 ? 00:00:00 ora_w000_zhongwc3
oracle 18647 1 0 11:32 ? 00:00:00 ora_o000_zhongwc3
grid 18650 1 0 11:32 ? 00:00:00 oracle+ASM3_o000_zhongwc3 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 18705 1873 0 11:33 pts/0 00:00:00 ps -ef
grid 18706 1873 0 11:33 pts/0 00:00:00 egrep oracle|grid
[grid@node3 ~]$