管理10G clusterware.

1.check the status of crs and start/stop crs:
# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
# crsctl stop crs
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
#crsctl start crs

2.check the status of resource:
# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....ST1.srv application    OFFLINE   OFFLINE               
ora....ST2.srv application    OFFLINE   OFFLINE               
ora....RTST.cs application    OFFLINE   OFFLINE               
ora....ST1.srv application    OFFLINE   OFFLINE               
ora....TST1.cs application    OFFLINE   OFFLINE               
ora....T1.inst application    OFFLINE   UNKNOWN   tst-...-eqx
ora....ST2.srv application    OFFLINE   OFFLINE               
ora....TST2.cs application    OFFLINE   OFFLINE               
ora....T2.inst application    OFFLINE   UNKNOWN   tst-...-eqx
ora.RTST.db    application    OFFLINE   OFFLINE               
ora....QX.lsnr application    OFFLINE   OFFLINE               
ora....eqx.gsd application    ONLINE    OFFLINE               
ora....eqx.ons application    ONLINE    OFFLINE               
ora....eqx.vip application    OFFLINE   OFFLINE               
ora....QX.lsnr application    OFFLINE   OFFLINE               
ora....eqx.gsd application    ONLINE    OFFLINE               
ora....eqx.ons application    ONLINE    OFFLINE               
ora....eqx.vip application    OFFLINE   OFFLINE      
3.startup the resource
# crs_start -all
Attempting to start `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.tst-rac02-ud-us-eqx.vip` on member `tst-rac02-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac01-ud-us-eqx` succeeded.
Attempting to start `ora.tst-rac01-ud-us-eqx.LISTENER_TST-RAC01-UD-US-EQX.lsnr` on member `tst-rac01-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.LISTENER_TST-RAC01-UD-US-EQX.lsnr` on member `tst-rac01-ud-us-eqx` succeeded.
Start of `ora.tst-rac02-ud-us-eqx.vip` on member `tst-rac02-ud-us-eqx` succeeded.
Attempting to start `ora.tst-rac02-ud-us-eqx.LISTENER_TST-RAC02-UD-US-EQX.lsnr` on member `tst-rac02-ud-us-eqx`
Start of `ora.tst-rac02-ud-us-eqx.LISTENER_TST-RAC02-UD-US-EQX.lsnr` on member `tst-rac02-ud-us-eqx` succeeded.
Attempting to start `ora.RTST.RTST.RTST1.srv` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST.RTST2.srv` on member `tst-rac02-ud-us-eqx`
Attempting to start `ora.RTST.RTST1.cs` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST1.RTST1.srv` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST2.cs` on member `tst-rac02-ud-us-eqx`
Attempting to start `ora.RTST.RTST.cs` on member `tst-rac02-ud-us-eqx`
Attempting to start `ora.RTST.RTST2.RTST2.srv` on member `tst-rac02-ud-us-eqx`
Start of `ora.RTST.RTST1.cs` on member `tst-rac01-ud-us-eqx` failed.
Attempting to start `ora.RTST.RTST1.cs` on member `tst-rac02-ud-us-eqx`
Start of `ora.RTST.RTST2.cs` on member `tst-rac02-ud-us-eqx` failed.
Attempting to start `ora.RTST.RTST2.cs` on member `tst-rac01-ud-us-eqx`
Start of `ora.RTST.RTST.cs` on member `tst-rac02-ud-us-eqx` failed.
Attempting to start `ora.RTST.RTST.cs` on member `tst-rac01-ud-us-eqx`
Start of `ora.RTST.RTST1.cs` on member `tst-rac02-ud-us-eqx` failed.
CRS-1006: No more members to consider

Attempting to start `ora.tst-rac01-ud-us-eqx.gsd` on member `tst-rac01-ud-us-eqx`
Start of `ora.RTST.RTST2.cs` on member `tst-rac01-ud-us-eqx` failed.
CRS-1006: No more members to consider

Attempting to start `ora.tst-rac01-ud-us-eqx.ons` on member `tst-rac01-ud-us-eqx`
Start of `ora.RTST.RTST.cs` on member `tst-rac01-ud-us-eqx` failed.
CRS-1006: No more members to consider

Attempting to start `ora.tst-rac02-ud-us-eqx.gsd` on member `tst-rac02-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.ons` on member `tst-rac01-ud-us-eqx` succeeded.
Attempting to start `ora.tst-rac02-ud-us-eqx.ons` on member `tst-rac02-ud-us-eqx`
Start of `ora.tst-rac02-ud-us-eqx.ons` on member `tst-rac02-ud-us-eqx` succeeded.
Attempting to start `ora.RTST.db` on member `tst-rac01-ud-us-eqx`
Start of `ora.tst-rac02-ud-us-eqx.gsd` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.db` on member `tst-rac01-ud-us-eqx` failed.
Attempting to start `ora.RTST.db` on member `tst-rac02-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.gsd` on member `tst-rac01-ud-us-eqx` failed.
tst-rac02-ud-us-eqx : CRS-1019: Resource ora.tst-rac01-ud-us-eqx.gsd (application) cannot run on tst-rac02-ud-us-eqx


Start of `ora.RTST.db` on member `tst-rac02-ud-us-eqx` failed.
CRS-1006: No more members to consider

Start of `ora.RTST.RTST2.RTST2.srv` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST.RTST2.srv` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST1.RTST1.srv` on member `tst-rac01-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST.RTST1.srv` on member `tst-rac01-ud-us-eqx` succeeded.
CRS-0215: Could not start resource 'ora.RTST.RTST.cs'.

CRS-0215: Could not start resource 'ora.RTST.RTST1.cs'.

CRS-0215: Could not start resource 'ora.RTST.RTST2.cs'.

CRS-0215: Could not start resource 'ora.RTST.db'.

CRS-0215: Could not start resource 'ora.tst-rac01-ud-us-eqx.gsd'.
4.check the status of resource again
# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....RTST.cs application    ONLINE    OFFLINE               
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST1.cs application    ONLINE    ONLINE    tst-...-eqx
ora....T1.inst application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST2.cs application    ONLINE    OFFLINE               
ora....T2.inst application    ONLINE    ONLINE    tst-...-eqx
ora.RTST.db    application    ONLINE    OFFLINE               
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    OFFLINE               
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
5.log on the other box and startup the left resource
# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....RTST.cs application    ONLINE    ONLINE    tst-...-eqx
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST1.cs application    ONLINE    ONLINE    tst-...-eqx
ora....T1.inst application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST2.cs application    ONLINE    ONLINE    tst-...-eqx
ora....T2.inst application    ONLINE    ONLINE    tst-...-eqx
ora.RTST.db    application    ONLINE    ONLINE    tst-...-eqx
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    OFFLINE               
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
# crs_start -all
Attempting to start `ora.tst-rac01-ud-us-eqx.gsd` on member `tst-rac01-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.gsd` on member `tst-rac01-ud-us-eqx` succeeded.
# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....RTST.cs application    ONLINE    ONLINE    tst-...-eqx
ora....ST1.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST1.cs application    ONLINE    ONLINE    tst-...-eqx
ora....T1.inst application    ONLINE    ONLINE    tst-...-eqx
ora....ST2.srv application    ONLINE    ONLINE    tst-...-eqx
ora....TST2.cs application    ONLINE    ONLINE    tst-...-eqx
ora....T2.inst application    ONLINE    ONLINE    tst-...-eqx
ora.RTST.db    application    ONLINE    ONLINE    tst-...-eqx
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
ora....QX.lsnr application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.gsd application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.ons application    ONLINE    ONLINE    tst-...-eqx
ora....eqx.vip application    ONLINE    ONLINE    tst-...-eqx
6.manage your RAC database with srvctl
$ srvctl status database -d RTST
Instance RTST1 is running on node tst-rac01-ud-us-eqx
Instance RTST2 is running on node tst-rac02-ud-us-eqx
$ srvctl status instance -i RTST1,RTST2
PRKO-2001 : Invalid command line syntax
$ srvctl status instance -i RTST1,RTST2 -d RTST
Instance RTST1 is running on node tst-rac01-ud-us-eqx
Instance RTST2 is running on node tst-rac02-ud-us-eqx
$ srvctl
Usage: srvctl <command> <object> [<options>]
    command: enable|disable|start|stop|relocate|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|instance|service|nodeapps|asm|listener
For detailed help on each command and object and its options use:
    srvctl <command> <object> -h
7. Adding mirrors for OCT and voting disk
$ ls -lrt /cluster_files
total 24528
drwx------   2 root     root       65536 Apr 17 19:40 lost+found
-rw-r-----   1 root     dba      2277376 Apr 20 18:09 ocr
-rw-r--r--   1 oracle   dba      10240000 Apr 20 18:09 voting_disk
$ mkdir /oradata1/cluster_files2
$ $ sudo su -
Password:
# ocrcheck
ocrcheck: not found
# pwd
/
# cd /opt/app/oracle/product/10*/crs*
# cd bin
# ocrcheck
ocrcheck: not found
# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262144
         Used space (kbytes)      :       1992
         Available space (kbytes) :     260152
         ID                       :  611528974
         Device/File Name         : /cluster_files/ocr
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

#
Run the following command to add an OCR mirror location using either destination_file or disk to designate the target location of the

additional OCR:
ocrconfig -replace ocrmirror destination_file or disk


# cp /cluster_files/ocr /oradata1/cluster_files2/ocr
# ./ocrconfig -replace ocrmirror /oradata1/cluster_files2/ocr
# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262144
         Used space (kbytes)      :       1992
         Available space (kbytes) :     260152
         ID                       :  611528974
         Device/File Name         : /cluster_files/ocr
                                    Device/File integrity check succeeded
         Device/File Name         : /oradata1/cluster_files2/ocr
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded
switch VIP
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
e1000g0: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER> mtu 1500 index 2
        inet 10.12.32.192 netmask ffffff00 broadcast 10.12.32.255
        groupname sc_ipmp0
        ether 0:14:4f:7e:8c:2e
e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
        inet 10.12.32.203 netmask ffffff00 broadcast 10.12.32.255
e1000g0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
        inet 10.12.32.204 netmask ffffff00 broadcast 10.12.32.255
e1000g2: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4
        inet 172.16.0.130 netmask ffffff80 broadcast 172.16.0.255
        ether 0:14:4f:7e:8c:30
e1000g3: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3
        inet 172.16.1.2 netmask ffffff80 broadcast 172.16.1.127
        ether 0:14:4f:7e:8c:31
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500 index 5
        inet 172.16.4.2 netmask fffffe00 broadcast 172.16.5.255
        ether 0:0:0:0:0:2
#  crs_relocate ora.tst-rac01-ud-us-eqx.vip
Attempting to stop `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac02-ud-us-eqx`
Stop of `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac02-ud-us-eqx` succeeded.
Attempting to start `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac01-ud-us-eqx`
Start of `ora.tst-rac01-ud-us-eqx.vip` on member `tst-rac01-ud-us-eqx` succeeded.
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
e1000g0: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER> mtu 1500 index 2
        inet 10.12.32.192 netmask ffffff00 broadcast 10.12.32.255
        groupname sc_ipmp0
        ether 0:14:4f:7e:8c:2e
e1000g0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
        inet 10.12.32.203 netmask ffffff00 broadcast 10.12.32.255
e1000g2: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4
        inet 172.16.0.130 netmask ffffff80 broadcast 172.16.0.255
        ether 0:14:4f:7e:8c:30
e1000g3: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3
        inet 172.16.1.2 netmask ffffff80 broadcast 172.16.1.127
        ether 0:14:4f:7e:8c:31
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500 index 5
        inet 172.16.4.2 netmask fffffe00 broadcast 172.16.5.255
        ether 0:0:0:0:0:2
#