RAC删除节点失败重启大法解决

近期需要测试一下RAC增加、删除节点操作。

在LINUX5.7+ORACLE10.2.0.5 RAC上，先是为两节点RAC增加一个节点，然后再删除节点2.

在删除节点2时遇到无法删除的错误，如下信息，

[root@rac1 install]# ./rootdeletenode.sh rac2,2
CRS-0210: Could not find resource 'ora.rac2.ons'.
CRS-0210: Could not find resource 'ora.rac2.vip'.
CRS-0210: Could not find resource 'ora.rac2.gsd'.
CRS-0210: Could not find resource ora.rac2.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
CSS indicates failure 17 acknowledging the deletion.
Attempting to roll back the deletion.
Rollback of node deletion successful.
Node deletion failed but configuration is intact.
Failed in deleting nodes 'rac2,2'

此时节点2的CSS信息：
[root@rac2 crs]# ps -ef|grep css
root      3348     1  0 18:54 ?        00:00:24 /bin/sh /etc/init.d/init.cssd fatal
root      4356  3348  0 18:54 ?        00:00:00 /bin/sh /etc/init.d/init.cssd oprocd
root      4371  3348  0 18:54 ?        00:00:00 /bin/sh /etc/init.d/init.cssd oclsomon
root      4401  3348  0 18:54 ?        00:00:00 /bin/sh /etc/init.d/init.cssd daemon
root      4902  4371  0 18:54 ?        00:00:00 /sbin/runuser -l oracle -c /bin/sh -c 'cd /u01/app/oracle/product/crs/log/rac2/cssd/oclsomon; ulimit -c unlimited; /u01/app/oracle/product/crs/bin/oclsomon  || exit $?'
oracle    4903  4902  0 18:54 ?        00:00:00 /bin/sh -c cd /u01/app/oracle/product/crs/log/rac2/cssd/oclsomon; ulimit -c unlimited; /u01/app/oracle/product/crs/bin/oclsomon  || exit $?
oracle    4975  4401  0 18:54 ?        00:00:36 /u01/app/oracle/product/crs/bin/ocssd.bin
root     29835  5821  0 22:39 pts/3    00:00:00 grep css

解决办法：

重启节点2，此时将要删除的节点关闭或者重启，然后重新删除即可：

[root@rac1 install]# ./rootdeletenode.sh rac2,2
CRS-0210: Could not find resource 'ora.rac2.ons'.
CRS-0210: Could not find resource 'ora.rac2.vip'.
CRS-0210: Could not find resource 'ora.rac2.gsd'.
CRS-0210: Could not find resource ora.rac2.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.noderac2 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'rac2,2' deleted successfully