实验:心跳网络网卡更换或者修改错误重启后不可用恢复。
调整配置:心跳eth1:100.100.100.X修改为心跳eth2:10.10.10.X。日常中有可能是网卡损坏更换网卡。
eth1 Link encap:Ethernet HWaddr 08:00:27:F7:67:C6
inet addr:100.100.100.100 Bcast:100.100.100.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fef7:67c6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:128219 errors:0 dropped:0 overruns:0 frame:0
TX packets:164287 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:79906709 (76.2 MiB) TX bytes:125379610 (119.5 MiB)
eth1:1 Link encap:Ethernet HWaddr 08:00:27:F7:67:C6
inet addr:169.254.176.136 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 08:00:27:43:81:9A
inet addr:10.10.10.11 Bcast:10.10.10.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe43:819a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1125 errors:0 dropped:0 overruns:0 frame:0
TX packets:261 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178079 (173.9 KiB) TX bytes:48566 (47.4 KiB)
[root@rac1 crsd]# ifdown eth1
[root@rac2 ~]# ifdown eth1
关闭群集,重新启动群集:
[root@rac1 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlock
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown
ora.cluster_interconnect.haip
1 ONLINE OFFLINE STARTING
ora.crf
1 ONLINE OFFLINE
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE rac1
ora.cssdmonitor
1 ONLINE ONLINE rac1
ora.ctssd
1 ONLINE ONLINE rac1 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE OFFLINE
ora.evmd
1 ONLINE INTERMEDIATE rac1
ora.gipcd
1 ONLINE ONLINE rac1
ora.gpnpd
1 ONLINE ONLINE rac1
ora.mdnsd
1 ONLINE ONLINE rac1
检查各种日志,都表明心跳网络有问题。检查操作系统网络,及gpnp profile 文件对比。
检查是发现原先的心跳网卡已经受损,需要更换至eth2的网卡;
[root@rac1 ~]# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 08:00:27:43:81:9A
inet addr:10.10.10.11 Bcast:10.10.10.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe43:819a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1128 errors:0 dropped:0 overruns:0 frame:0
TX packets:315 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178389 (174.2 KiB) TX bytes:59045 (57.6 KiB)
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/11.2.0/grid/bin/gpnptool.bin get -o-
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="21" ClusterUId="afd4fb8a969e6ff0ff47999b3fa206d6" ClusterName="cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" Adapter="bond0" IP="192.168.100.0" Use="public"/><gpnp:Network id="net2" Adapter="eth1" IP="100.100.100.0" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+OCR/cluster/asmparameterfile/registry.253.963054397"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>fF43VW1hAMTBFOZ+tuw3qbQSQDo=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>pYENVeL1XgZ2/fYmDY7xxMw+qY4AOYu+RApbJtVfIUA2muPuDmKVLYDddQOrX0XfwRdS3fFdO77cJuv1ApFUEQIDVjQdxVnzQWK+QhJUOvfsl1oE1g+rMNuJq3S6VWoRXLt4pr4wzY7CBkKdnoCdqsRu6u3yECWFiECUr6guk2Y=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
Success
[grid@rac1 gpnp]$ gpnptool get -o=/home/grid/gpnp/p.xml
Resulting profile written to "/home/grid/gpnp/p.xml".
Success.
[root@rac1 rac1]# /u01/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
[grid@rac1 gpnp]$ gpnptool getpval -p=/home/grid/gpnp/p.xml -prf_sq -o-
21
[grid@rac1 gpnp]$ gpnptool getpval -p=/home/grid/gpnp/p.xml -net -o-
net1 net2
[grid@rac1 gpnp]$ gpnptool edit -p=/home/grid/gpnp/p.xml -o=/home/grid/gpnp/p.xml -ovr -prf_sq=22 -net2:net_ip=10.10.10.0 -net2:net_ada=eth2
Resulting profile written to "/home/grid/gpnp/p.xml".
Success.
[grid@rac1 gpnp]$ cat p.xml
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="22" ClusterUId="afd4fb8a969e6ff0ff47999b3fa206d6" ClusterName="cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" Adapter="bond0" IP="192.168.100.0" Use="public"/><gpnp:Network id="net2" Adapter="eth2" IP="10.10.10.0" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+OCR/cluster/asmparameterfile/registry.253.963054397"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>fF43VW1hAMTBFOZ+tuw3qbQSQDo=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>pYENVeL1XgZ2/fYmDY7xxMw+qY4AOYu+RApbJtVfIUA2muPuDmKVLYDddQOrX0XfwRdS3fFdO77cJuv1ApFUEQIDVjQdxVnzQWK+QhJUOvfsl1oE1g+rMNuJq3S6VWoRXLt4pr4wzY7CBkKdnoCdqsRu6u3yECWFiECUr6guk2Y=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
[grid@rac1 gpnp]$ gpnptool getpval -p=/home/grid/gpnp/p.xml -prf_sq -o-
22
[grid@rac1 gpnp]$ gpnptool sign -p=/home/grid/gpnp/p.xml -o=/home/grid/gpnp/p.xml -ovr -w=cw-fs:peer
Resulting profile written to "/home/grid/gpnp/p.xml".
Success.
[grid@rac1 gpnp]$ gpnptool put -p=/home/grid/gpnp/p.xml
Success.
[grid@rac1 gpnp]$ gpnptool find -c=cluster
Found 1 instances of service 'gpnp'.
mdns:service:gpnp._tcp.local.://rac1:34523/agent=gpnpd,cname=cluster,host=rac1,pid=23098/gpnpd h:rac1 c:cluster
[grid@rac1 gpnp]$ gpnptool rget -c=cluster
Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/11.2.0/grid/bin/gpnptool.bin rget -c=cluster -o-
Found 1 gpnp service instance(s) to rget profile from.
RGET from tcp://rac1:34523 (mdns:service:gpnp._tcp.local.://rac1:34523/agent=gpnpd,cname=cluster,host=rac1,pid=23098/gpnpd h:rac1 c:cluster):
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="22" ClusterUId="afd4fb8a969e6ff0ff47999b3fa206d6" ClusterName="cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" Adapter="bond0" IP="192.168.100.0" Use="public"/><gpnp:Network id="net2" Adapter="eth2" IP="10.10.10.0" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+OCR/cluster/asmparameterfile/registry.253.963054397"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>VKMru/kdgAc2kEYY5hhmJy0lRB0=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>fTepQy99aHHXqbA9aFxGLBSJq0iRZOSRzIjgJdfx6tD2I5fl0boZIPUqrIezRr3L4SBMja4jJbqaNxJzPbpsKHtecWg14FQhkmPbZsqU55pYQ5ldO0BIRmAcZ47NHMaUmtsyw+2Wid5li/g8B6+4FmyOfggd7T/bp+kDSYFDgog=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
Success.
重新启动群集:
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop has
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start has
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.OCR.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.cube.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac1
总结:接上文,如果网卡损坏,只需修改相应的配置就行,不需要重新安装群集软件等操作。 gpnptool 工具帮助 可执行gpnptool help查看