MOS 文档 ID 1212703.

为没有MOS的网友提供方便

Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement (文档 ID 1212703.1)

转到底部

 

In this Document

 

Symptoms

Changes

Cause

Solution

ScalabilityRAC Community

References

 

 

APPLIES TO:

OracleDatabase - Enterprise Edition - Version 11.2.0.2 to 12.1.0.1 [Release 11.2 to12.1]

Informationin this document applies to any platform.

Thisissue impacts environments that do not have multicast enabled for the privatenetwork in the following situations:

 

Newinstallations of Oracle Grid Infrastructure 11.2.0.2 where multicast is notenabled on 230.0.1.0

Upgradesto Oracle Grid Infrastructure 11.2.0.2 from a pre-11.2.0.2 release wheremulticast is not enabled on 230.0.1.0 or 224.0.0.251

Installationof GI PSU 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 where multicast is not enabled on230.0.1.0 or 224.0.0.251

Installationor upgrade to 12.1.0.1.0 where multicast is not enabled on 230.0.1.0 or224.0.0.251

 

 

SYMPTOMS

Ifmulticast based communication is not enabled as required either on the nodes ofthe cluster or on the network switches used for the private interconnect, theroot.sh, which is called as part of a fresh installation of Oracle GridInfrastructure 11.2.0.2, or the rootupgrade.sh (called as part of an upgrade toOracle Grid Infrastructure 11.2.0.2) will only succeed on the first node of thecluster, but will fail on subsequent nodes with the symptoms shown below:

CRS-4402: The CSS daemon was started inexclusive mode but found an active CSS daemon on node node1, number 1, and isterminating

An active cluster was found during exclusivestartup, restarting to join the cluster

Failed to start Oracle Clusterware stack

Failed to start Cluster Synchorinisation Servicein clustered mode at /u01/app/crs/11.2.0.2/crs/install/crsconfig_lib.pm line1016.

/u01/app/crs/11.2.0.2/perl/bin/perl-I/u01/app/crs/11.2.0.2/perl/lib -I/u01/app/crs/11.2.0.2/crs/install/u01/app/crs/11.2.0.2/crs/install/rootcrs.pl execution failed

 

Note:  The symptoms will be the same whether anupgrade or a fresh installation of Oracle Grid Infrastructure 11.2.0.2 isperformed; so will be the required diagnostics. This issue is also documentedin the Oracle Database Readme 11g Release 2 Section 2.39 - "Open Bugs" under BUG: 9974223. 

 

Note: This issue also impacts the following11.2.0.3 PSUs: 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 as well as 12.1.0.1installations if multicast is not enabled on the 230.0.1.0 or 224.0.0.251multicast addresses (one of the 2 must be enabled/functional). With 11.2.0.3 GIwas enhanced to utilize broadcast or multicast to bootstrap. However the11.2.0.3.5 GI PSU introduced a new issue that effectively disables thebroadcast functionality (Bug 16547309). 

 

 

Symptomverification 

To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.logfile (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must bereviewed. In case, joining the cluster fails because of such an issue, thefollowing can be observed:

 

1. When CSS starts in clustered mode to join an existing cluster, wewill see an entry in the CSSD log indicating that CSS will attempt to establishcommunication with a peer in the cluster. For this analysis, we see in the CSSDlog for node2 that communication isattempted with node1, which looks similar to:  

 

2010-09-16 23:13:14.862: [GIPCHGEN][1107937600]gipchaNodeCreate: adding new node 0x2aaab408d4a0 { host 'node1', haName'CSS_ttoprf10cluster', srcLuid 54d7bb0e-ef4a0c7e, dstLuid 00000000-00000000numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [0 : 0], createTime9563084, flags 0x0 }

2.  Shortly after the above log entrywe will see an attempt to establish communication to node1 from node2 viamulticast address 230.0.1.0, port 42424 on the private interconnect (here:192.168.1.2):

2010-09-16 23:13:14.862: [GIPCHTHR][1106360640]gipchaWorkerUpdateInterface: created remote interface for node 'node1', haName'CSS_mycluster', inf 'mcast://230.0.1.0:42424/192.168.1.2'

 

3.  If the communication can beestablished successfully, we will see a log entry on node2 containing"gipchaLowerProcessAcks:ESTABLISH finished" for the peer node (node1). Ifthe communication cannot be established, we will not see this log entry.Instead, we will see an entry indicating that the network communicationcannot be established. This entry will look similar to the one shown below:

2010-09-16 23:13:15.839: [CSSD][1087465792]clssnmvDHBValidateNCopy: node 1, node1, has a disk HB, but nonetwork HB, DHB has rcfg 180134562, wrtcnt, 8627, LATS 9564064, lastSeqNo 8624,uniqueness 1284701023, timestamp 1284703995/10564774

 

The above log entry indicates that CSSDis unable to establish network communication on the interface used for theprivate interconnect. In this particular case, the issue was thatmulticast communication on the 230.0.1.0 IP was blocked on the network used asthe private interconnect.

 

 

CHANGES

• New installations of Oracle Grid Infrastructure 11.2.0.2
• Upgrades of a previous release to Oracle Grid Infrastructure 11.2.0.2
• Installation of the 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
• New installations of Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
• Upgrades of a previous release to Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses

Note: 11.2.0.4 Grid Infrastructure is not impacted by this issue.

 

CAUSE

Assuming that Cluster Verify (cluvfy) has succeeded regarding thenetwork checks on all nodes of the cluster or the symptoms described above areobserved as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2 (whichmeans that the current release does not encounter such communication issues),this issue is probably caused by multicast not being enabled on the networkused as the private interconnect.

Background information for11.2.0.2

OracleGrid Infrastructure 11.2.0.2 introduces a new feature called "RedundantInterconnect Usage", which provides an Oracle internal mechanism to makeuse of physically redundant network interfaces for the Oracle (private)interconnect. As part of this new feature, multicast based communication on theprivate interconnect is utilized to establish communication with peers in thecluster on each startup of the stack on a node. Once the connection with thepeers in the cluster has been established, the communication is switched backto unicast. Per default, the 230.0.1.0 address (port 42424) on the privateinterconnect network is used for multicasting. Another IP can be enabled usingthe patch mentioned below, if it is determined that using the 230.0.1.0 IPcauses the multicast communication to fail. Multicasting on either of these IPsand the respective port must, however, be enabled and functioning across thenetwork and on each node meant to be part of the cluster. If multicasting isnot enabled as required, nodes will fail to join the cluster with the symptomsdiscussed.

Background information for11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs and 12.1.0.1

With 11.2.0.3 GI was enhanced to utilizebroadcast or multicast (on 230.0.1.0 or 224.0.0.251 addresses) to bootstrap.However the 11.2.0.3.5 GI PSU introcuces a new issue with effectivly disablesthe broadcast functionality (Bug 16547309).  Do note that most networks do supportmulticast on the 224.0.0.251 multicast address without any specialconfiguration, therefore the odds of this being an issue for 11.2.0.3.5 -11.2.0.3.7 and 12.1.0.1 are greatly reduced.

Note:  The Oracle CSS daemon may fail toestablish network communication with peer nodes for other reasons thanmulticast not working as required on the private interconnect, which isdiscussed in this note. Therefore, refer to Note: 1054902.1 forgeneral network communication troubleshooting, if you determine thatmulticasting is not the root cause for such issues on your system.

 

Configuring various networkswitches for multicast

While the most common reason for multicast based communicationto fail with Oracle Grid Infrastructure 11.2.0.2 is probably the 230.0.1.0multicast address being used by default, multicast being disabled ormisconfigured on the network switches is an equally common cause. The mcasttest.pl-toolprovided as part of this note will give you an indication whether or notmulticasting has been disabled on the network switches, if it fails to usemulticast on either the 230.0.1.0 or 224.0.0.251 multicast address for example. In general, work with yournetwork administrator, if you suspect multicasting has been disabled onthe network switches, affecting the private interconnect communicationaccordingly.

Cisco Nexus Switches -general 

The Cisco Nexus switches can be complicated to configure withmulticasting. If multicasting cannot be properlyconfigured, then these switches have the ability to turn off IGMP Snooping on aper-VLAN basis. This has the effect of having the switch handle multicasts likeit would a broadcast. Since the network used for the interconnect is supposedto be private (where only the members of the cluster have access to thisnetwork), broadcasting every multicast packet to every NIC on the VLAN shouldpose no problem. Thecommand to turn off IGMP Snooping on a VLAN is:

no ip igmp snooping 

(Consult the Cisco Nexus manual for the exactsyntax of this command.)

Cisco Catalyst 

Turning off IGMP Snooping on a Cisco Catalyst switch is a morecomplex process, which is not covered in this note. For the Catalyst family of switches, IGMP Snooping can only beturned off globally and not on a per-VLAN basis. Consult the Cisco Catalystmanuals for how to properly configure multicasting for the switches over-allusage. Note that turning off IGMP Snooping on the Catalyst may cause othersystems or programs to fail.

Cisco Nexus 7000 Switch 

A feature of the Cisco Nexus 7000 switch is a filter that effectslarge UDP messages. This command is designed to filter IP packets with the"do not fragment" bit set along with a non-zero "Fragmentoffset". Oracle RAC will send UDP messages that exceed the carryingcapacity of a standard UDP packet. The IP layer sees this and sends theleft-over data in IP Fragments. These IP Fragments will then have the "donot fragment" bit set along with a non-zero "Fragment offset".The switch will not deliver these packets as they are filtered out by thisrule. 

 

Tovalidate this issue, perform a tcpdump (with the -s 0 flag to capture theentire packet) on each node during the Oracle Clusterware installation at thetime the root.sh script is run. The root.sh command will start the Oraclecluster services on the first node. When it is run on the second node, largeUDP packets can occur, which will generate the IP Fragments that will bedropped. When one reads the two tcpdump files, on will see an IP Fragmentpacket leave the second node, yet it will not arrive on the first node. Thefirst node will send out ICMP "Fragment reassembly time exceeded"packets as a result. To turn this rule off issue the following command:

no hardware ipverify fragment 

(Consult the Cisco Nexus manual for the exactsyntax of this command.)

These settings apply only to theCisco Nexus 7000 switch. This command may be available on other models of theNexus switch, but not all of the Nexus models support it. Check the Ciscomanual for the model of concern.

 

 

SOLUTION

For 11.2.0.2 Installations

It has been found that using a 230.0.1.0, port 42424, networkaddress for multicasting can be problematic with somenetwork configurations. Therefore, Oracle has released Patch: 9974223 ontop of Oracle Grid Infrastructure 11.2.0.2. This patchenables multicasting on the 224.0.0.251 multicast address (port 42424) inaddition to the 230.0.1.0 (port 42424) address used by default. Multicast mustgenerally be enabled on one of these two addresses to allow Oracle GridInfrastructure to successfully start on all nodes configured to join aparticular cluster.

For 11.2.0.3.5,11.2.0.3.6 and 11.2.0.3.7 GI PSU  and 12.1.0.1 Installations

11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1installations will ONLY be impacted if those installations were relying on thebroadcast functionality that was initially introduced in 11.2.0.3 base.  For these installations the mostsimplistic solution is to enable multicast on either the 230.0.1.0 or224.0.0.251 address (only 1 of the 2 addresses needs to be functional). For those installations that are unable to enable multicast, Patch 16547309 isavailable for some platforms to re-enable broadcast functionality.  If yourequire this patch (due to not being able to enable multicast) and it is notavailable for your version/platofrm please raise an SR with support to requestthe fix.  The fix for Bug 16547309 willbe included in the 11.2.0.3.8 and 12.1.0.1.2 GI PSUs.

 

Problem validation

Before applying Patch: 9974223, Patch 16547309 orany subsequent Oracle Grid Infrastructure (GI) Bundle Patch including whichinclude the mentioned fixes, understand that the issue described in this notedoes not apply under the following circumstances:

Note: The issue described in this note does not apply

• on Windows
  • The Redundant Interconnect Usage feature does not exist on Windows.
• on Solaris when using Solaris Cluster
  • When using Sun Cluster, Oracle Clusterware 11.2.0.2 will detect the presence of clprivnet0 and not enable any HAIPs. The Redundant Interconnect Usage feature is thereby disabled. clprivnet0 will provide the required availability for redundant network interfaces used for the interconnect.
• on HP-UX or AIX when using Infiniband network cards
  • If on HP-UX or AIX using Infiniband network interfaces the installation of Oracle Grid Infrastructure will fail due to OS and network hardware related issues, unrelated to multicast.
  • These are known BUGs and Oracle is working with the respective OS vendors for a fix.
  • Contact Oracle Support if you plan on upgrading to Oracle Grid Infrastructure 11.2.0.2 on either of those platforms using Infiniband network cards.
• using Primecluster
  • The Redundant Interconnect Usage feature is currently disabled when using Primecluster.

 If you have already run the root.sh (rootupgrade.sh) on anynode of the cluster:

Note: The README for Patch: 9974223 and Patch 16547309 containinstructions for those installations in which root.sh (or rootupgrade.sh) hasfailed due to this issue.  A complete reinstall of GridInfrastructure should  not berequired.

Multicast can be vaildated using the mcasttest.pl toolattached to this note to validate whether or not multicast on the230.0.1.0 or 224.0.0.251 address can be used on the private interconnectinterface(s).  CLUFFY in 11.2.0.3 and above does contain similar checksbut will report pass if the broadcast check passes, with mentioned issue whichbreaks broadcast in 11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 this pass is afalse positive.  For this reason it is recommended to use mcasttest.pl programto validate multicast functionality before patching/upgrading/installing thementioned releases.

Note: Multicast based communication only needs to be successful on eitherthe 230.0.1.0 address or the 224.0.0.251 address. A successful multicast communication on both addresses isnot required. 

 

How to use themcasttest-tool

• Download the mcasttest.pl

 perl program from this note and extract thecontents of the package to node 1 of your cluster:

# gunzip mcasttest.tgz 

# tar xvf mcasttest.tar

• The mcasttest.pl program can be found in the mcasttest directory extracted in the previous step.
• The mcasttest.pl program requires two arguments:
  1. The node list (specified with -n)
  2. The list of interfaces to be used for the private interconnect (specified with -i).
• The mcasttest is only to be executed on a single node in the cluster. It will use password-less SSH based access (also required for the Grid Infrastructure software installation) to distribute 
  and execute the program on the remote nodes. The command string to execute mcasttest is:

# perl mcasttest.pl -n<node1>,<node2>,<node_n...> -i<interface1>,<interface2><interface_n...>

The example below tests multicast for atwo node cluster (ratlnx01 and ratlnx02), in 

which two network interfaces (eth1 andeth2) are to be used for the private interconnect:

# perl mcasttest.pl -n ratlnx01,ratlnx02 -ieth1,eth2

########### Setup for node ratlnx01 ##########

Checking node access 'ratlnx01'

Checking node login 'ratlnx01'

Checking/Creating Directory /tmp/mcasttest forbinary on node 'ratlnx01'

Distributing mcast2 binary to node 'ratlnx01'

########### Setup for node ratlnx02 ##########

Checking node access 'ratlnx02'

Checking node login 'ratlnx02'

Checking/Creating Directory /tmp/mcasttest forbinary on node 'ratlnx02'

Distributing mcast2 binary to node 'ratlnx02'

########### testing Multicast on all nodes##########

 

Test for Multicast address 230.0.1.0

 

Nov 8 09:05:33 | Multicast Failed for eth1 using address230.0.1.0:42000

Nov 8 09:05:34 | Multicast Failed for eth2 using address230.0.1.0:42001

 

Test for Multicast address 224.0.0.251

 

Nov 8 09:05:35 | Multicast Succeeded for eth1 using address224.0.0.251:42002

Nov 8 09:05:36 | Multicast Succeeded for eth2 using address224.0.0.251:42003

As shown in the example above, the test hasfailed for the 230.0.1.0 address, but succeeded for the 224.0.0.251 multicastaddress. In this case, Patch: 9974223 mustbe applied to enable Oracle Grid Infrastructure to use the 224.0.0.251multicast address.  

Interpreting the outcome ofthe mcasttest-tool correctly:

• Should the mcasttest.pl test-tool have failed for both, the 230.0.1.0 and the 224.0.0.251 address
  • you must work with your System and / or Network Administrator to enable multicast on one of the addresses.
• Should the mcasttest.pl test-tool have failed for the 230.0.1.0 address only
  • Apply Patch: 9974223 or any subsequent GI PSU (fix provided GI PSU 1 and above)
• Should the mcasttest.pl test-tool have returned "success" for both, the 230.0.1.0 and the 224.0.0.251 address
  • No specific patch application is required and you can proceed with the installation.  

When and how to patch for11.2.0.2

Ideally, Patch: 9974223 orthe latest 11.2.0.2 GI PSU  (fix included in GI PSU 1 and above) isapplied before any root.sh or rootupgrade.sh has ever been run on any ofthe nodes meant to form a cluster. It is highly recommended that the latest11.2.0.2 GI PSU is installed to obtain the fix forBug: 9974223 (insteadof Patch: 9974223 standalone).In order to apply Patch: 9974223 eitherdirectly or as part of a Bundle Patch, perform the following steps:

• Perform an Oracle Grid Infrastructure 11.2.0.2 installation or upgrade
• Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.
  • Do not close the installer or abort an operation in progress.
  • Leave the current installation as-is and open a new terminal.
• Download and prepare the application of Patch: 9974223 either directly or as part of a GI Bundle.
• Unlike described in the patch readme (of the GI Bundle Patch),
  • do not use "opatch auto"
  • Instead, use: "opatch napply -local"
    • IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

 Note: as Opatchis used with the "-local" flag here, you need to perform this operation on every node.

• After you have patched every node in the cluster, return to the original installation
• Proceed to run the root.sh (rootupgrade.sh) on all nodes and follow the instructions on the screen.

When and how to patch for11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7

As previously mentioned, the most simplisticsolution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 addressto address th.  If this is notpossible within your environment, the recommended solution is to install thefix for Bug 16547309 priorto executing "rootcrs.pl -patch".

When and how to patch for12.1.0.1

Again, the most simplistic solution is to enablemulticast on either the 230.0.1.0 or 224.0.0.251 address to address this issue. If this is not possiblewithin your environment, the recommended solution is to install the fixfor Bug 16547309 orthe latest GI PSU containing the fix (12.1.0.1.2) prior to running root.sh orrootupgrade.sh.  In order to apply Patch 16547309 priorto the execution of root.sh or rootupgrade.sh, perform the following steps:

• Perform an Oracle Grid Infrastructure 12.1.0.1 installation or upgrade
• Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.
  • Do not close the installer or abort an operation in progress.
  • Leave the current installation as-is and open a new terminal.
• Download and prepare the application of Patch 16547309 either directly or as part of a GI Bundle.
• Unlike described in the patch readme (of the GI Bundle Patch),
  • do not use "opatch auto"
  • Instead, use: "opatch napply -local"
    • IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

 

Disclaimer and summary: Due to differences in network hardware andoverall network topologies that may be used for the interconnect network, it isnot possible for Oracle to provide a single solution for enabling multicastwithin your network environment. That being said, it is important that you workwith your System and / or  Network Administrators to enable multicastfunctionality for the 230.0.1.0 or the 224.0.0.251 multicast address(using Patch: 9974223) .The mcasttest.pl programattached to this note should be used to ensure that multicast communication isfunctioning properly to support Oracle Grid Infrastructure.

 

Scalability RAC Community

To discuss this topic further with Oracle experts andindustry peers, we encourage you to review, join or start a discussion in theMy Oracle SupportScalabilityRAC Community.

REFERENCES

NOTE:1054902.1 -How to Validate Network and Name Resolution Setup for the Clusterware and RAC