Exadata基本知识

Configuring Multi-Node Disk Heartbeat on Oracle RAC 10gR2and 11g

 

Configuring Multi-Node Disk Heartbeat on OracleRAC 10gR2 and 11g 
1. Introduction 
This section documents the required software levels and new configurationrequirements for using Oracle RAC 10gR2 or 11g with HACMP/PowerHA on AIX. Inorder to ensure high availability and data integrity in the event of systemfailures the proper levels of Oracle RAC 10gR2 or 11g Clusterware, AIX andHACMP/PowerHA software must be installed and the system must be properlyconfigured, including the new HACMP/PowerHA capability for Multi-Node Disk Heartbeat(MNDHB). The certified solution requires one MNDHB network to be defined foreach Oracle RAC Voting Disk. Each MNDHB and Voting Disk pair must be collocatedon a single hdisk, separate from the other pairs. Users must also configureMNDHB so that the node is halted if access is lost to a quorum of the MNDHBnetworks in the enhanced concurrent volume group. The following providesgeneral guidance on how to do this. 

The recommended version of HACMP 5.3 is SP6, (APAR IZ07791). This levelincludes support for MNDHB and eliminates the need to apply an ifix on HACMP5.3 and SP5 (APAR IY94307).

HACMP 5.4.1, (APAR IZ02620) is the certified version of HACMP 5.4. 

The MNDHB network works in conjunction with Oracle Cluster SynchonizationServices (CSS) changes available in Cluster Ready Services (CRS) bundled patch6160398 (necessary for 10.2.0.3 only) to maintain database integrity in theevent of a cluster partition. The MNDHB network must be configured andcollocated on hdisks with the Oracle RAC Voting disks, to properly fencepartitioned nodes. Previous recommendations for configuring multiple heartbeatpaths over both IP and non-IP networks still apply. Customers should no longeruse CRS one-off patch 5497221 (available for 10.2.0.3 only), the bundled patchprovides all required functionality, and improves serviceability. 

The intent of this paper is to document the configuration of the new MNDHBnetwork required to use Oracle RAC 10gR2 and 11g. Please refer to theAdditional Information section for more cluster configurationdocumentation. 
2. Planning 
When applying patch 6160398 with a shared CRS Home, a full cluster outage mustbe planned. The patch will update the shared copy of the binaries and nodaemons can be online while the binaries are modified. When each node in thecluster has its own CRS Home, the patch should be applied as a rolling upgrade.Do not patch two nodes at once. 
3. Configuration 
In order to reduce the likelihood of an unnecessary cluster partition failover,multiple IP networks and at least one non-IP network are recommended for heartbeating. The non-IP networks can be implemented using RS232 or disk heartbeating. For systems using Oracle RAC 10gR2 or 11g and HACMP/PowerHA enhancedconcurrent resources (enhanced concurrent logical volumes) for databasestorage, MNDHB networks must be configured.  http://www-03.ibm.com/systems/clusters/whitepapers/hacmp_bestpractices.html  


1. Configuring MNDHB networks for use with RAC 10gR2 and 11g 

Install, configure and have HACMP/PowerHA running prior to installing OracleRAC. For Oracle RAC 10gR2 and 11g configurations do not use HACMP/PowerHA forIP failovers on the RAC network interfaces (public, VIP or private). Thesenetwork interfaces should not be configured to use HACMP/PowerHA IP failover.VIP failovers are managed by RAC CRS. The RAC network interfaces are bound toindividual nodes and RAC instances. Problems can occur with the CRS ifHACMP/PowerHA reconfigures IP addresses over different interfaces or fails overaddresses across nodes. You can use HACMP/PowerHA for failover of IP address onthe RAC node if these addresses are not used by RAC. 

For AIX 6.1, to resolve Oracle Bug 7006789 à ƒÂƒÃ‚ƒÃ‚¢Ã‚€ÂœVIP does not failover when public whenpublic network cables are removed on one node”, follow thisconfiguration recommendation. 
Oracle RAC clusterware has strict timeout requirements for VIP address failoverin case of a public network failure. When DNS servers are unreachable due to apublic network failure, DNS name resolution calls such as getaddrinfo may hangfor the default AIX query timeout duration of 5 minutes. Name resolution callsmade by Oracle processes can thus delay the VIP failover. To reduce suchdelays, the DNS query timeout can be reduced to 1 minute, by adding thefollowing options line in /etc/resolv.conf for all RAC cluster nodes: 
"options timeout:1” 

No reboot is necessary to activate this change. 

If you need even faster VIP failover the timeout can be further reduced to avalue of 0; provided your network infrastructure (network and DNS servers) hasthe speed to serve name queries within a few (5-6) seconds. If you use a valueof 0 for timeout and your DNS or network is slow to respond, DNS name lookupswill start to fail prematurely. 
2. Sample configuration steps 

Prior to beginning the configuration steps please make sure the HACMP clcomdESdaemon is running. 
# lssrc -s clcomdES 
If the daemon is not active start it: 
# startsrc -s clcomdES 

Also setup the following configuration file: 
/usr/es/sbin/cluster/etc/rhosts 

Add the HACMP cluster node IP names to this file. In HACMP release 5.1 andhigher, the Cluster Communications daemon uses the trusted/usr/es/sbin/cluster/etc/rhosts file, and removes reliance on an /.rhosts file.In HACMP 5.2 and up, the daemon provides support for message authentication andencryption. 

Follow these steps to create the HACMP cluster: 
1. Create HACMP cluster and add RAC nodes 
# smitty cm_add_change_show_an_hacmp_cluster.dialog 
* Cluster Name [] 

2. Create HACMP cluster nodes; repeat for each RAC node. 
# smitty cm_add_a_node_to_the_hacmp_cluster_dialog 
* Node Name [] 
Communication Path to Node [] 

3. Create HACMP ethernet heartbeat networks. The HACMP configuration requiresnetworkdefinitions, however we will select NO for the IP address takeover forthese networks, since they are used by RAC. Create at least TWO networkdefinitions, one for the Oracle public interface and a second one for theOracle private (cluster interconnect) network. Additional ethernet heartbeatnetworks can be added if desired. 
# smitty cm_add_a_network_to_the_hacmp_cluster_select 
- select ether network 
* Network Name [] 
* Network Type ether 
* Netmask [] 
* Enable IP Address Takeover via IP Aliases [No] 
IP Address Offset for Heart beating over IP Aliases [] 

4. For each of the networks added in step 3, define all of the IP names foreach RAC node associated with that network. For example, for the Oracle privatenetwork, define the Oracle private IP name for each RAC node. 
# smitty cm_add_communication_interfaces_devices.select 
- select: Add Pre-defined Communication Interfaces and Devices / CommunicationInterfaces / desired network 
* IP Label/Address [] 
* Network Type ether 
* Network Name some_network_name 
* Node Name [] 
Network Interface [] 

5. Create an HACMP resource group for the enhanced concurrent volume groupresource with the following options. 
# smitty config_resource_group.dialog.custom 
* Resource Group Name [] 
* Participating Nodes (Default Node Priority) [
Startup Policy Online On All Available Nodes 
Failover Policy Bring Offline (On Error Node Only) 
Fallback Policy Never Fallback 

6. Create an AIX enhanced concurrent logical volume group (Big VG, or ScalableVG) from 'smitty mkvg' or command line. The VG must contain at least one hdiskfor each voting disk (three voting disks are recommended). The volume groupwill normally also contain Oracle datafiles, which must use a DS_LVZ (mklv with-T O) device subtype LV and therefore must have a Big VG or a Scalable VG. AScalable VG is recommended, in which case all LV default to using a devicesubtype of DS_LVZ. 
# smitty _mksvg 
VOLUME GROUP name [] 
Physical Partition SIZE in megabytes 
* PHYSICAL VOLUME names [] 
Force the creation of a volume group? no 
Activate volume group AUTOMATICALLY no 
at system restart? 
Volume Group MAJOR NUMBER [] 
Create VG Concurrent Capable? enhanced concurrent 
Max PPs per VG in kilobytes 
Max Logical Volumes 

7. Under "Change/Show Resources for a Resource Group (standard)" ADDthe concurrent volume group to the resource group added in above steps 
# smitty cm_change_show_resources_std_resource_group_menu_dmn.select 
- select resource group created in step 5 
Resource Group Name shared_storage 
Participating Nodes (Default Node Priority) 
Startup Policy Online On All Available Nodes 
Failover Policy Bring Offline (On Error Node Only) 
Fallback Policy Never Fallback 
Concurrent Volume Groups [
Use forced varyon of volume groups, if necessary false 
Application Servers [] 

8. One MNDHB network is defined for each RAC Voting Disk. Each MNDHB and VotingDisk pair must be collocated on a single hdisk, separate from the other pairs.The MNDHB network and Voting Disks exist on shared logical volumes in anenhanced concurrent logical volume managed by HACMP as an enhanced concurrentresource. Create a MNDHB LV on each of the hdisks in the VG created in step 6,for the hdisks which will also have a voting disk LV. 
# smitty cl_add_mndhb_lv 
- select resource group defined in step 5 
* Physical Volume name 
Logical Volume Name [] 
Logical Volume Label [] 
Volume Group name ccvg 
Resource Group Name shared_storage 
Network Name [net_diskhbmulti_01] 
Note: when you define the LVs for the RAC voting disks, they should be definedon the same disks, one per disk, as used in this step for the MNDHB LVs. 

9. Configure MNDHB so that the node is halted if access is lost to a quorum ofthe MNDHB networks in the enhanced concurrent volume group. 
# smitty cl_set_mndhb_response 
- select the VG created in step 6 
On loss of access Halt the node 
Optional notification method [] 
Volume Group ccvg 

10. Verify and Synchronize HACMP configuration. 
# smitty cm_initialization_and_standard_config_menu_dmn 
- select "Verify and Synchronize HACMP Configuration" 
Enter Yes if prompted: "Would you like to import shared VG: ccvg, inresource group: onto node: to node: racha702 [Yes / No]:" 
3. Sample configuration output 
# /usr/es/sbin/cluster/utilities/cltopinfo 
Cluster Name: rachacmp7 
Cluster Connection Authentication Mode: Standard 
Cluster Message Authentication Mode: None 
Cluster Message Encryption: None 
Use Persistent Labels for Communication: No 
There are 4 node(s) and 5 network(s) defined 
NODE racha701: 
Network net_diskhbmulti_01 
racha701_1 /dev/lv01 
Network net_diskhbmulti_02 
racha701_2 /dev/lv02 
Network net_diskhbmulti_03 
racha701_3 /dev/lv03 
Network net_ether_01 
racha701 130.35.166.38 
Network net_ether_02 
racha701p 192.168.10.71 
NODE racha702: 
Network net_diskhbmulti_01 
racha702_2 /dev/lv01 
Network net_diskhbmulti_02 
racha702_3 /dev/lv02 
Network net_diskhbmulti_03 
racha702_4 /dev/lv03 
Network net_ether_01 
racha702 130.35.166.39 
Network net_ether_02 
racha702p 192.168.10.72 
NODE racha703: 
Network net_diskhbmulti_01 
racha703_3 /dev/lv01 
Network net_diskhbmulti_02 
racha703_4 /dev/lv02 
Network net_diskhbmulti_03 
racha703_5 /dev/lv03 
Network net_ether_01 
racha703 130.35.166.40 
Network net_ether_02 
racha703p 192.168.10.73 
NODE racha704: 
Network net_diskhbmulti_01 
racha704_4 /dev/lv01 
Network net_diskhbmulti_02 
racha704_5 /dev/lv02 
Network net_diskhbmulti_03 
racha704_6 /dev/lv03 
Network net_ether_01 
racha704 130.35.166.41 
Network net_ether_02 
racha704p 192.168.10.74 

Resource Group shared_storage 
Startup Policy Online On All Available Nodes 
Failover Policy Bring Offline (On Error Node Only) 
Fallback Policy Never Fallback 
Participating Nodes racha701 racha702 racha703 racha704 

# /usr/es/sbin/cluster/sbin/cl_show_mndhb_vg 
The following network is configured for Multi-Node Disk Heartbeat: 

Network net_diskhbmulti_01 
Network net_diskhbmulti_01 uses Logical Volume [/dev/lv01] in 
Volume Group [ccvg] for heartbeat. 
Volume Group [ccvg] is managed by Resource Group [shared_storage] 
The following nodes participate in this network: 
racha701 
racha702 
racha703 
racha704 
On loss of access, HACMP will: 
Halt the node 

The following network is configured for Multi-Node Disk Heartbeat: 

Network net_diskhbmulti_02 
Network net_diskhbmulti_02 uses Logical Volume [/dev/lv02] in 
Volume Group [ccvg] for heartbeat. 
Volume Group [ccvg] is managed by Resource Group [shared_storage] 
The following nodes participate in this network: 
racha701 
racha702 
racha703 
racha704 
On loss of access, HACMP will: 
Halt the node 

The following network is configured for Multi-Node Disk Heartbeat: 

Network net_diskhbmulti_03 
Network net_diskhbmulti_03 uses Logical Volume [/dev/lv03] in 
Volume Group [ccvg] for heartbeat. 
Volume Group [ccvg] is managed by Resource Group [shared_storage] 
The following nodes participate in this network: 
racha701 
racha702 
racha703 
racha704 
On loss of access, HACMP will: 
Halt the node 

Note: there are important considerations for installing future HACMP updates toa cluster with multi-node disk heart beat configured. The README file includedwith the certified versions of HACMP describes these considerations and thecorrect procedures to be used for future updates. 

4. Adding MNDHB to an existing HACMP/RAC Voting disk configuration 
Upgrading to HACMP 5.3 or above / Oracle RAC 10gR2 + patch can be accomplishedby following these basic steps. NOTE: It is highly recommended that alldatabases and OCR be backed up prior to doing an upgrade. 
1. Shut down all Oracle RAC Databases, all nodeapps and CRS on all nodes. 
2. Disable CRS from starting on reboot: 
crsctl disable crs 

3. Shut down HACMP on all nodes. 
4. Install the required version and upgrades to AIX, RSCT and HACMP asdescribed in section 2. Follow the directions in the READMEs included with theselected packages. 
5. Identify if the existing Voting Disk LVs are already on separate hdisks andif these each have space (at least 32MB) for the MNDHB LVs. If so, create aMNDHB LV on each of the hdisks. If not, then create new MNDHB LVs and newVoting Disk LVs, located on hdisks as described in section 3.2 step 8. 
6. Verify and Synchronize HACMP configuration. 
7. Start HACMP on all nodes. 
8. Install the latest Oracle CRS bundled patchset on each node by following theREADME included with the patch. 
9. If new LVs for voting disks were added in step 5, then replace each of theexisting voting disks with the new ones. Refer to MetaLink note 452486.1 forhow to change the voting disks. 
10. Re-enable CRS using 
crsctl enable crs 
11. Start the CRS on all nodes and verify all resources start correctly.