Step-By-Step Installation of 9i RAC on IBM AIX(1)

最近有个活要用这个，看文档的时候在metalink看到了这篇文档。
9i RAC的中心思想其实跟10g是相同的，只不过clusterware不同，一个是HACMP/Sun cluster等，而一个是Oracle clusterware.
而这些clusterware其实完成的作用是基本上相同的。

Note: This note was created for 9i RAC. The 10g Oracle documentation provides installation instructions for 10g RAC. These instructions can be found on OTN:

Purpose

This document will provide the reader with step-by-step instructions on how to install a cluster, install Oracle Real Application Clusters (RAC) and start a cluster database on IBM AIX HACMP/ES (CRM) 4.4.x. For additional explanation or information on any of these steps, please see the references listed at the end of this document. This note does not cover IBM SP2 platform.  

Disclaimer: If there are any errors or issues prior to step 3.3, please contact IBM Support.
The information contained here is as accurate as possible at the time of writing.

• 1. Configuring the Cluster Hardware
  • 1.1 Minimal Hardware list / System Requirements
    • 1.1.1 Hardware
    • 1.1.2 Software
    • 1.1.3 Patches
  • 1.2 Installing Disk Arrays
  • 1.3 Installing Cluster Interconnect and Public Network Hardware
• 2. Creating a cluster
  • 2.1 HACMP/ES Software Installation
  • 2.2 Configuring Cluster Topology
  • 2.3 Synchronizing Cluster Topology
  • 2.4 Configuring Cluster Resources
    • 2.4.1 Create volume groups to be shared concurrently on one node
    • 2.4.2 Create Shared RAW Logical Volumes
    • 2.4.3 Import the Volume Group on to the Other Nodes
    • 2.4.4 Add a Concurrent Cluster Resource Group
    • 2.4.5 Configure the Concurrent Cluster Resource Group
    • 2.4.6 Creating Parallel Filesystems (GPFS)
  • 2.5 Synchronizing Cluster Resources
  • 2.6 Joining Nodes Into the Cluster
  • 2.7 Basic Cluster Administration
• 3. Preparing for the Installation of RAC
  • 3.1 Configure the shared disks and UNIX preinstallation tasks
    • 3.2.1 Configure the shared disks
    • 3.2.2 UNIX preinstallation tasks
  • 3.2 Using the Oracle Universal Installer for Real Application Clusters
  • 3.3 Create a RAC Database using the Oracle Database Configuration Assistant
• 4. Administering Real Application Clusters Instances
• 5. References

---

1. Configuring the Clusters Hardware

1.1 Minimal Hardware list / System Requirements

For a two node cluster the following would be a minimum recommended hardware list.
Check the RAC/IBM AIX certification matrix for RAC updates on currently supported hardware/software.

1.1.1 Hardware

• IBM servers - two IBM servers capable of running AIX 4.3.3 or 5L 64bit
• For IBM or third-party storage products, Cluster interconnects, Public networks, Switch options, Memory, swap & CPU requirements consult with the operating system vendor or hardware vendor.
• Memory, swap & CPU requirements
  • Each server must have a minimum of 512Mb of memory, at least 1Gb swap space or twice the physical memory whichever is greater.

    To determine system memory use:-

    $ /usr/sbin/lsattr -E -l sys0 -a realmem

    To determine swap space use:-

    $ /usr/sbin/lsps -a

  • 64-bit processors are required.

1.1.2 Software

• When using IBM AIX 4.3.3:
  • HACMP/ES CRM 4.4.x
  • Only RAW Logical Volumes (Raw Devices) for Database Files supported
  • Oracle Server Enterprise Edition 9i Release 1 (9.0.1) or 9i Release 2 (9.2.0)

• When using IBM AIX 5.1 (5L):
  • For Database Files residing on RAW Logical Volumes (Raw Devices):
  • HACMP/ES CRM 4.4.x
  • For Database files residing on Parallel Filesystem (GPFS):
  • HACMP/ES 4.4.x (HACMP/CRM is not required)
  • GPFS 1.5
  • IBM Patch PTF12 and IBM patch IY34917
    or
    IBM Patch PTF13
  • Oracle Server Enterprise Edition 9i Release 2 (9.2.0)
  • Oracle Server Enterprise Edition 9i for AIX 4.3.3 and 5L are in separate CD packs
    and include Real Application Cluster (RAC)

1.1.3 Patches

The IBM Cluster nodes might require patches in the following areas:

• IBM AIX Operating Environment patches
• Storage firmware patches or microcode updates

Patching considerations:

• Make sure all cluster nodes have the same patch levels
• Do not install any firmware-related patches without qualified assistance
• Always obtain the most current patch information
• Read all patch README notes carefully.

• For a list of required operating system patches check the sources in Note 211537.1 and contact IBM corporation for additional patch requirements.

To see all currently installed patches use the following command:

    % /usr/sbin/instfix -i

To verify installation of a specific patch use:

   % /usr/sbin/instfix -ivk <Patchnumber>

e.g.: % /usr/sbin/instfix -ivk IY30927

 

1.2 Installing Disk Arrays

Follow the procedures for an initial installation of the disk enclosures or arrays, prior to installing the IBM AIX operating system environment and HACMP software. Perform this procedure in conjunction with the procedures in the HACMP for AIX 4.X.1 Installation Guide and your server hardware manual.

 

1.3 Installing Cluster Interconnect and Public Network Hardware

The cluster interconnect and public network interfaces do not need to be configured prior to the HACMP installation but must be configured and available before the cluster can be configured.

• If not already installed, install host adapters in your cluster nodes. For the procedure on installing host adapters, see the documentation that shipped with your host adapters and node hardware. Install the transport cables (and optionally, transport junctions), depending on how many nodes are in your cluster:
• A cluster with more than two nodes requires two cluster transport junctions. These transport junctions are Ethernet-based switches (customer-supplied).  

You install the cluster software and configure the interconnect after you have installed all other hardware.

• ---

2. Creating a Cluster

2.1 IBM HACMP/ES Software Installation

The HACMP/ES 4.X.X installation and configuration process is completed in several major steps. The general process is:

• install hardware
• install the IBM AIX operating system software
• install the latest IBM AIX maintenance level and required patches
• install HACMP/ES 4.X.X on each node
• install HACMP/ES required patches
• configure the cluster topology
• synchronize the cluster topology
• configure cluster resources
• synchronize cluster resources

Follow the instructions in the HACMP for AIX 4.X.X Installation Guide for detailed instructions on insalling the required HACMP packages. The required/suggested packages include the following:

• cluster.adt.es.client.demos
• cluster.adt.es.client.include
• cluster.adt.es.server.demos
• cluster.clvm.rte HACMP for AIX Concurrent
• cluster.cspoc.cmds HACMP CSPOC commands
• cluster.cspoc.dsh HACMP CSPOC dsh and perl
• cluster.cspoc.rte HACMP CSPOC Runtime Commands
• cluster.es.client.lib ES Client Libraries
• cluster.es.client.rte ES Client Runtime
• cluster.es.client.utils ES Client Utilities
• cluster.es.clvm.rte ES for AIX Concurrent Access
• cluster.es.cspoc.cmds ES CSPOC Commands>
• cluster.es.cspoc.dsh ES CSPOC dsh and perl
• cluster.es.cspoc.rte ES CSPOC Runtime Commands
• cluster.es.hc.rte ES HC Daemon
• cluster.es.server.diag ES Server Diags
• cluster.es.server.events ES Server Events
• cluster.es.server.rte ES Base Server Runtime
• cluster.es.server.utils ES Server Utilities
• cluster.hc.rte HACMP HC Daemon
• cluster.msg.En_US.cspoc HACMP CSPOC Messages - U.S.
• cluster.msg.en_US.cspoc HACMP CSPOC Messages - U.S.
• cluster.msg.en_US.es.client
• cluster.msg.en_US.es.server
• cluster.msg.en_US.haview HACMP HAView Messages - U.S.
• cluster.vsm.es ES VSM Configuration Utility
• cluster.clvm.rte HACMP for AIX Concurrent
• cluster.es.client.rte ES Client Runtime
• cluster.es.clvm.rte ES for AIX Concurrent Access
• cluster.es.hc.rte ES HC Daemon
• cluster.es.server.events ES Server Events
• cluster.es.server.rte ES Base Server Runtime
• cluster.es.server.utils ES Server Utilities
• cluster.hc.rte HACMP HC Daemon
• cluster.man.en_US.client.data
• cluster.man.en_US.cspoc.data
• cluster.man.en_US.es.data ES Man Pages - U.S. English
• cluster.man.en_US.server.data
• rsct.basic.hacmp RS/6000 Cluster Technology
• rsct.basic.rte RS/6000 Cluster Technology
• rsct.basic.sp RS/6000 Cluster Technology
• rsct.clients.hacmp RS/6000 Cluster Technology
• rsct.clients.rte RS/6000 Cluster Technology
• rsct.clients.sp RS/6000 Cluster Technology
• rsct.basic.rte RS/6000 Cluster Technology

You can verify the installed HACMP software with the "clverify" command.  

    # /usr/sbin/cluster/diag/clverify

At the "clverify>" prompt enter "software" then at the "clverify.software>" prompt enter "lpp". You should see a message similar to:

Checking AIX files for HACMP for AIX-specific modifications...

*/etc/inittab not configured for HACMP for AIX.

If IP Address Takeover is configured, or the Cluster Manager is to be started on boot, then /etc/inittab must contain the proper HACMP for AIX entries.

Command completed.

--------- Hit Return To Continue ---------

Contact IBM support if there were any failure messages or problems executing the "clverify" command.

 

 

2.2 Configuring the Cluster Topology

Using the "smit hacmp" command:

# smit hacmp

Note: The following is an example of a generic HACMP configuration to be used as an example only. See the HACMP installation and planning documentation for specific examples. All questions concerning the configuration of your cluster should be directed to IBM Support. This configuration does not include an example of a IP takeover network. "smit" fastpaths are being used to navigate the "smit hacmp" configuration menus. Each one of these configuration screens are obtainable from "smit hacmp". All configuration is done from one node and then synchronized to the other participating nodes.

Add the cluster definition:

Smit HACMP -> Cluster Configuration -> Cluster Topology -> Configure Cluster -> Add a Cluster Definintion

Fastpath:

# smit cm_config_cluster.add

• Add a Cluster Definition Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] **NOTE: Cluster Manager MUST BE RESTARTED in order for changes to be acknowledged.** * Cluster ID [0] * Cluster Name [cluster1]

The "Cluster ID " and "Cluster Name " are arbitrary. The "Cluster ID " must be a valid number between 0 and 99999 and the "Cluster Name " can be any alpha string up to 32 characters in length.  

Configuring Nodes:

Smit HACMP -> Cluster Configuration -> Cluster Topology -> Configure Nodes -> Add Cluster Nodes

FastPath:

# smit cm_config_nodes.add

• Add Cluster Nodes Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Node Names [node1 node2]

"Node Names " should be the hostnames of the nodes. They must be alpha numeric and contain no more than 32 characters. All nodes participating in the cluster must be entered on this screen separated by a space.  

Next to be configured is the network adapters. This example will utilize two ethernet adapters on each node as well as one RS232 serial port on each node for heartbeat.

• Node Name	address	IP Label (/etc/hosts)	Type
  node1	192.168.0.1	node1srvc	service
  	192.168.1.1	node1stby	standby
  	/dev/tty0		serial
  node2	192.168.0.2	node2srvc	service
  	192.168.1.2	node2stby	standby
  	/dev/tty0		serial

   

The following screens are configuration settings needed to configure the above networks into the cluster configuration:

Smit HACMP -> Cluster Configuration -> Cluster Topology -> Configure Nodes -> Add an Adapter

FastPath:

# smit cm_confg_adapters.add

• Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node1srvc] * Network Type [ether] + * Network Name [ipa] + * Network Attribute public + * Adapter Function service + Adapter Identifier [] Adapter Hardware Address [] Node Name [node1] +

It is important to note that the "Adapter IP Label " must match what is in the "/etc/hosts" file otherwise the adapter will not map to a valid IP address and the cluster will not synchronize. The "Network Name " is an arbitrary name for the network configuration. All the adapters in this ether configuration should have the same "Network Name ". This name is used to determine what adapters will be used in the event of an adapter failure.

•  

  Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node1stby] * Network Type [ether] + * Network Name [ipa] + * Network Attribute public + * Adapter Function standby + Adapter Identifier [] Adapter Hardware Address [] Node Name [node1] +

   

  Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node2srvc] * Network Type [ether] + * Network Name [ipa] + * Network Attribute public + * Adapter Function service + Adapter Identifier [] Adapter Hardware Address [] Node Name [node2] +

   

  Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node2stby] * Network Type [ether] + * Network Name [ipa] + * Network Attribute public + * Adapter Function standby + Adapter Identifier [] Adapter Hardware Address [] Node Name [node2] +

  The following is the serial configuration:

  Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node1_tty] * Network Type [rs232] + * Network Name [serial] + * Network Attribute serial + * Adapter Function service + Adapter Identifier [/dev/tty0] Adapter Hardware Address [] Node Name [node1] +

   

  Add an Adapter Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] * Adapter IP Label [node2_tty] * Network Type [rs232] + * Network Name [serial] + * Network Attribute serial + * Adapter Function service + Adapter Identifier [/dev/tty0] Adapter Hardware Address [] Node Name [node2] +

Since this is not on the same network as the ethernet cards the "Network Name " is different. The same name is used for the network name.

Use "smit mktty" to configure the RS232 adapters:

# smit mktty

• Add a TTY Type or select values in entry fields. Press Enter AFTER making all desired changes. [TOP] [Entry Fields] TTY type tty TTY interface rs232 Description Asynchronous Terminal Parent adapter sa0 * PORT number [0] + Enable LOGIN disable + BAUD rate [9600] + PARITY [none] + BITS per character [8] + Number of STOP BITS [1] + TIME before advancing to next port setting [0] +# TERMINAL type [dumb] FLOW CONTROL to be used [xon] + [MORE...31]

Be sure that "Enable LOGIN " is set to the default of "disable". The "PORT number " is the value that is to be used in the /dev/tt# where "# " is the port number. So if you defined this as "0 " the device would be "/dev/tty0".

 

 

 

2.3 Synchronizing the Cluster Topology

After the topology is configured it needs to be synchronized. The synchronization performs topology sanity checks as well as pushes the configuration data to each of the nodes in the cluster configuration. For the synchronization to work user equivalence must be configured for the root user. There is several ways to do this.   One way would be to create a ".rhosts" file on each node in the "/" directory.  

Example of a ".rhosts" file:

node1 root

      node2 root

Be sure permissions on the "/.rhosts" file is 600.

# chmod 600 /.rhosts

Use a remote command such as "rcp" to test equivalence from each node:

From node1:

# rcp /etc/group node2:/tmp

Frome node2:

# rcp /etc/group node1:/tmp

View your IBM operating system documentation for more information or contact IBM support if you have any questions or problems setting up user equivalence for the root user.

Smit HACMP -> Cluster Configuration -> Cluster Topology -> Synchronize Cluster Topology

FastPath:

# smit configchk.dialog

• Synchronize Cluster Topology Type or select values in entry fields. Press Enter AFTER making all desired changes. [TOP] [Entry Fields] Ignore Cluster Verification Errors? [No] + * Emulate or Actual? [Actual] + Note: Only the local node's default configuration files keep the changes you make for topology DARE emulation. Once you run your emulation, to restore the original configuration rather than running an actual DARE, run the SMIT command, "Restore System Default Configuration from Active Configuration." We recommend that you make a snapshot before running an emulation, just in case uncontrolled cluster events happen during emulation. NOTE: If the Cluster Manager is active on this node, synchronizing the Cluster Topology will cause the Cluster Manager to make any changes take effect once the synchronization has successfully completed. [BOTTOM]