Oracle database is fairly simple to install, but RAC integration is a real pain.

I will try to remember in this post all the problems I had to install this.
A good way to work with Oracle, do not trust Oracle, just think before doing what they write in their docs.

Note the special section: O.D.A.D = Oracle Developpers are dumbs ;)

Terminology

What is ASM ?

ASM is a layer provided by Oracle that will act as a filesystem (UFS,OCFS or VXFS...) and a volume manager (SDS,SVM,VXVM ..) in one shot.

What is MPXIO ?

It's a Solaris driver that gives you a way to have multipathing and load sharing on your SAN's fibers. In practice, it will show you one disk, but gives the OS two paths to access it.

What is RAC ?

RAC stands for Real Application cluster, and yes this is a REAL cluster, with x nodes active at the same time, sharing the same disks and same context.

System Requirement

In our installation we are using MPXIO, cause we have dual fc cards, each port connected to a different switch fabric connected to a different port to the storage.
To use MPXIO you need to activate it , as follow on solaris 10.

edit /kernel/drv/fp.conf and modify mpxio-disable="yes"; to no or use the stmsboot that will do the same job but also will rewrite your fstab if needed, (not the case for us as we will not use any filesystem).

After that you should see your disks like this:

      2. c4t60060E8004F258000000F2580000320Bd0 
/scsi_vhci/ssd@g60060e8004f258000000f2580000320b
3. c4t60060E8004F258000000F25800003212d0 
/scsi_vhci/ssd@g60060e8004f258000000f25800003212

Disk formating

Do not use the first cylinder of each disk, only Solaris UFS know how to mix disk label infos and Filesystem data, ASM doesn't, nor OCR or voting. For example if you want to use the slice 6 your format should be like this:

Part      Tag    Flag     Cylinders         Size            Blocks
0 unassigned    wm       0                0         (0/0/0)             0
1 unassigned    wu       0                0         (0/0/0)             0
2     backup    wu       0 - 13650       49.99GB    (13651/0/0) 104839680
3 unassigned    wm       0                0         (0/0/0)             0
4 unassigned    wm       0                0         (0/0/0)             0
5 unassigned    wm       0                0         (0/0/0)             0
6        usr    wm       1 - 13650       49.99GB    (13650/0/0) 104832000
7 unassigned    wm       0                0         (0/0/0)             0

Voting disk, OCR disk and ASM disks

Oracle RAC needs 3 kind of disks, Voting disk is a special data space ~300megs (used as a raw device) that will be used by the cluster, to elect which nodes will stay the master, when a node problem will occur.
OCR, oracle cluster registry, is also a special device that will be used as raw device, also very short ~300megs, used as registry between all nodes.
ASM just where your datas will relies. When designing your datas, you will be prompted to secure them (RAID part), Oracle gives you the possibility, as every volume manager, to create multiple copies of your data, as a RAID 1, if your storage is not.
Cause our storage is already secured by RAID 6, we choose the security to be "external", stands for ASM will not handle all the RAID parts.

To always know where are your devices, you can group them into directory.
For example our ASM disks devices, have been recreated in /opt/oracle/asmdisks/disk1 .../disk2
To create this devices, you just need to find the minor and major numbers:

ls -l /dev/rdsk/c4t60060E8004F258000000F25800000200d0s6

Don't forget to point to the slice s6 not to the disk

 lrwxrwxrwx   1 root     root          67 Nov  6 21:22 /dev/rdsk/c4t60060E8004F258000000F25800000200d0s6 -> ../../devices/scsi_vhci/ssd@g60060e8004f258000000f25800000200:g,raw
ls -l /devices/scsi_vhci/ssd@g60060e8004f258000000f25800000200:g,raw
crw-r-   1 root     sys      118, 86 Nov  9 11:19 /devices/scsi_vhci/ssd@g60060e8004f258000000f25800000200:g,raw

Then mknod disk1 c 118 86
Note that the path you will use MUST be the same on each node.
Don't forget to chown all this devices to oracle:oinstall.
Run a little dd to blank the start of each device, just in case, you reinstall everything 10 times ...

dd f=/opt/oracle/asmdisk/disk1 if=/dev/zero bs=2048 count=100

Network requirement

Oracle cluster (clusterware) is now also an ip cluster, it knows how to manage your ips (system land), and switch the ip to a working node in case of problem.
Honestly this approch is not the good one, (gratuitous arp, relies on gateway check...), this comes from another age. Industrials are using layer 4 switching to do that.
But to complete clusterware, you need to meet all the ips needed:

  • A real IP per node (an Ip that will never be moved, and remains attached to a node)
  • A virtual IP per node (the futur listerner ip where your clients will connect, this ips will be managed by the cluster),
  • A private IP per node (a private network where the cluster will exchange private informations with the cluster)

A good starts it to put everything in your /etc/hosts

Important note Oracle Developpers are dumbs Section 1
Clusterware NEEDS that your unix hostname to be exactly the same as your public name. It meens your server MUST be named as your service...
All software companies wants a dedicated host for their applications, hey guys in 2006, our servers (like T2000) have 32 visibles cpus, we don't want to buy a server per product.

Setup SSH keys between nodes to the Oracle user.

Important note Oracle Developpers are dumbs Section 2
Clusterware needs ssh AND scp to be in /usr/local/bin/ssh and /usr/local/bin/scp
Don't miss that or you will have some problems at the end of your installation. Thanks Oracle Developpers to not using the PATH env variable as everyone do.
Ssh the other host on each node and ssh yourself on each node, for the key server to be placed in known_hosts.

Oracle profile

Put this in your oracle user .profile, (change the path as your needs)

ORA_DB_HOME=/opt/oracle/product/10.2.0/db_1
export ORA_DB_HOME
ORA_CRS_HOME=/opt/oracle/product/10.2.0/crs_1
export ORA_CRS_HOME
ORA_ASM_HOME=/opt/oracle/product/10.2.0/asm_1
export ORA_ASM_HOME

Needed Software

You need Oracle10gr2 iso but also CRS10gr2 iso, don't spend your time like me to look for CRS in the 1st cd ;)

Clusterware installation

Starts on your first node

export ORACLE_HOME=ORA_CRS_HOME

Then run the crs runinstaller command. (X11 installer, export DISPLAY ...)
You will be prompted for your voting disk and ocr disk, just give the previous paths you've created.
Gives all the node names you choosed before.
Run /opt/oracle/product/10.2.0/crs_1/root.sh on first node, it should passed, but on the second node ...

Important note Oracle Developpers are dumbs Section 3
The given interface(s), "e1000g0" is not public.
Oracle devs think that your oracle cluster will run on a public internet ip ...This should be fixed in the next release

Also if your server doesn't have a default route , which sounds good on a private network

vi /opt/oracle/product/10.2.0/crs_1/bin/racgvip
# set it to 0 for checkIf() to return success if default gateway is not found,
# otherwise set it to 1
FAIL_WHEN_DEFAULTGW_NOT_FOUND=0

Set it to 0 to prevent the cluster to test the default gw, woh really safe cluster what a test !!

run /opt/oracle/product/10.2.0/crs_1/bin/vipca fill the tab.
Beware of the default filled 255.255.255.0 netmask, the dumb installer can't find the netmask.

Click next on the first installer and that's it CRS is installed. Check everything is ok with the crs_stat -t command, eveything should be ONLINE.

ASM installation

ASM informations are stored in an Oracle database ... A good deal is to dedicate a complete oracle installation to ASM (for future patching).

export ORACLE_HOME=$ORA_ASM_HOME

Then run the normal DB-10g/runInstaller
Tell that you want to use ASM for your storage, change the discovery path to /opt/oracle/asmdisks/*, then finish with the /opt/oracle/product/10.2.0/asm_1/root.sh

DB installation

export ORACLE_HOME=$ORA_DB_HOME

Run again the normal DB-10g/runInstaller
Tell you have an existing ASM configuration, provide the good password. Finish with /opt/oracle/product/10.2.0/db_1/root.sh

You now have a working secured Oracle installation, used the VIP ips to connect, they are safe.
Like I said before a better approach, is to used layer 4 switching in front of Oracle, to only have one VIP, it will defacto isolate the reals servers, to the clients world
For Oracle FAN events just route one port to each node.
For the curious DSR load balancing is working with this configuration, just mount your vip on localhost and set up a listener to bind this ip

Fabrice akh NOSPAM nobugware.com 2006 v1.0