Oracle10GR2 RAC+ASM on RHAS4U4在VMWare上的安装笔记

OS:
Redhat linux AS 4U4(2.6.9-42.ELsmp)
Oracle 10G database 10GR2
Oracle Clusterware
Patch for RAC and DB
VMWare 1.0.6

 

1.Install OS
A:(HDD>=8G,Memory 500M),Include the following:/boot(100M),/(2G)+insfile(2G),swap(1G),/home/compard/crs(<1G),/home/compard/10.2.0(<1.5G)

B.add 4 hdd(2*100M(orc.vmdk,vote.vmdk),2*2GB(asm1.vmdk,asm2.vmdk)) and 1 ethernet card:
Location of new hdd file must in another folder to share with other node.

If you use the partition that used by root,you can execute "partprobe" to take the affect to kernel
 
C.Add the following to vmware viture machine file(.vmx)
disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.DataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
scsi1:1.deviceType = "disk"
scsi1:2.deviceType = "disk"
scsi1:3.deviceType = "disk"
scsi1:4.deviceType = "disk"

2.Install Tools and PKG:
vnc,telnet,gsftp,vmware-tools
After install vmware-tools, you must use it to sync the time between the two machine.
Need PKG in oracle official document(RHEL4):
make-3.80-5
gcc-3.4.3-9.EL4
compat-db-4.1.25-9

installing order:
glibc-kernheaders-2.4-9.1.98.EL.i386.rpm
glibc-headers-2.3.4-2.25.i386.rpm
glibc-devel-2.3.4-2.25.i386.rpm
gcc-3.4.6-3.i386.rpm
oracleasm-support-2.1.3-1.el4.i386.rpm
oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm
oracleasmlib-2.0.4-1.el4.i386.rpm
libstdc++-devel-3.4.6-3.i386.rpm(Need by 10GR2)
gcc-c++-3.4.6-3.i386.rpm(Need by 10GR2)
sysstat-5.0.5-11.rhel4.i386.rpm

Check:
rpm -q binutils compat-db compat-libstdc++-296 control-center gcc gcc-c++ glibc glibc-common gnome-libs libstdc++ sysstat libstdc++-devel make pdksh xscreensaver setarch

3.system Setting
A.add the following to the end of the file "/etc/sysctl.conf",use command "sysctl -p" to apply the affect after doing this.
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024     65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

B./etc/security/limits.conf
compard soft    nproc   2047
compard hard    nproc   16384
compard soft    nofile 1024
compard hard    nofile 65536

C.Hangcheck Timer:/etc/rc.d/rc.local,add to the end and execute
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
D./etc/hosts,priv IP and host IP are assign to eth1 and eth0
127.0.0.1       localhost.compard.com   localhost
192.168.1.106   myrac1.compard.com      myrac1
1.1.1.10 myrac1priv.compard.com  myrac1priv
192.168.1.116   myrac2.compard.com      myrac2
1.1.1.11 myrac2priv.compard.com  myrac2priv
10.10.10.10 myrac1vip.compard.com myrac1vip
10.10.10.11 myrac2vip.compard.com myrac2vip

E:create user group dba and compard belong to it.
groupadd dba
useradd -g dba compard

F:/home/compard/.bash_profile
export ORACLE_BASE=/home/compard
export ORACLE_HOME=$ORACLE_BASE/10.2.0
export CRS_HOME=$ORACLE_BASE/crs
export ORA_CRS_HOME=$ORACLE_BASE/crs
export ORACLE_SID=RAC1
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8
export PATH=.:$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH:/lib:/usr/lib:/usr/local/lib

G:/etc/pam.d/login
session required /lib/security/pam_limits.so

H:stop no use serice
chkconfig cups off
chkconfig sendmail off
chkconfig isdn off
chkconfig smartd off
chkconfig iptables off
chkconfig pcmcia off

4.RAW Setting:
fdisk /dev/sd[b,c,d,e]      n --- p --- 1 -- Enter --Enter  --  w
A:/etc/sysconfig/rawdevices
/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1
/dev/raw/raw3 /dev/sdd1
/dev/raw/raw4 /dev/sde1

B.execute the following command(root):
/sbin/service rawdevices restart
chown compard:dba /dev/raw/raw[1-4]
chmod 660 /dev/raw/raw[1-4]
ls -lat /dev/raw/raw*

C./etc/udev/permissions.d/50-udev.permissions
# raw devices
ram*:root:disk:0660
#raw/*:root:disk:0660  #mark this
raw/*:compard:dba:0660  #add this

D.check if the user compard have the privilege to write on the raw device:
dd if=/dev/zero of=/dev/raw/raw1 bs=1024 count=1024

5.Clone to a new machine named myrac2
A.copy
B.update the displayName ="<New name>" in file .vmx
C.Start OS,delete all ethernet card,then re-add it
C.Update hostname/IP
D.update ORACLE_SID

6.RAC Setting:

A.SSH(ALL)
su - compard
mkdir ~/.ssh
chmod 755 ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

(RAC1)
cat ~/.ssh/id_rsa.pub >> ./.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ./.ssh/authorized_keys
ssh myrac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys myrac2:~/.ssh/authorized_keys
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add
(RAC2)
ssh myrac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac2priv.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv.compard.com cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh myrac1priv.compard.com cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Check
ssh myrac1 date
ssh myrac1.compard.com date
ssh myrac1priv date
ssh myrac1priv.compard.com date
ssh myrac2 date
ssh myrac2.compard.com date
ssh myrac2priv date
ssh myrac2priv.compard.com date

B.ASM
(ALL)
/etc/init.d/oracleasm configure(compard,dba,y,y)

(RAC1)
/etc/init.d/oracleasm createdisk VOL1 /dev/sdd1
/etc/init.d/oracleasm createdisk VOL2 /dev/sde1

(RAC2)
/etc/init.d/oracleasm scandisks
(Check)
/etc/init.d/oracleasm listdisks

7.Clusterware Install
A.ftp and unzip ,then chmod 0777
you can check before installation,but the error can be ignored
su - compard
cd /usr/insfile/clusterware/cluvfy
./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
B.Verify the time between RAC1 and RAC2.
C.Install ...
Error during install:
The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs
Solution:
(RAC2),X manager root Login
/home/compard/crs/bin/vipca

After install,use command "olsnodes" to check the result(run in compard).

8.DB software only

9.execute the netca to setup the listener.
name myrac
RAC1:name will auto-change to myrac_myrac1
RAC2:name will auto-change to myrac_myrac2

10.create db instance
After install, use the following SQL to login DB:
Server:
sqlplus system/<PWD>@rac1
sqlplus system/<PWD>@rac2

Client:
tnsnames.ora
RAC_DB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = myrac1.compard.com)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = myrac2.compard.com)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RAC.compard.com)
    )
  )
C:/WINDOWS/system32/drivers/etc/hosts:
192.168.1.106 myrac1.compard.com
192.168.1.116 myrac2.compard.com

Toad,PL/SQL,SQLProgress.exe to Login:
SQLProgress:

@echo off
set ORACLE_HOME=C:/oracle/product/10.2.0/client_1
set PATH=%ORACLE_HOME%/bin;%PATH%

"D:/Program Files/Compard/sqlprogress.exe"

 

screen-snapshot when login successfully:

11.additional information when finished:
A.df -k
[compard@myrac2 admin]$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              3526204   2820012    527068  85% /
/dev/sda1               101086     11422     84445  12% /boot
none                    259252         0    259252   0% /dev/shm
/dev/sda5              2482524   2144040    212376  91% /home
/dev/hdc               2191906   2191906         0 100% /media/cdrom

[compard@myrac1 admin]$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              3526204   2929712    417368  88% /
/dev/sda1               101086     11422     84445  12% /boot
none                    259252         0    259252   0% /dev/shm
/dev/sda5              2482524   1971304    385112  84% /home
/dev/hdc               2191906   2191906         0 100% /media/cdrom

B.Tablespace Used:
TABLESPACE_NAME TOTAL(M) USED(M) USED(%) FREE(M) MAX_BLOCK
SYSAUX  250  244.25 97.7 5.75 5.75
SYSTEM  480  470.75 98.073 9.25 8.938
TEMP  20  19 95 1 20
UNDOTBS1 35  33.375 95.357 1.625 0.938
UNDOTBS2 25  4.125 16.5 20.875 20.875
USERS  5  0.437 8.74 4.563 4.563

C.select count(1) from v$session
35

11:Stop/Start RAC
stop
A.listener
B:DB
C:RAC Service
srvctl stop nodeapps -n myrac2(compard)
srvctl stop nodeapps -n myrac1(compard)
/etc/init.d/init.crs stop(root)
start:
srvctl start nodeapps -n myrac1
srvctl start nodeapps -n myrac2
B:DB
C.listener

All Service in RAC(crs_stat -t)

ora....C1.inst application    OFFLINE   OFFLINE              
ora....C2.inst application    OFFLINE   OFFLINE              
ora.RAC.db     application    OFFLINE   OFFLINE              
ora....SM1.asm application    OFFLINE   OFFLINE              
ora....C1.lsnr application    OFFLINE   OFFLINE              
ora.myrac1.gsd application    OFFLINE   OFFLINE              
ora.myrac1.ons application    OFFLINE   OFFLINE              
ora.myrac1.vip application    OFFLINE   OFFLINE              
ora....SM2.asm application    OFFLINE   OFFLINE              
ora....C2.lsnr application    OFFLINE   OFFLINE              
ora.myrac2.gsd application    OFFLINE   OFFLINE              
ora.myrac2.ons application    OFFLINE   OFFLINE              
ora.myrac2.vip application    OFFLINE   OFFLINE  

 

Q&A:

1.某次开启环境后发现有个节点开机后几分钟自动重启，原因不明，最后换成了RHES4U8后正常。也许是低版本中的BUG导致。

2.VMWare1.0.6安装后机器重启慢了一半，最后换成了Vmware workstation版本，目前运行正常。