Veritas HA 安装配置手册

The utility vxfentsthdw works on the two nodes of the cluster. The utility verifies that the shared storage one intends to use isconfigured to support I/O fencing. It issues a series of vxfenadm commands to setup SCSI-3 registrations on the disk, verifies the registrations on the disk, and removes the registrations from the disk.

******** WARNING!!!!!!!! ********

THIS UTILITY WILL DESTROY THE DATA ON THE DISK!!

Do you still want to continue : [y/n] (default: n) y

Please respond with (y/n) n

# I/O fencing 策略

对于采用 cluster file system 的 veritas ha 方案，一定要采用共享磁盘做 I/O fencing ；

对于单机的 file system ，在确保有 2 个直连网线和 1 个 bond 的业务网络的情况下，可以不考虑 I/O fencing 的策略。

Make sure that the I/O fencing coordinator disks meet the following requirements:

* For disk-based I/O fencing, you must have three coordinator disks.

* The coordinator disks can be raw devices, DMP devices, or iSCSI devices.

You must use DMP disk policy for iSCSI-based coordinator disks.

* Each of the coordinator disks must use a physically separate disk or LUN.

Symantec recommends using the smallest possible LUNs for coordinator disks.

* The coordinator disks must support SCSI-3 persistent reservations.

*Symantec recommends using hardware-based mirroring for coordinator disks.

* Coordinator disks must not be used to store data or must not be included in disk groups that store user data.

* Coordinator disks cannot be the special devices that array vendors use. For example, you cannot use EMC gatekeeper devices as coordinator disks.

# 配置环境变量

vi /root/.bash_profile

export MANPATH=$MANPATH:/opt/VRTS/man

PATH=$PATH:$HOME/bin

PATH=$PATH:/

/usr/lib/vxvm/bin:/opt/VRTSvcs/bin:/

/opt/VRTS/bin:/opt/VRTSvcs/rac/bin:/

/opt/VRTSob/bin;

export PATH

# 配置 linux I/O 调度策略

Symantec recommends using the Linux 'deadline' I/O scheduler for database

workloads. Configure your system to boot with the 'elevator=deadline' argument

to select the 'deadline' scheduler.

查看当前的 IO 调度策略，默认选择 CFQ 做为 I/O 调度器

cat /sys/block/sd*/queue/scheduler

修改调度策略

vi /boot/grub/menu.lst

kernel /boot/vmlinuz-2.6.18-128.el5 ro root=LABEL=/

改为

kernel /boot/vmlinuz-2.6.18-128.el5 ro root=LABEL=/ elevator=deadline

重启OS ，检查是否修改生效。

# 网络要求

网络要求 1 个走交换机的业务网段， 1 个网线直连的私有网段。

私有网段的网口要用直连网线连接，作为首选心跳，争取可以配 2 条直连网线。

业务网络网卡要求 2 张独立网卡做 bond ，可以作为备用心跳线。

每个机器业务网段的 2 个 bond 网口分别上联到 2 个独立的网络交换机上面。

1.3 安装 SF for RAC

# 安装前检查环境

#cd /root/source

# tar xzf VRTS_SF_HA_Solutions_5.1_PR1_RHEL5_SLES10_x64.tar.gz

# cd /root/source/dvd1-linux/rhel5_x86_64/storage_foundation_for_oracle_rac

# ./installsfrac -precheck host1 host2

无误后可以正常安装

# 安装过程

# cd /root/source/rhel5_x86_64/storage_foundation_for_oracle_rac

# ./installsfrac host1 host2

Do you agree with End User License Agreement (EULA) specified in EULA.pdf file present on media? [y,n,q,?] (y) y

1) Install minimal required Veritas Storage Foundation for Oracle RAC RPMs

2) Install recommended Veritas Storage Foundation for Oracle RAC RPMs

3) Install all Veritas Storage Foundation for Oracle RAC RPMs

4) Display RPMs to be installed for each option

Select the RPMs to be installed on all systems? [1-4,q,?] (2) 3

To comply with the terms of Symantec's End User License Agreement,

you have 60 days to either:

1) Enter a valid license key

2) Enable keyless licensing

How would you like to license the systems? [1-2,q] 1

Checking system licensing

SF Oracle RAC is unlicensed on all systems

Enter a SF Oracle RAC license key: [b,q,?] XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-P

Storage Foundation for Oracle successfully registered on host1

SF Oracle RAC is unlicensed on all systems

Enter a SF Oracle RAC license key: [b,q,?] b

Would you like to enable the Veritas Volume Replicator? [y,n,q] (n) n

Would you like to enable the Global Cluster Option? [y,n,q] (n) n

Registering SF Oracle RAC license

Do you wish to enter additional licenses? [y,n,q,b] (n) n

Veritas Storage Foundation for Oracle RAC Install completed successfully

Would you like to configure SF Oracle RAC on host1 host2? [y,n,q] (n) y

Veritas Storage Foundation for Oracle RAC 5.1 PR1 Install Program

1) Configure SF Oracle RAC sub-components

2) Prepare to Install Oracle

3) Install Oracle Clusterware/Grid Infrastructure and Database

4) Post Oracle Installation Tasks

5) Exit SF Oracle RAC Configuration

Choose option: [1-5,q] (1) 1

Enter the unique cluster name: [q,?] myha

Enter a unique Cluster ID number between 0-65535: [b,q,?] (0) 0

Discovering NICs on host1 ................................................

Discovered eth0 eth1 eth2 eth3 bond0 sit0

To use aggregated interfaces for private heartbeat,

enter the name of an aggregated interface.

To use a NIC for private heartbeat, enter a NIC which is not part of an aggregated interface.

Enter the NIC for the first private heartbeat link on host1: [b,q,?] eth1

Are you sure you want to use eth1 for the first private heartbeat link? [y,n,q,b,?] (n) y

Would you like to configure a second private heartbeat link? [y,n,q,b,?] (y) y

Enter the NIC for the second private heartbeat link on host1: [b,q,?] eth3

Are you sure you want to use eth3 for the second private heartbeat link? [y,n,q,b,?] (n) y

Would you like to configure a third private heartbeat link? [y,n,q,b,?] (n) y

Enter the NIC for the third private heartbeat link on host1: [b,q,?] bond0

Are you sure you want to use bond0 for the third private heartbeat link? [y,n,q,b,?] (n) y

Is bond0 a bonded NIC? [y,n,q] (n) y

Would you like to configure a fourth private heartbeat link? [y,n,q,b,?] (n) n

Are you using the same NICs for private heartbeat links on all systems? [y,n,q,b,?] (y) y

Checking Media Speed for eth1 on host1 .............................................1000Mb/s

Checking Media Speed for eth3 on host1......................................... .... 1000Mb/s

Checking Media Speed for bond0 on host1 .. ....................................... Unknown

Checking Media Speed for eth1 on host2 ............................................. 1000Mb/s

Checking Media Speed for eth3 on host2 ............................................. 1000Mb/s

Checking Media Speed for bond0 on host2 .......................................... Unknown

CPI WARNING V-0-40-2487 installsfrac can not detect media speed for the selected private NICs properly. Consult your Operating

System manual for information on how to set the Media Speed.

Cluster information verification:

Cluster Name: myha

Cluster ID Number: 0

Private Heartbeat NICs for host1:

link1=eth1

link2=eth3

link3=bond0

Private Heartbeat NICs for host2:

link1=eth1

link2=eth3

link3=bond0

Is this information correct? [y,n,q,b,?] (y) y

Would you like to configure VCS to use Symantec Security Services? [y,n,q] (n) n

Do you want to set the username and/or password for the Admin user

( default username = 'admin', password='password' )? [y,n,q] (n) n

Do you want to add another user to the cluster? [y,n,q] (n) n

VCS User verification:

User: admin Privilege: Administrators

Passwords are not displayed

Is this information correct? [y,n,q] (y) y

Do you want to configure SMTP notification? [y,n,q,?] (n) n

Do you want to configure SNMP notification? [y,n,q,?] (n) n

Do you want to stop SF Oracle RAC processes now? [y,n,q,?] (y) y

Veritas Storage Foundation for Oracle RAC 5.1 PR1 Install Program

1) Configure SF Oracle RAC sub-components

2) Prepare to Install Oracle

3) Install Oracle Clusterware/Grid Infrastructure and Database

4) Post Oracle Installation Tasks

5) Exit SF Oracle RAC Configuration

Choose option: [1-5,q] (1) 5

Would you like to send the information about this installation to Symantec

to help improve installation in the future? [y,n,q,?](y) n

installsfrac log files, summary file, and response file are saved at:

/opt/VRTS/install/logs/installsfrac-201009100405ebi

装完 SF 重启机器

# reboot

1.4 重新配置 SF for Oracle RAC

需要时可以重新配置 SF for Oracle RAC

# cd /root/source/dvd1-linux/rhel5_x86_64/storage_foundation_for_oracle_rac

# ./installsfrac -configure host1 host2

#./installsfrac –help 可以查看所有选项

2 用 VEA 配置 Disk Group

用 veritas 的 VEA 图形界面划 Disk Group ， Volumes ，初始化 File System 。

2.1 安装 VEA

安装

rpm -ivh /root/source/VRTSobgui-3.4.15.0-0.i686.rpm

卸载

rpm -e VRTSobgui

安装完成后，要启动服务 vxsvcctrl

/opt/VRTSob/bin/vxsvcctrl activate

/opt/VRTSob/bin/vxsvcctrl start

本机配置Xmanager ，启动vea 图形界面

export DISPLAY=10.16.200.84:0

/opt/VRTSob/bin/vea &

[select profile] 用default ，之后选择[connect to a Host or Domain] ，

[host name] 选 localhost ， Connect Using root@localhost ，用本机 root 账号登陆，

进入vea 管理界面。

2.2 新建 DG

[Disk Group] --> 新建DG ，New Disk Group --> 填写Group Name ，选择要加入这个DG 的Disk ，其余默认配置，不用修改。

最后创建出 3 个 DG ： redodg ， archdg ， datadg 。

2.3 创建 Volumes 和 File System

[Volumes]-->[New Volume]--> 选择 DG-->Let VM decide what disks to use for this volume

每个DG 单独创建1 个volume ，用vxfs 文件系统格式化这个volume 。

Volume 的创建方式：

Concatenated Volume ，级联方式，创建 lv 时，用完 1 块盘，再用另外 1 块。

Striped Volume ，条带方式，指定 stripe size 和盘数 columns 。

我们要求建Volumn 时，stripe size 用128 ，columns 用所有的盘。

创建 lv_redodg

fs_redodg 配置

File system type ： vxfs

Blocksize ： 8k

Mount Point /redodg

不选择 {Add to file system table}

其他配置不动

lv_datadg

fs_datadg

lv_archdg

fs_archdg

#df –lh

Filesystem Size Used Avail Use% Mounted on

/dev/sda1 4.9G 2.1G 2.6G 45% /

/dev/sda5 20G 240M 19G 2% /home

/dev/sda2 29G 4.7G 23G 18% /opt

/dev/sda6 9.7G 368M 8.8G 4% /var

/dev/sda7 9.7G 151M 9.0G 2% /tmp

/dev/sda8 9.7G 2.5G 6.8G 27% /usr

tmpfs 16G 0 16G 0% /dev/shm

/dev/sda9 34G 3.9G 28G 13% /data

tmpfs 4.0K 0 4.0K 0% /dev/vx

/dev/vx/dsk/redodg/lv_redodg 67G 68M 66G 1% /redodg

/dev/vx/dsk/datadg/lv_datadg 798G 283M 792G 1% /datadg

/dev/vx/dsk/archdg/lv_redodg 200G 72M 198G 1% /archdg

2.4 在另外一个节点上配置 DG

先在本节点 umount filesystem ， deport dg ，

在另外1 个节点上import dg

[Disk Groups] -->Import Disk Group --> OK

Import DG 之后， Volumes 会自动激活，需要重新创建 File System

[Volumes]-->volume-->File System--> New File System

3 安装 oracle 软件，创建数据库

3 .1 安装 oracle 软件

参考Oracle 安装文档。

3 .2 用 Veritas 的 ODM 代替 Oracle 的 libodm 库

修改 Oracle 的libodm 库，指向 veritas 的 ODM ，之后创建 oracle 数据库。

刚安装完 Oracle 软件后的 oracle libodm

lrwxrwxrwx 1 oracle oinstall 20 Sep 6 05:34 libodm11.so -> /usr/lib64/libodm.so

替换为 veritas 的 ODM

#su - oracle

#cd $ORACLE_HOME/lib

#mv libodm11.so libodm11.so.oracle

#ln -s /usr/lib64/libodm.so libodm11.so

#ls -ltr libodm11*

lrwxrwxrwx 1 oracle oinstall 12 Aug 17 08:12 libodm11.so.oracle -> libodmd11.so

lrwxrwxrwx 1 oracle oinstall 20 Sep 6 05:34 libodm11.so -> /usr/lib64/libodm.so

3.3 手工创建数据库

要注意spfile 需要放在存储上面。

4 用 VCM 配置 Veritas HA

4.1 安装 VCM 图形界面，管理 VCS

安装

rpm -ivh /root/source/VCS_Cluster_Manager_Java_Console_5.1_for_Linux.rpm

卸载 VCM

rpm -e VRTScscm

打开图形界面

export DISPLAY=10.16.200.84:0

/opt/VRTSvcs/bin/hagui &

默认账号　admin/password

4. 2 增加服务组

Add Service Group

添加Service Group Name, 选择节点, Service Group Type 用默认 FailOver,

Templates 选择OracleGroup ，OK.

4. 3 增加资源组

在Service Group 中 Add Resource

添加资源的时候注意选择enabled

# 添加 DiskGroup 资源

--archdg

ResourceName archdg

ResourceType DiskGroup

DiskGroup archdg

Enabled

--datadg

ResourceName datadg

ResourceType DiskGroup

DiskGroup datadg

Enabled

--redodg

ResourceName redodg

ResourceType DiskGroup

DiskGroup redodg

Enabled

删除 Volume 资源组，不用配置。

#Mount

--fs_archdg

ResourceName fs_archdg

ResourceType Mount

MountPoint /archdg

BlockDevie /dev/vx/dsk/archdg/lv_archdg

FSType vxfs

FsckOpt -y

Enabled

--fs_datadg

ResourceName fs_datadg

ResourceType Mount

MountPoint /datadg

BlockDevie /dev/vx/dsk/datadg/lv_datadg

FSType vxfs

FsckOpt -y

Enabled

--fs_redodg

ResourceName fs_redodg

ResourceType Mount

MountPoint /redodg

BlockDevie /dev/vx/dsk/redodg/lv_redodg

FSType vxfs

FsckOpt -y

Enabled

# NIC

--Ora_NIC

Device bond0

Enabled

#Netlsnr

--Ora_Netlsnr

Owner oracle

Home /opt/oracle/products/11.1.0/db_1

Enabled

注意 listener 不选 Critical 属性，可以保证监听本身故障时不会导致 HA 切换。

#VIP

--Ora_IP

Device bond0

Address 172.16.11.51

NetMask 255.255.255.0

Enabled

#Oracle

--Ora_Oracle

SID orasid

Owner oracle

Home /opt/oracle/products/11.1.0/db_1

Enabled

4.4 在 resource 中设置 link 关系

4.5 ha 配置文件

所有的 ha 配置，最后都保存到 /etc/VRTSvcs/conf/config/main.cf

HA 监控配置

5.1 监控 oracle instance

默认监控脚本 /opt/VRTSagents/ha/bin/Oracle/SqlTest.pl

我们使用 /opt/VRTSagents/ha/bin/Oracle/SimpleTest.pl

SID dbsid

Owner oracle

Home /opt/oracle/products/11.1.0/db_1

Pfile

StartupOpt STARTUP_FORCE

ShutDownOpt IMMEDIATE

EnvFile 设置环境变量，不需要

AutoEndBkup true

User monitor

Pword monitor 密码会自动加密

Table veritas_update

MonSctipt ./bin/Oracle/SimpleTest.pl

DetailMonitor 1

Encoding

MonitorOption 0 Oracle Instance 的监控选项，默认为0 。0 为进程检查，1 为健康检查。10g 之前必须配置为0 。

-- 配置detail monitoring for Oracle

sqlplus ‘/as sysdba’

drop user monitor CASCADE;

create user monitor

identified by monitor

default tablespace USERS

temporary tablespace TEMP

;

grant connect,resource,SELECT_CATALOG_ROLE to monitor;

revoke UNLIMITED TABLESPACE from monitor;

alter user monitor quota unlimited on USERS;

create table monitor.veritas_update( tstamp date );

insert into monitor.veritas_update (tstamp) values (sysdate);

commit;

在Ora_oracle 资源的配置项里面，要配置参数

User monitor

Pword monitor 密码会自动加密

Table veritas_update

MonSctipt ./bin/Oracle/SimpleTest.pl

DetailMonitor 1

在 Oracle 的配置项里面，配置 Oracle 实例监控间隔：

MonitorInterval 60

FaultOnMonitorTimeouts 2

MonitorTimeout 20

含义：

监控间隔60 秒，超时2 次fault ，超时时间20 秒，

1 次更新不成功开始报警，2 次连续更新不成功，就failover 。

也就是 update 80 秒不成功， fault ，开始切换。

/opt/VRTSagents/ha/bin/Oracle/SimpleTest.pl 这个脚本的动作 :

# SQL> connect <User>/<Pword>

# SQL> update <User>.<Table> set ( tstamp ) = SYSDATE;

# SQL> select TO_CHAR(tstamp, 'MON DD, YYYY HH:MI:SS AM')tstamp

# 2> from <User>.<Table>;

# SQL> exit

5.2 listener 监控

默认监控脚本 /opt/VRTSagents/ha/bin/Netlsnr/LsnrTest.pl

Owner oracle

Home /opt/oracle/products/11.1.0/db_1

TnsAdmin

Listener

EnvFile

MonScript ./bin/Netlsnr/LsnrTest.pl

LsnrPwd

Encoding

HA 切换

6.1 HA 切换日志

Veritas 所有的日志放在 /var/VRTSvcs/log/ 下面，

/var/VRTSvcs/log/engine_A.log HA 所有动作的详尽日志， 2 个节点同步生成。

/var/VRTSvcs/log/Oracle_A.log Oracle 资源的日志

Veritas HA 的日志条目按关键词分为 4 个级别： INFO/NOTICE/WARNING/ERROR ，

在 /var/VRTSvcs/log/engine_A.log 文件中监控 WARNING/ERROR 即可。

6.2 HA 切换测试

# 环境：

host1 和 host2 两个节点，资源，应用跑在 host1 。

有 eth1-4 ，共 4 个网口。 eth1 ， eth3 这 2 个独立网口用直连线做首选心跳， eth0 ， eth2 走业务，做 bond 后上联到 2 个网络交换机。

默认资源配置：

MonitorInterval 60s

OfflineMonitorInterval 300s

Oracle 资源参数配置 :

MonitorInterval 60 ，

FaultOnMonitorTimeouts 2 ，

MonitorTimeout 20

# 测试场景：

(1) host1 OS reboot

测试结果： 60 秒内发现问题，资源切换到 host2 。

(2) 网络测试

Ora_NIC 监控业务网卡bond0 ，Ora_IP 是VIP ，附在bond0:0

** 单独断掉心跳 eth1 ， eth3

ifdown eth1

ifup eth1

测试结果：对 ha 无影响， headbeat 链路会切换

** 单独断掉业务网卡

ifdown bond0

测试结果： down 掉 bond0 ，导致 Ora_NIC 资源 down 掉， vip 附在 bond0:0 上，业务 bond0 当掉，会导致 Ora_IP 当掉，监听 down ， oracle down ，资源切换到 host2

(3) 监听 down 掉

测试结果： 60 秒内发现问题，不做处理。

标记资源Faulted ，节点Partial Online 。

监听恢复 5 分钟后，会把资源标记为正常，节点状态 Online 。

(4) oracle shutdown abort

测试结果：资源正常切换到 host2

(5) db hang

测试结果： db 无法执行 update ， 60+20 秒内检测到，进行切换。

连续出现 2 次 update 超时事件就会切换。

常用命令

7.1 Veritas DMP

# 磁盘的多路径信息

vxdmpadm list dmpnode all |more

vxdmpadm list dmpnode dmpnodename=hds9500-alua0_0

#vxdmpadm -h list

Report detailed list of attributes associated with the specified entities such as DMP devices.

Usage:

vxdmpadm list dmpnode [all]

vxdmpadm list dmpnode [dmpnodename=<dmp-device-name>]

vxdmpadm list dmpnode [enclosure=<enclr-name>]

# 查看lun 信息

dmp 把磁盘初始化之后才能用vxdisk list 看到盘

#vxdisk list|more

DEVICE TYPE DISK GROUP STATUS

hds9500-alua0_0 auto:cdsdisk redovg01 redodg online

hds9500-alua0_1 auto:cdsdisk redovg02 redodg online

hds9500-alua0_2 auto:cdsdisk archvg01 archdg online

hds9500-alua0_3 auto:cdsdisk archvg02 archdg online

vxdisk list hds9500-alua0_0 可以显示这个 lun 的 path

7.2 Veritas VxVM

显示dg

#vxdg list

# vxdg list redodg

7.3 Veritas VCS

启动vcs

# hastart 在本地启动 ha

停止vcs

# hastop -all 停掉所有节点上的 vcs

# hastop -local 停止本节点上的 vcs ，也停掉本节点 running 的资源

# hastop -all -force 停止 vcs ，但保持所有资源可用

资源组

#hagrp -online myha -sys host1 把资源组 myha 在 host1 拉起来

#hagrp -offline myha -sys host1 资源组从 host1 离线

资源组切换

# hagrp -switch myha -to host2 资源从 host1 切换到 host 2

# tail -f /var/VRTSvcs/log/engine_A.log 查看切换日志

检查ha 状态

#hastatus ha 详细状态信息，包含每个资源

#hastatus -sum ha 基本情况

资源 fault 导致切换后，需要手工 clear 资源，才能使节点返回正常可切换状态。

#hares -clear Ora_Oracle -sys host1

#hares -clear Ora_IP -sys host1

资源组冻结

# hagrp -freeze myha 冻结 ha 后，维护资源不会导致 ha 切换。

# hagrp -unfreeze myha 解冻

探测资源

# hares -probe Ora_Oracle -sys host2 资源恢复正常后，需要探测一下， ha 状态才能正常。

SMTP 报警邮件配置

参考Linux5.1_vcs_admin.pdf P170

Tools --> Notifier Wizard --> Click Next

--> Service Group Configuration

resource name Notifier

nodes all

--> Configure Modes

选smtp

--> SMTP Server

SMTP Server mass-smtp.vip.xyi.cn.alidc.net

SMTP Recipients weijie.chenwj@alibaba-inc.com

Severity Level Information

--> Nic Resource Configuration

指定Notifier 资源依赖的NIC 资源

Resource Name Notifier_Nic

Device 建议用业务网卡，这个机器是bond0

目前我们的邮件网关只能发出发件人以 alibaba-inc.com 为结尾的邮件， Veritas SMTP 不能配置发件人，所以邮件没有办法发出来。

参考文档

Linux5.1PR1_sfrac_install.pdf

Linux5.1_vcs_admin.pdf

Linux5.1_vcs_oracle_agent.pdf

Linux5.1_vxvm_admin.pdf

Linux5.1_vxfs_admin.pdf

SF 文档中心

http://sfdoccentral.symantec.com/

SF 5.1 for Linux

http://sfdoccentral.symantec.com/Storage_Foundation_HA_51_Linux.html

SF 5.1 PR1 for Linux

http://sfdoccentral.symantec.com/Storage_Foundation_HA_51PR1_Linux.html

Storage Foundation Version 5.x for UNIX/Linux

http://www.symantec.com/business/support/overview.jsp?pid=15107

opendba

关注

0
点赞
踩
7

收藏

觉得还不错? 一键收藏
2
评论
Veritas HA 安装配置手册

1 安装 Veritas Storage Foundation 31.1 下载软件 31.2 准备工作 41.3 安装 SF for RAC 61.4 重新配置 SF for Oracle RAC 92 用 VEA 配置 Disk Group 102.1 安装 VEA 102.2 新建 DG 102.3 创建 Volumes 和 File System 112.4 在另外一个节点上配置 DG 163 安装 oracle 软件，创
复制链接

扫一扫