9.Oracle Clusterware Administration
Oracle Clusterware: Overview
• Portable cluster infrastructure that provides HA to RAC databases and/or other applications:
– Monitors applications’ health - 监控应用运行情况
– Restarts applications on failure - 重启失败的应用
– Can fail over applications on node failure - 将应用failover到其他节点
Oracle Clusterware Run-Time View
在Unix平台上,clusterware stack的启动是根据/etc/inittab文件中定义的respawn条目
A basic description of each process:
• Cluster Synchronization Services Daemon (OCSSD) - 通过network interconnect和voting disk实现节点间的健康监控,一旦发现异常,则重启问题节点以避免发生data corruption或split brain。由oracle用户运行,vendor或nonvendor clusterware情况下都使用。
• Process Monitor Daemon (OPROCD) - nonvendor情况下生成,以root运行,用于检测机器硬件或驱动层面的freeze。Linux和windows上使用kernel driver(如hangcheck-timer)来取代这个进程的作用。
• Cluster Ready Services Daemon (CRSD) - Oracle Clusterware高可用的核心,负责管理注册应用的start、stop、monitor和failover。CRS为每个实例起一个专用的RACGIMON进程负责监控各应用的运行情况,CRSD进行负责管理OCR中的各资源的配置及状态。以root用户运行,在进程失败后自动重启。此外,CRSD还可临时生成以下进程来执行某些操作:
- racgeut (Execute Under Timer), to kill actions that do not complete after a certain amount of time
- racgmdb (Manage Database), to start/stop/check instances
- racgchsn (Change Service Name), to add/delete/check service names for instances
- racgons, to add/remove ONS configuration to OCR
- racgvip, to start/stop/check instance virtual IP
• Event Management Daemon (EVMD) - 负责事件发生时event的发送,会生成一个evmlogger的永久进行,并在需要时生成racgevtf进程来执行callouts。以oracle用户运行,进程失败后自动重启。
Manually Control Oracle Clusterware Stack
# crsctl stop crs
# crsctl start crs
# crsctl disable crs
# crsctl enable crs
以root身份运行,用于一些计划停机场合。为使节点重启时不自动启动crs,必须首先disable掉crs。Disable命令并不停止当前运行的crs。
CRS Resources
CRS管理的应用被成为resource。它的profile及相关属性存放在OCR中。Profile中的属性包含Check interval、Action script、Dependencies、Failure policies、Privileges等。每个resource必须拥有action script,其中必须包含startstopcheck操作。
一个resource的生命周期一般包含:
创建或修改profile(crs_profile) -> 将resource注册到OCR中(crs_register)->启动(crs_start) ->查看状态(crs_stat)->重新分配到其他节点(crs_relocate)->停止(crs_stop)->取消注册(crs_unregister)
RAC Resources
可使用crs_stat –t来查看各resource的运行情况。在默认情况下,Oracle Clusterware管理的resouce包括databases、database and asm instances、VIP/ONS/GSD/Listener(统称为nodeapps)、services和service members等。
当某个resouce的state为unknown时,可使用crs_stop –f resoucename命令强行终止。
可使用crs_stat –p resourcename来查看某个resouce的各项属性。
Main Voting Disk Function
CSS使用private network和voting disk来管理节点的驱逐策略。各节点间通过彼此的心跳来检测各个节点是否处于正常状态。当节点间的网络连接出现问题时,各节点通过voting disk来交换状态信息,从而决定哪个节点需要被驱逐。被驱逐的节点从voting disk接收到驱逐信息后自行重启。
在数据库各实例间也有类似机制。有时在各节点间网络正常的情况下也需要驱逐某个实例(被称为Instance Membership Reconfiguration (IMR)),这一功能通过控制文件来实现。
Important CSS Parameters
• MISSCOUNT: - 网络心跳超时以及reconfiguration时的磁盘I/O超时
– Represents network heartbeat timeouts
– Determines disk I/O timeouts during reconfiguration
– Defaults to 30 seconds (60 for Linux)
– Defaults to 600 when using vendor (non-Oracle) clusterware
– Should not be changed
• DISKTIMEOUT: - 非reconfiguration时的磁盘I/O超时,可临时修改
– Represents disk I/O timeouts outside reconfiguration
– Defaults to 200 seconds
– Can be temporarily changed when experiencing very long I/O latencies to voting disks:
1. Shut down Oracle Clusterware on all nodes but one.
2. As root on available node, use: crsctl set css disktimeout M+1
3. Reboot available node.
4. Restart all other nodes.
Multiplexing Voting Disks
Voting disk如要作冗余,则至少要三份。
使用以下公式决定冗余度:v=f*2+1 - v代表voting disk的个数,f代表可以容忍的问题磁盘个数。
Change Voting Disk Configuration
# crsctl add css votedisk
# crsctl delete css votedisk
# crsctl add css votedisk -force
# crsctl delete css votedisk -force
OU教材上说可以动态增加或删除voting disk的副本,但在linux上实验没有成功。实验中只能通过加force选项完成。命令执行前必须停止clusterware。
Back Up and Recover Your Voting Disks
由于voting disk的路径配置在OCR中,难以直接修改,因此推荐使用link文件,使得当原始路径不可用时能较容易地恢复。
• Back up one voting disk by using the dd command. 使用dd命令进行备份,可联机备份。
– After Oracle Clusterware installation
– After node addition or deletion
– Can be done online
$ crsctl query css votedisk
$ dd if= of= bs=4k
OCR Architecture
OCR记录了集群的配置信息。在集群中的每个节点上,都有一份OCR的拷贝保留在内存中,同时有一个CRSD进程对OCR cache进行访问。但只有一个CRSD进程实际地读取和修改OCR文件。该进行负责刷新所在节点以及其他节点上的OCR cache。当本地可户端需要读取OCR的信息时,直接和本地的CRS进程进行交互并读取。当客户端需要修改OCR时,它们通过本地的CRS进程,与负责修改OCR的CRS进程进行通信并完成修改。
主要的客户端包括OUI、srvctl、EM、DBCA、DBUA、NetCA以及VIPCA等。
OCR记录并维护resource间的依赖关系以及状态信息。在安装cluserware时,OUI会提示是否需要对OCR作镜象。在UNIX平台,/etc/oracle/ocr.loc文件中记录了OCR以及其镜象文件的所在路径。
OCR Contents and Organization
OCR中的信息以key-value的形式按照树状结构存储。
The main branches composing the OCR structure:
• The SYSTEM keys - 存储Oracle Clusterware进程(CSSDCRSDEVMD等)相关的数据。
• The DATABASE keys - 存储RAC db(instancenodeappsservices等)相关数据。
• The CRS keys - 存储其他注册应用的相关数据。
Managing OCR Files and Locations: Overview
使用ocrconfig命令对OCR进行维护:
• 使用-export/import导出或导入逻辑备份
• Upgrade/downgrade OCR
• 使用-showbackup查看系统自动生成的备份,使用-backuploc修改自动备份生成的路径,使用-restore恢复备份
• 使用-replace ocr/ocrmirror来增加、修改或删除OCR及OCR镜象
• 在Oracle support的支持下,在一个或多个节点由于OCR corruption无法启动时,使用-overwrite命令修改OCR保护机制
• 使用-repair修改OCR或OCR镜象的相关参数
ocrcheck - 验证OCR及镜象的完整性
ocrdump - 将OCR内容dump成文本或XML文件
Automatic OCR Backups
OCR文件自动备份频率(无法修改):
每四小时一次(保留最近三份备份)
每天结束时(保留最近两份备份)
每周结束时(保留最近两份备份)
修改默认备份路径:# ocrconfig –backuploc /shared/bak
Back Up OCR Manually
• Daily backups of your automatic OCR backups to a different storage device:
– Use your favorite backup tool.
• Logical backups of your OCR before and after making significant changes:
# ocrconfig –export file name
• Make sure that you restore OCR backups that match your current system configuration.
Recover OCR Using Physical Backups
1. Locate a physical backup:
$ ocrconfig –showbackup
2. Review its contents:
# ocrdump –backupfile file_name
3. Stop Oracle Clusterware on all nodes:
# crsctl stop crs
4. Restore the physical OCR backup:
# ocrconfig –restore /cdata/jfv_clus/day.ocr
5. Restart Oracle Clusterware on all nodes:
# crsctl start crs
6. Check OCR integrity:
$ cluvfy comp ocr -n all
Recover OCR Using Logical Backups
1. Locate a logical backup created using an OCR export.
2. Stop Oracle Clusterware on all nodes.
3. Restore the logical OCR backup:
# ocrconfig –import /shared/export/ocrback.dmp
4. Restart Oracle Clusterware on all nodes.
5. Check OCR integrity:
$ cluvfy comp ocr -n all
Replace an OCR Mirror: Example
# ocrconfig –replace ocrmirror /oradata/OCR2
/etc/oracle/ocr.loc文件同步修改,经实验,文件必须存在才能增加OCR或镜象。可用于新增、修改、删除OCR及镜象文件。
Repair OCR Configuration: Example
当不同节点间对OCR配置信息存在不一致时(一般发生在某个节点未起的情况下进行了OCR/ocrmirror的配置修改),需要使用-repair命令来进行修复。Repair操作必须在crs停止的情况下进行。该命令仅修改OCR配置信息(ocr.loc文件),不对OCR 本身进行修复。
1. Stop Oracle Clusterware on Node2:
# crsctl stop crs
2. Add OCR mirror from Node1:
# ocrconfig –replace ocrmirror /OCRMirror
3. Repair OCR mirror location on Node2:
# ocrconfig –repair ocrmirror /OCRMirror
4. Start Oracle Clusterware on Node2:
# crsctl start crs
OCR Considerations
• If using raw devices to store OCR files, make sure they exist before add or replace operations.
• You must be the root user to be able to add, replace, or remove an OCR file while using ocrconfig.
• While adding or replacing an OCR file, its mirror needs to be online.
• If you remove a primary OCR file, the mirror OCR file becomes primary.
• Never remove the last remaining OCR file.
Change VIP Addresses
1. Determine the interface used to support your VIP:
$ ifconfig -a
2. Stop all resources depending on the VIP:
$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1
3. Verify that the VIP is no longer running:
$ ifconfig -a + $ crs_stat
4. Change IP in /etc/hosts and DNS.
5. Modify your VIP address using srvctl:
# srvctl modify nodeapps -n node1 –A 192.168.2.125/255.255.255.0/eth0
6. Start nodeapps and all resources depending on it:
# srvctl start nodeapps -n node1
7. Repeat from step 1 for the next node.
如修改了虚拟主机名。则相应的listener以及tnsnames也需要修改。
Change Public/Interconnect IP Subnet Configuration: Example
1. You get the current interfaces information by using the getif option.
$ /bin/oifcfg getif
eth0 139.2.156.0 global public
eth1 192.168.0.0 global cluster_interconnect
2. You delete the entry corresponding to public interface first by using the delif option, and
then enter the correct information by using the setif option.
$ oifcfg delif -global eth0
$ oifcfg setif –global eth0/139.2.166.0:public
3. You do the same for your private interconnect.
$ oifcfg delif –global eth1
$ oifcfg setif –global eth1/192.168.1.0:cluster_interconnect
4. You check that the new information is correct.
$ oifcfg getif
eth0 139.2.166.0 global public
eth1 192.168.1.0 global cluster_interconnect
Third-Party Application Protection: Overview
• High Availability framework: - 将应用纳入CRS以提供高可用性
– Command-line tools to register applications with CRS
– Calls control application agents to manage applications
– OCR used to describe CRS attributes for the applications
• High Availability C API:
– Modify directly CRS attributes in OCR
– Modify CRS attributes on the fly
• Application VIPs: - 应用虚拟IP,提供failover功能(VIP与应用同时failover)
– Used for applications accessed by network means
– NIC redundancy
– NIC failover
• OCFS: - 为应用提供共享存储
– Store application configuration files
– Share files between cluster nodes
Application VIP and RAC VIP Differences
• RAC VIP is mainly used in case of node down events: - resource无须failover,RAC VIP被接管后,并不再接受连接,而只是立刻发送失败信息以便于客户端更换地址重连。
– VIP is failed over to a surviving node.
– From there it returns NAK to clients forcing them to reconnect.
– There is no need to fail over resources associated to the VIP.
• Application VIP is mainly used in case of application down events:
– VIP is failed over to another node together with the application(s).
– From there, clients can still connect through the VIP.
– Although not recommended, one VIP can serve many applications.
Use CRS Framework: Overview
1. Create an application VIP, if necessary:
a) Create a profile: Network data + usrvip predefined script
b) Register the application VIP.
c) Set user permissions on the application VIP.
d) Start the application VIP by using crs_start.
2. Write an application action script that accepts three parameters:
• start: Script should start the application.
• check: Script should confirm that the application is up.
• stop: Script should stop the application.
3. Create an application profile:
• Action script location
• Check interval
• Failover policies
• Application VIP, if necessary
4. Set permissions on your application.
5. Register the profile with Oracle Clusterware.
6. Start your application by using crs_start.
Prevent Automatic Instance Restarts
将as值改为2可disable掉自动重启(1-自动重启,0-保持重启前的状态)
# crs_register ora.….inst -update -o as=2,ra=1,ut=7d
10.Diagnosing Oracle Clusterware and RAC Components
The One Golden Rule in RAC Debugging
• Always make sure that your nodes have exactly the same system time to:
– Facilitate log information analysis
– Ensure accurate results when reading GV$ views
– Avoid untimely instance evictions
• The best recommendation is to synchronize nodes using Network Time Protocol.
避免节点间时钟不同步引发的问题
Oracle Clusterware Main Log Files
The main directories used by Oracle Clusterware to store its log files:
• CRS日志在$ORA_CRS_HOME/log//crsd/下,每10MB一归档
• CSS 日志在$ORA_CRS_HOME/log//cssd/下,每20MB一归档
• EVM 日志在$ORA_CRS_HOME/log//evmd下
• 各个resource的日志,格式为imon_.log,每10MB一归档,在
$ORA_CRS_HOME/log//racg 及$ORACLE_HOME/log//racg下
RACG下各可执行命令的日志在该路径下与命令同名的子目录中。
• SRVM (srvctl) 和OCR (ocrdump, ocrconfig, ocrcheck) 等客户端工具的日志在
$ORA_CRS_HOME/log//client/及$ORACLE_HOME/log//client/下
• Oracle Clusterware的重要日志在$ORA_CRS_HOME/log/下,文件名为alert.log
Diagnostics Collection Script
• Script to collect all important log files:
– Must be executed as root
– Is located in $ORA_CRS_HOME/bin/
– Is called diagcollection.pl
• Generates the following files in the local directory:
– basData_.tar.gz
– crsData _. tar.gz
– ocrData _. tar.gz
– oraData _. tar.gz
Cluster Verify: Overview
cluvfy是一种提供各项集群验证功能的工具,在10.2版本中包含该工具。Cluvfy命令可分为两种:
• Stage commands - 在软件安装、数据库创建、配置项更改等各场景前后进行系统环境检查
• Component commands - 对单个组件的状态及完整性等进行验证
Cluster Verify Stages
• -post hwos: Postcheck for hardware and operating system
• -pre cfs: Precheck for CFS setup
• -post cfs: Postcheck for CFS setup
• -pre crsinst: Precheck for CRS installation
• -post crsinst: Postcheck for CRS installation
• -pre dbinst: Precheck for database installation
• -pre dbcfg: Precheck for database configuration
Cluster Verify Components
• nodereach: Checks reachability between nodes
• nodecon: Checks node connectivity
• cfs: Checks Oracle Cluster File System integrity
• ssa: Checks shared storage accessibility
• space: Checks space availability
• sys: Checks minimum system requirements
• clu: Checks cluster integrity
• clumgr: Checks cluster manager integrity
• ocr: Checks OCR integrity
• crs: Checks CRS integrity
• nodeapp: Checks node applications existence
• admprv: Checks administrative privileges
• peer: Compares properties with peers
[@more@]来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/8558093/viewspace-1017157/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/8558093/viewspace-1017157/