RAC: HAS CRS CLUSTER

1.

crsctl stop has=shutdown abort ，这样太暴力啦。

正确的操作,大家需要注意啦。

1. srvctl stop database -d xxxx

2. crsctl stop has

2. 看日志：

3. <roidb1:orcl1:/u01/app/oracle/diag/rdbms/orcl/orcl1/trace>$tail -f al*.log

5. Thu May 24 20:37:36 2018

6. **Shutting down instance (abort)**

7. License high water mark = 5

2.

1 通过命令查看cluster/has/crs管理的内容

[root@11rac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@11rac1 ~]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@11rac1 ~]# crsctl check cluster

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

这里可以看到crs显示的内容=has+cluster显示的内容

2 分别执行3条命令查看结果显示

2.1 crsctl stop cluster

root@11rac1 ~]# crsctl stop cluster

CRS-2673: Attempting to stop ‘ora.crsd’ on ’11rac1′

[root@11rac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

[root@11rac1 ~]# crsctl check has

CRS-4638: Oracle High Availability Services is online

[root@11rac1 ~]# crsctl check cluster

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

这里可以看到stop cluster停了Clusterware stack，其实也就是官方文档中指的Cluster Ready Services Stack。

2.2 crsctl stop has

继续上面的操作

[root@11rac1 ~]# crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ’11rac1′

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ’11rac1′

CRS-2673: Attempting to stop ‘ora.crf’ on ’11rac1′

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ’11rac1′ has completed

CRS-4133: Oracle High Availability Services has been stopped.

stop has停的就是官方文档中的Oracle High Availability Services Stack，但是Oracle High Availability Services Stack属于Cluster Ready Services Stack依赖的底层，所以在停Oracle High Availability Services Stack会自动停Cluster Ready Services Stack，如下：

[root@11rac1 ~]# crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ’11rac1′

CRS-2673: Attempting to stop ‘ora.crsd’ on ’11rac1′

CRS-2677: Stop of ‘ora.gpnpd’ on ’11rac1′ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ’11rac1′ has completed

CRS-4133: Oracle High Availability Services has been stopped.

这里可以看到在停has的时候，自动把上层服务也停了。

2.3 crsctl stop crs 本机的

这里的CRS其实指的是整个ORACLE集群，也不是 Cluster Ready Services Stack的简写。 那是cluster

[root@11rac1 ~]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ’11rac1′

CRS-4133: Oracle High Availability Services has been stopped.

通过上面的简单测试，可以得出两点：

1，crsctl stop cluster=crsctl stop crs+crsctl stop has，前提是先停cluster后，再停has

错了吧crsctl stop cluster+crsctl stop has--安全停库=crsctl stop crs=crsctl stop has abort库

2，crsctl stop crs=crsctl stop has，如果之前没有Oracle手动停cluster，那么crsctl stop crs与crsctl stop has的效果一样

另外crsctl stop crs与crsctl stop has都只能操作当前节点，crsctl stop cluster可以一次操作集群中多个节点，前提是HAS服务正常运行。

这里可以看到crs显示的内容=has+cluster显示的内容

Oracle 10gRAC 启动与关闭

http://www.cndba.cn/Dave/article/1016

 

Oracle RAC 常用维护工具和命令

http://www.cndba.cn/Dave/article/1015
 

一．启动和停止集群
   在Oracle 11gR2 下的RAC，架构发生了变化。CRS的信息也是放在ASM 实例里的，所以要关asm,必须关闭crs, 如果还使用了acfs的话，一关crs那么acfs里的信息也不能访问了，所以一般不重启机器，不轻易关crs, 其他的service可以根据自己的需要去stop/start。

 

注意：

11g RAC 开启资源相对比较慢(即使命令后面显示的资源都start succeeded,通过crs_stat -t查看都不一定online), 需要耐心并查看log。

 

    在Oracle 11gR2 里引入了Oracle Restart的特性，其用来管理Cluster Resource。关于Oracle Restart，参考我的Blog：

Oracle 11g 新特性 -- Oracle Restart 说明

http://www.cndba.cn/Dave/article/1288
 

1.1 使用crsctl stop has/crsctl stopcrs
用root用户，在Oracle11gR2中停止和启动集群的命令如下：

#crsctl stop has [-f]

#crsctl start has

 

注意：

    对于crsctl stop has 只有一个可选的参数就是-f，该命令只能停执行该命令服务器上的HAS.而不能停所有节点上的。所以要把RAC 全部停掉，需要在所有节点执行该命令。

 

下面的2个命令：使用crs 和 使用has 效果是完全一样的：

#crsctl stop crs [-f]

#crsctl start crs

 

 

示例：

--停止HAS

[root@rac1 bin]# ./crsctl stophas

CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on'rac1'

CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac1' hascompleted

CRS-4133:Oracle High Availability Services has been stopped.

[root@rac1bin]#

 

注意：

       我这里测试的是Oracle11gR2的环境，我们在节点1上执行该命令，只把节点1上的进程停了，而把相关的资源转移到我们的节点2上了，因此也证实了我们上面的说的，该命令只对当前服务器有效。

 

--启动HAS：

[root@rac1bin]# ./crsctl start has

CRS-4123:Oracle High Availability Services has been started.

[root@rac1bin]#

 

从上面看只是启动了HAS。实际上后面会把Oracle Restart 管理的资源都会启动。这个可以使用crs_stat命令来进程验证，不过Oracle 11g的进程启动过程比较慢，需要耐心等待。

 

[root@rac1u01]# sh crs_stat.sh

Name                           Target     State     Host     

---------------------------------------- ---------  -------  

ora.DATA.dg                    ONLINE     ONLINE    rac1     

 

 

1.2 使用crsctl stop cluster [-all]…
该命令的语法如下：

crsctl stop cluster [[-all]|[-n<server>[...]]] [-f]

crsctl start cluster [[-all]|[-n<server>[...]]]

 

与1.1 节最大的不同，该参数支持的选项更多，可以同时操控所有的节点。如果不指定参数，那么只对当前节点有效。

 

如：

[root@rac1 ~]# ./crsctl start cluster -n rac1 rac2

 

--停止当前节点集群：

[root@rac1 bin]# ./crsctl stop cluster

CRS-2673: Attempting to stop 'ora.crsd' on'rac1'

 

这里只停了节点1，相关进程全飘到节点2了，如下：

[root@rac2 u01]# sh crs_stat.sh

Name                           Target     State     Host     

------------------------------ -------------------  -------  

ora.DATA.dg                    ONLINE     ONLINE    rac2     

 

--启用当前节点：

[root@rac1 bin]#./crsctl start cluster

CRS-2672: Attempting to start'ora.cssdmonitor' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on'rac1' succeeded

[root@rac1 bin]#

 

1.3 检查集群的状态
 

--简单的输出:

[root@rac1 bin]# ./crsctl check cluster

[root@rac1 bin]# ./crsctl check crs

 

--详细的信息:

# crs_stat -t

 

 

使用crsctl check crs命令，如下：

[root@rac1 bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Servicesis online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Servicesis online

CRS-4533: Event Manager is online

 

[root@rac1 bin]# ./crsctl check cluster

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Servicesis online

CRS-4533: Event Manager is online

 

二．停止和启动Resource
    在第一节中，看到直接停止集群，相关的Resource 也会被停止。 但一些实际情况下，我们操作更多的是对某些资源的操作。具体就是使用SRVCTL 命令

 

1.1. 官网说明
 

查看命令帮助：

Srvclt –h

 --或者命令不敲全：

[grid@rac1 ~]$ srvctl database

Usage: srvctl <command><object> [<options>]

    

1.2 示例
1.2.1 检查集群状态
[grid@rac1 ~]$ crsctl check cluster

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Servicesis online

CRS-4533: Event Manager is online

 

1.2.2 所有Oracle实例（数据库状态）
[grid@rac1 ~]$ srvctl status database -d sdd      

Instance sdd1 is running on node rac1

Instance sdd2 is running on node rac2

 

1.2.3 检查单个实例状态
[grid@rac1 ~]$ srvctl status instance -d sdd -i sdd1

Instance sdd1 is running on node rac1

[grid@rac1 ~]$

 

1.2.4节点应用程序状态
[grid@rac1 ~]$ srvctl status nodeapps

VIP rac1-vip is enabled

VIP rac1-vip is running on node: rac1

VIP rac2-vip is enabled

VIP rac2-vip is running on node: rac2

Network is enabled

Network is running on node: rac1

Network is running on node: rac2

GSD is disabled

GSD is not running on node: rac1

GSD is not running on node: rac2

ONS is enabled

ONS daemon is running on node: rac1

ONS daemon is running on node: rac2

 

 

1.2.5 列出所有的配置数据库
[grid@rac1 ~]$ srvctl config database

sdd

 

1.2.6 数据库配置
[grid@rac1 ~]$ srvctl config database -d sdd -a

Database unique name: sdd

Database name: sdd

Oracle home: /u01/app/oracle/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/sdd/spfilesdd.ora

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: sdd

Database instances: sdd1,sdd2

Disk Groups: DATA,FRA

Mount point paths:

Services:

Type: RAC

Database is enabled

Database is administrator managed

 

1.2.7 ASM状态以及ASM配置
[grid@rac1 ~]$ srvctl status asm

ASM is running on rac2,rac1

 

[grid@rac1 ~]$ srvctl status asm -a

ASM is running on rac2,rac1

ASM is enabled.

 

1.2.8 TNS监听器状态以及配置
[grid@rac1 ~]$ srvctl status listener

Listener LISTENER is enabled

Listener LISTENER is running on node(s):rac2,rac1

 

[grid@rac1 ~]$ srvctl config listener -a

Name: LISTENER

Network: 1, Owner: grid

Home: <CRS home>

 /u01/app/grid/11.2.0 on node(s) rac2,rac1

End points: TCP:1521

 

 

1.2.8 SCAN状态以及配置
[grid@rac1 ~]$ srvctl status scan

SCAN VIP scan1 is enabled

SCAN VIP scan1 is running on node rac1

 

[grid@rac1 ~]$ srvctl config scan

SCAN name: rac-scan, Network:1/192.168.16.0/255.255.255.0/eth0

SCAN VIP name: scan1, IP:/rac-scan/192.168.16.207

 

1.2.9 VIP各个节点的状态以及配置
[grid@rac1 ~]$ srvctl status vip -n rac1

VIP rac1-vip is enabled

VIP rac1-vip is running on node: rac1

 

[grid@rac1 ~]$ srvctl status vip -n rac2

VIP rac2-vip is enabled

VIP rac2-vip is running on node: rac2

 

[grid@rac1 ~]$ srvctl config vip -n rac1

VIP exists: /rac1-vip/192.168.16.201/192.168.16.0/255.255.255.0/eth0,hosting node rac1

 

[grid@rac1 ~]$ srvctl config vip -n rac2

VIP exists:/rac2-vip/192.168.16.203/192.168.16.0/255.255.255.0/eth0, hosting node rac2

[grid@rac1 ~]$

 

1.2.10 节点应用程序配置（VIP、GSD、ONS、监听器）
[grid@rac1 ~]$ srvctl config nodeapps -a -g-s -l

Warning:-l option has been deprecated andwill be ignored.

Network exists:1/192.168.16.0/255.255.255.0/eth0, type static

VIP exists:/rac1-vip/192.168.16.201/192.168.16.0/255.255.255.0/eth0, hosting node rac1

VIP exists:/rac2-vip/192.168.16.203/192.168.16.0/255.255.255.0/eth0, hosting node rac2

GSD exists

ONS exists: Local port 6100, remote port6200, EM port 2016

Name: LISTENER

Network: 1, Owner: grid

Home: <CRS home>

 /u01/app/grid/11.2.0 on node(s) rac2,rac1

End points: TCP:1521

 

1.3 语法简汇
1.3.1 数据库与实例
srvctl config database # 列出安装的数据库
srvctl config database -d rac -a # 检查数据库相关的信息
srvctlstatus database -d RAC # 检查数据库的状态
srvctl status instance -d RAC -i rac1 # 检查指定实例的状态

srvctl config asm -a # ASM配置
srvctl status asm # ASM的状态

srvctl start database -d rac # 启动数据库
srvctl stop database -d rac # 关闭数据库

srvctl start instance -d rac -i rac1 # 启动指定的实例
srvctl stop instance -d rac -i rac2 # 关闭指定实例

 

1.3.2 网络相关的命令
 

srvctl status listener # 检查TNS listener的状态
srvctl config scan # SCAN的配置
srvctl status scan # SCAN listener状态, 包含当前运行节点的信息

# 检查VIP的配置及状态
srvctl status vip -n rac1
srvctl config vip -n rac1

 

1.3.3 查看各资源状态(nodeapps节点应用程序，ASM实例，数据库等)
 

[root@db02 u01]# crs_stat -t        

Name           Type           Target   State     Host       

------------------------------------------------------------

ora.DATA.dg    ora....up.type ONLINE    ONLINE   db02 

ora.FRA.dg     ora....up.type ONLINE    ONLINE   db02 

ora....ER.lsnr ora....er.type ONLINE    ONLINE   db02 

ora....N1.lsnr ora....er.type ONLINE    ONLINE   db02 

ora....N2.lsnr ora....er.type ONLINE    ONLINE   db02 

ora....N3.lsnr ora....er.type ONLINE    ONLINE   db02 

ora.OCR.dg     ora....up.type ONLINE    ONLINE   db02 

ora.asm       ora.asm.type   ONLINE   ONLINE    db02 

ora....-01.vip ora....t1.type ONLINE    ONLINE   db02 

ora....SM2.asm application    ONLINE   ONLINE    db02 

ora....02.lsnr application    ONLINE   ONLINE    db02 

ora....-02.gsd application    OFFLINE  OFFLINE              

ora....-02.ons application    ONLINE   ONLINE    db02 

ora....-02.vip ora....t1.type ONLINE    ONLINE   db02 

ora.cvu        ora.cvu.type   ONLINE   ONLINE    db02 

ora.gsd        ora.gsd.type   OFFLINE  OFFLINE              

ora....network ora....rk.type ONLINE    ONLINE   db02 

ora.oc4j       ora.oc4j.type  ONLINE   ONLINE    db02 

ora.ons        ora.ons.type   ONLINE   ONLINE    db02 

ora.orcl.db    ora....se.type ONLINE    ONLINE   db02 

ora....taf.svc ora....ce.type ONLINE    ONLINE   db02 

ora.scan1.vip  ora....ip.type ONLINE    ONLINE   db02 

ora.scan2.vip  ora....ip.type ONLINE    ONLINE   db02 

ora.scan3.vip  ora....ip.type ONLINE    ONLINE   db02 

 

 在11g R2中，默认 oc4j和gsd资源是 disable的；oc4j 是用于WLM 的一个资源, WLM在 11.2.0.2 才可用；gsd是 CRS 用于跟 9i RAC 进行通信的一个模块,是为了向后兼容才保留的，不影响性能；建议不要刪除, 也不要尝试开启他们, 忽略即可。

 

ora.gsd  is OFFLINE by default ifthere is no 9i database in the cluster.

ora.oc4j is OFFLINE in 11.2.0.1 as DatabaseWorkload Management(DBWLM) is unavailable.  these can be ignored in11gR2 RAC.

 

状态检查也可以使用如下命令：

crsctl stat resource –t

或者

crsctl stat resource

 

更多内容参考：

Oracle 11gR2RAC 进程说明

http://blog.csdn.net/tianlesoftware/article/details/6009962

 

1.3.4 综合
1、通过SRVCTL命令来start/stop/check所有的实例:

 srvctl start|stop|status database -d<db_name>

 

2、start/stop指定的实例：

srvctl start|stop|statusinstance -d <db_name> -i <instance_name>

 

3、列出当前RAC下所有的

 srvctl config database -d <db_name>

 

4、start/stop/check 所有的nodeapps，比如：VIP, GSD, listener, ONS：

 srvctl start|stop|status nodeapps -n<node_name>

 

5、如果你使用ASM，srvctl也可以start/stop ASM实例：

  srvctl start|stop asm -n <node_name>[-i <asm_inst_name>] [-o<oracle_home>]

 

6、可以获取所有的环境信息：

 srvctl getenv database -d <db_name> [-i<instance_name>]

 

7、设置全局环境和变量：

srvctl setenv database -d<db_name> -t LANG=en

 

8、从OCR中删除已有的数据库：

  srvctl remove database -d <db_name>

 

9、向OCR中添加一个数据库的实例：

 srvctl add instance -d <db_name> -i<instance_name> -n <node1>

 srvctl add instance -d <db_name> -i<instance_name> -n <node2>

 

 

其他的相关操作参考：

Oracle 10gRAC OCR 和 VotingDisk 的备份与恢复

http://www.cndba.cn/Dave/article/1013
 

Oracle RACASM disk header 备份 恢复 与 重建 示例说明

http://www.cndba.cn/Dave/article/773
 

Oracle 使用BBED 查看 ASM Disk Header 内容

http://www.cndba.cn/Dave/article/774
————————————————
 

从11R2起oracle引入init.ohasd，将其配置在/etc/inittab中，用以启动和管理clusterware相关资源

Pre 11R2

/etc/inittab

h1:2:respawn:/etc/init.evmd run >/dev/null 2>&1

h2:2:respawn:/etc/init.cssd fatal >/dev/null 2>&1

h3:2:respawn:/etc/init.crsd run >/dev/null 2>&1

$  who -r

   .        run-level 2 Jan 22 05:49       2    0    S

11R2

/etc/inittab

h1:2:respawn:/etc/init.ohasd run >/dev/null 2>&1

oracle> who -r

   .        run-level 2 Jul 03 07:52       2    0    S

 

clusterware资源大致可分为4级，其相应启动顺序如下：

先由INIT启动init.ohasd，后者再启动相应的4个进程，然后按顺序依次启动

Level 1: OHASD Spawns:

• cssdagent - Agent responsible for spawning CSSD.
• orarootagent - Agent responsible for managing all root owned ohasd resources.
• oraagent - Agent responsible for managing all oracle owned ohasd resources.
• cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).

Level 2: OHASD rootagent spawns:

• CRSD - Primary daemon responsible for managing cluster resources.
• CTSSD - Cluster Time Synchronization Services Daemon
• Diskmon
• ACFS (ASM Cluster File System) Drivers 

Level 2: OHASD oraagent spawns:

• MDNSD - Used for DNS lookup
• GIPCD - Used for inter-process and inter-node communication
• GPNPD - Grid Plug & Play Profile Daemon
• EVMD - Event Monitor Daemon
• ASM - Resource for monitoring ASM instances

Level 3: CRSD spawns:

• orarootagent - Agent responsible for managing all root owned crsd resources.
• oraagent - Agent responsible for managing all oracle owned crsd resources.

Level 4: CRSD rootagent spawns:

• Network resource - To monitor the public network
• SCAN VIP(s) - Single Client Access Name Virtual IPs
• Node VIPs - One per node
• ACFS Registery - For mounting ASM Cluster File System
• GNS VIP (optional) - VIP for GNS

Level 4: CRSD oraagent spawns:

• ASM Resouce - ASM Instance(s) resource
• Diskgroup - Used for managing/monitoring ASM diskgroups.  
• DB Resource - Used for monitoring and managing the DB and instances
• SCAN Listener - Listener for single client access name, listening on SCAN VIP
• Listener - Node listener listening on the Node VIP
• Services - Used for monitoring and managing services
• ONS - Oracle Notification Service
• eONS - Enhanced Oracle Notification Service
• GSD - For 9i backward compatibility
• GNS (optional) - Grid Naming Service - Performs name resolution