虚拟机下11g r2 RAC安装root时报：Timed out waiting for the CRS stack to start. .

最新推荐文章于 2022-11-21 14:17:43 发布

chengxuefei_bj

最新推荐文章于 2022-11-21 14:17:43 发布

阅读量977

点赞数

分类专栏： oracle故障处理

oracle故障处理专栏收录该内容

29 篇文章 0 订阅

订阅专栏

虚拟机Linux环境下，安装11G r2的RAC，使用的是grid用户，共两个节点，第二个节点 root时遇到Timed out waiting for the CRS stack to start.
全部完全卸载、磁盘dd操作后重新安装多次，问题依旧。如下：

[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /apps/oracle/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-06-04 03:04:12: Parsing the host name
2012-06-04 03:04:12: Checking for super user privileges
2012-06-04 03:04:12: User has super user privileges
Using configuration parameter file: /apps/oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-4-8

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node dbserver1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dbserver2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'dbserver2' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbserver2'
CRS-2676: Start of 'ora.mdnsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'dbserver2'
CRS-2676: Start of 'ora.gipcd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbserver2'
CRS-2676: Start of 'ora.gpnpd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbserver2'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbserver2'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbserver2'
CRS-2676: Start of 'ora.diskmon' on 'dbserver2' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'dbserver2'
CRS-2676: Start of 'ora.ctssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbserver2'
CRS-2676: Start of 'ora.asm' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'dbserver2'
CRS-2676: Start of 'ora.crsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'dbserver2'
CRS-2676: Start of 'ora.evmd' on 'dbserver2' succeeded
Timed out waiting for the CRS stack to start.

重新执行root还是不行，如下：

[root@dbserver2 ~]# cd /apps/oracle/11.2.0/grid/crs/install/
[root@dbserver2 install]# /apps/oracle/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force
……
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node

[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
……
CRS-2676: Start of 'ora.evmd' on 'dbserver2' succeeded
Timed out waiting for the CRS stack to start.

查看日志：

[root@dbserver2 dbserver2]# pwd
/apps/oracle/11.2.0/grid/log/dbserver2

[root@dbserver2 dbserver2]# tail -f /apps/oracle/11.2.0/grid/log/dbserver2/alertdbserver2.log

2012-06-01 20:03:44.976
[crsd(11217)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:45.315
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:47.083
[crsd(11235)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:47.631
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:49.501
[crsd(11250)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:49.986
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:49.988
[ohasd(10460)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

继续查看： /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.

[root@dbserver2 crsd]# tail -200 /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log

gipcretSuccess (0)
2012-06-01 20:16:12.859: [ OCRASM][3086657216]proprasmo: Error in open/create file in dg [crs]
[ OCRASM][3086657216]SLOS : SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CRS.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CRS.255.4294967295
ORA-15001: diskgroup "CRS"
2012-06-01 20:03:49.499: [ OCRASM][3086726848]proprasmo: kgfoCheckMount returned [6]
2012-06-01 20:03:49.499: [ OCRASM][3086726848]proprasmo: The ASM disk group crs is not found or not mounted
2012-06-01 20:03:49.501: [ OCRRAW][3086726848]proprioo: Failed to open [+crs]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2012-06-01 20:03:49.502: [ OCRRAW][3086726848]proprioo: No OCR/OLR devices are usable
2012-06-01 20:03:49.502: [ OCRASM][3086726848]proprasmcl: asmhandle is NULL
2012-06-01 20:03:49.502: [ OCRRAW][3086726848]proprinit: Could not open raw device
2012-06-01 20:03:49.502: [ OCRASM][3086726848]proprasmcl: asmhandle is NULL
2012-06-01 20:03:49.505: [ OCRAPI][3086726848]a_init:16!: Backend init unsuccessful : [26]
2012-06-01 20:03:49.508: [ CRSOCR][3086726848] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error[SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CRS.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CRS.255.4294967295
ORA-15001: diskgroup "CRS"
] [8]
2012-06-01 20:03:49.508: [ CRSD][3086726848][PANIC] CRSD exiting: Could not init OCR, code: 26
2012-06-01 20:03:49.509: [ CRSD][3086726848] Done.

检查如下：

[grid@dbserver2 ~]$ asmcmd
ASMCMD> ls
没有任何内容。

检查磁盘：
[root@dbserver2 dbserver2]# fdisk -l
……
Disk /dev/sdh: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdh1 1 261 2096451 83 Linux

Disk /dev/sdi: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdi1 1 261 2096451 83 Linux

Disk /dev/sdj: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdj1 1 261 2096451 83 Linux

[root@dbserver2 dbserver2]# /etc/init.d/oracleasm listdisks

[root@dbserver2 dbserver2]# /etc/init.d/oracleasm listdisks
CRS1
CRS2
CRS3
[root@dbserver2 dbserver2]#

[root@dbserver2 dbserver2]# cd /dev/oracleasm/disks/
[root@dbserver2 disks]# ll
total 0
brw-rw---- 1 grid asmadmin 8, 113 Jun 2 10:44 CRS1
brw-rw---- 1 grid asmadmin 8, 129 Jun 2 10:44 CRS2
brw-rw---- 1 grid asmadmin 8, 145 Jun 2 10:44 CRS3
[root@dbserver2 disks]#

下面的检查：
[root@dbserver2 oracle]# cd /apps/oracle/11.2.0/grid/bin/
[root@dbserver2 bin]# ./crsctl query css votedisk
## STATE    File Universal Id                File Name Disk group
-- -----    -----------------                --------- ---------
1. ONLINE   6e49b9f535744ffcbf42cf5c1315be5c (/dev/oracleasm/disks/CRS1) [CRS]
2. ONLINE   8319759558834fc8bf89f783a8177b5b (/dev/oracleasm/disks/CRS2) [CRS]
3. ONLINE   94e522f261334f18bfc59d576f112016 (/dev/oracleasm/disks/CRS3) [CRS]
Located 3 voting disk(s).
[root@dbserver2 bin]#

重新删除后再安装一次，问题依旧。

dd if=/dev/zero of=/dev/sdh1 bs=1024 count=300
rm -rf /apps/*
chown -R grid:oinstall /apps/
rm -rf /tmp/*
rm -rf /etc/ora*

又未成功

继续检查：
[root@dbserver2 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=6, opn=kgfo, dep=0, loc=kgfoCkMt03
diskgroup CRS not mounted ()
] [6]
[root@dbserver2 bin]#
在另一个节点：
[root@dbserver1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2280
         Available space (kbytes) :     259840
         ID                       : 1856240294
         Device/File Name         :       +CRS
                                    Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded
SQL> select name from V$asm_diskgroup
2 ;

NAME
------------------------------
CRS

SQL> alter diskgroup CRS mount;
alter diskgroup CRS mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "CRS" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "CRS"
SQL> select group_number,disk_number,mode_status,name from V$asm_disk;

GROUP_NUMBER DISK_NUMBER MODE_ST NAME
------------ ----------- ------- ------------------------------
           0           0 ONLINE
           0           2 ONLINE
           0           1 ONLINE

重新安装ASMLIB

[root@dbserver1 bin]# rpm -qa|grep oracleasm
oracleasm-support-2.1.7-1.el4
oracleasmlib-2.0.4-1.el4
oracleasm-2.6.9-89.EL-2.0.5-1.el4
oracleasm-2.6.9-89.ELsmp-2.0.5-1.el4

[root@dbserver1 bin]# rpm -e oracleasm-2.6.9-89.EL-2.0.5-1.el4
[root@dbserver1 bin]# rpm -e oracleasmlib-2.0.4-1.el4
[root@dbserver1 bin]# rpm -e oracleasm-2.6.9-89.ELsmp-2.0.5-1.el4
[root@dbserver1 bin]# rpm -e oracleasm-support-2.1.7-1.el4
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
[root@dbserver1 soft]# rpm -ivh oracleasm*

SQL> show parameter asm

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string
asm_diskstring                       string
asm_power_limit                      integer     1
asm_preferred_read_failure_groups    string

SQL> select group_number,disk_number,mount_status,header_status,path from V$asm_disk;

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU PATH
------------ ----------- ------- ------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
           0           0 CLOSED MEMBER       /dev/oracleasm/disks/CRS1
           0           2 CLOSED MEMBER       /dev/oracleasm/disks/CRS3
           0           1 CLOSED MEMBER       /dev/oracleasm/disks/CRS2

SQL>

最后偿试的解决办法

在虚拟机的配置中加入了：
scsi1:0.deviceType = "disk"
scsi1:1.deviceType = "disk"
scsi1:2.deviceType = "disk"
scsi1:3.deviceType = "disk"
scsi1:4.deviceType = "disk"
scsi1:5.deviceType = "disk"
scsi1:6.deviceType = "disk"
scsi1:7.deviceType = "disk"
scsi1:8.deviceType = "disk"
scsi1:9.deviceType = "disk"
scsi1:10.deviceType = "disk"
scsi1:11.deviceType = "disk"
再次重试

dd if=/dev/zero of=/dev/sdh1 bs=1024 count=300
dd if=/dev/zero of=/dev/sdi1 bs=1024 count=300
dd if=/dev/zero of=/dev/sdj1 bs=1024 count=300
rm -rf /apps/*
chown -R grid:oinstall /apps/
rm -rf /tmp/*
rm -rf /etc/ora*
rm -f /usr/local/bin/dbhome
rm -f /usr/local/bin/oraenv
rm -f /usr/local/bin/coraenv
chown -R grid:oinstall /soft/

重新安装，居然成功，如下所示：

oracleasm scandisks
oracleasm listdisks
[root@dbserver1 ~]# oracleasm createdisk vol1 /dev/sdh1
[root@dbserver1 ~]# oracleasm createdisk vol2 /dev/sdi1
[root@dbserver1 ~]# oracleasm createdisk vol3 /dev/sdj1
[root@dbserver1 ~]# oracleasm listdisks
……
[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /apps/oracle/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-06-05 15:52:57: Parsing the host name
2012-06-05 15:52:57: Checking for super user privileges
2012-06-05 15:52:57: User has super user privileges
Using configuration parameter file: /apps/oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-4-8

Checking swap space: must be greater than 500 MB. Actual 3602 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /apps/oraInventory
'UpdateNodeList' was successful.

看来问题的原因还就是虚拟机的设置问题。不过此次安装两个节点都有另外的问题，
/apps/oracle/11.2.0/grid/bin/srvctl start nodeapps -n dbserver1 ... failed
Configure Oracle Grid Infrastructure for a Cluster ... failed
这个问题再通过另的办法解决。