主机启动后发现少了ocrsd.bin进程,检查grid alert日志,发现下面信息。 ASM OCR DG 无法启动。
- GMON updating disk modes for group 2 at 10 for pid 27, osid 7121
- ERROR: no read quorum in group: required 2, found 0 disks
- Fri Jan 15 10:33:21 2016
- NOTE: cache dismounting (not clean) group 2/0x676844DF (OCRDG)
- NOTE: messaging CKPT to quiesce pins Unix process pid: 7124, image: oracle@cntl202 (B001)
- Fri Jan 15 10:33:21 2016
- NOTE: halting all I/Os to diskgroup 2 (OCRDG) >>>>>>>>>>>>>>>OCRDG有IO故障。
- Fri Jan 15 10:33:21 2016
- NOTE: LGWR doing non-clean dismount of group 2 (OCRDG)
- NOTE: LGWR sync ABA=6.60 last written ABA 6.60
- WARNING: Offline for disk OCRDG_0000 in mode 0x7f failed.
- WARNING: Offline for disk OCRDG_0001 in mode 0x7f failed.
- WARNING: Offline for disk OCRDG_0002 in mode 0x7f failed.
- Fri Jan 15 10:33:21 2016
- NOTE: No asm libraries found in the system
ASM使用ASMLIB,"oracleasm listdisks" 发现少了一个ASM DISKS,
- --缺少了一个ASMDISK1
- [/root]# oracleasm listdisks
- ASMDISK2
- ASMDISK3
- DATADISK1
[/root]# oracleasm querydisk -v -p -d ASMDISK1
Disk "ASMDISK1" does not exist or is not instantiated
Disk "ASMDISK1" does not exist or is not instantiated
- --无法创建,提示已存在。
- [/root]#oracleasm createdisk ASMDISK1 /dev/sdd1
- Device "/dev/sdd1" is already labeled for ASM disk ""
-
- [/root]#oracleasm deletedisk ASMDISK1
- Disk "ASMDISK1" does not exist or is not instantiated
--清空磁盘头。
- [/root]#dd if=/dev/zero of=/dev/sdd1 bs=1024 count=100
- 100+0 records in
- 100+0 records out
- 102400 bytes (102 kB) copied, 0.0812539 s, 1.3 MB/s
-
- [/root]#oracleasm createdisk ASMDISK1 /dev/sdd1
- Writing disk header: done
- Instantiating disk: done
1. [ALL NODES] 停止CRS,crsd.bin进程不存在,无法正常停,需要 "-f" 强制停止
- [/root]#crsctl stop crs -f
- CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cntl069'
- CRS-2673: Attempting to stop 'ora.mdnsd' on 'cntl069'
- CRS-2673: Attempting to stop 'ora.ctssd' on 'cntl069'
- CRS-2673: Attempting to stop 'ora.asm' on 'cntl069'
- CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cntl069'
- CRS-2677: Stop of 'ora.mdnsd' on 'cntl069' succeeded
- CRS-2677: Stop of 'ora.asm' on 'cntl069' succeeded
- CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cntl069'
- CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cntl069' succeeded
- CRS-2677: Stop of 'ora.drivers.acfs' on 'cntl069' succeeded
- CRS-2677: Stop of 'ora.ctssd' on 'cntl069' succeeded
- CRS-2673: Attempting to stop 'ora.cssd' on 'cntl069'
- CRS-2677: Stop of 'ora.cssd' on 'cntl069' succeeded
- CRS-2673: Attempting to stop 'ora.gipcd' on 'cntl069'
- CRS-2677: Stop of 'ora.gipcd' on 'cntl069' succeeded
- CRS-2673: Attempting to stop 'ora.gpnpd' on 'cntl069'
- CRS-2677: Stop of 'ora.gpnpd' on 'cntl069' succeeded
- CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cntl069' has completed
- CRS-4133: Oracle High Availability Services has been stopped.
2. 删除CRS <--root user
- [/root]#$CRS_HOME/crs/install/rootcrs.pl -deconfig -force >>>>>>>>>其他节点
- Using configuration parameter file: /grid/11.2/product/crs/install/crsconfig_params
- PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
- PRCR-1068 : Failed to query resources
- Cannot communicate with crsd
- PRCR-1070 : Failed to check if resource ora.gsd is registered
- Cannot communicate with crsd
- PRCR-1070 : Failed to check if resource ora.ons is registered
- Cannot communicate with crsd
- ...
-
- [/root]#$CRS_HOME/crs/install/rootcrs.pl -deconfig -force -lastnode >>>>>>>>>最后一个节点
- CRS-4535: Cannot communicate with Cluster Ready Services
- CRS-4000: Command Stop failed, or completed with errors.
- CRS-4544: Unable to connect to OHAS
- CRS-4000: Command Stop failed, or completed with errors.
- Removing Trace File Analyzer
- error: package cvuqdisk is not installed
- Successfully deconfigured Oracle clusterware stack on this node
3. [ALL NODES] 执行root.sh 配置CS <-- root user
- [/root]#CRS_HOME/root.sh
- Performing root user operation for Oracle 11g
-
- The following environment variables are set as:
- ORACLE_OWNER= grid
- ORACLE_HOME= /grid/11.2/product
-
- Enter the full pathname of the local bin directory: [/usr/local/bin]:
- The contents of "dbhome" have not changed. No need to overwrite.
- The contents of "oraenv" have not changed. No need to overwrite.
- The contents of "coraenv" have not changed. No need to overwrite.
-
- Entries will be added to the /etc/oratab file as needed by
- Database Configuration Assistant when a database is created
- Finished running generic part of root script.
- Now product-specific root actions will be performed.
- Using configuration parameter file: /grid/11.2/product/crs/install/crsconfig_params
- User ignored Prerequisites during installation
- Installing Trace File Analyzer
- OLR initialization - successful
- Adding Clusterware entries to upstart
- CRS-2672: Attempting to start 'ora.mdnsd' on 'cntl069'
- CRS-2676: Start of 'ora.mdnsd' on 'cntl069' succeeded
- CRS-2672: Attempting to start 'ora.gpnpd' on 'cntl069'
- CRS-2676: Start of 'ora.gpnpd' on 'cntl069' succeeded
- CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cntl069'
- CRS-2672: Attempting to start 'ora.gipcd' on 'cntl069'
- CRS-2676: Start of 'ora.cssdmonitor' on 'cntl069' succeeded
- CRS-2676: Start of 'ora.gipcd' on 'cntl069' succeeded
- CRS-2672: Attempting to start 'ora.cssd' on 'cntl069'
- CRS-2672: Attempting to start 'ora.diskmon' on 'cntl069'
- CRS-2676: Start of 'ora.diskmon' on 'cntl069' succeeded
- CRS-2676: Start of 'ora.cssd' on 'cntl069' succeeded
-
- ASM created and started successfully.
-
- Disk Group CRS created successfully.
-
- clscfg: -install mode specified
- Successfully accumulated necessary OCR keys.
- Creating OCR keys for user 'root', privgrp 'root'..
- Operation successful.
- Successful addition of voting disk 966344bb8c054f61bf7627175f0d53e8.
- Successful addition of voting disk 3f3735b4d5f44f79bf27624845c0c81b.
- Successful addition of voting disk 8bf524ee376b4ff1bf5c3b7e69e5c98b.
- Successfully replaced voting disk group with +CRS.
- CRS-4266: Voting file(s) successfully replaced
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 966344bb8c054f61bf7627175f0d53e8 (ORCL:ASMDISK1) [CRS]
- 2. ONLINE 3f3735b4d5f44f79bf27624845c0c81b (ORCL:ASMDISK2) [CRS]
- 3. ONLINE 8bf524ee376b4ff1bf5c3b7e69e5c98b (ORCL:ASMDISK3) [CRS]
- Located 3 voting disk(s).
- CRS-2672: Attempting to start 'ora.asm' on 'cntl069'
- CRS-2676: Start of 'ora.asm' on 'cntl069' succeeded
- CRS-2672: Attempting to start 'ora.CRS.dg' on 'cntl069'
- CRS-2676: Start of 'ora.CRS.dg' on 'cntl069' succeeded
- Preparing packages for installation...
- cvuqdisk-1.0.9-1
- Configure Oracle Grid Infrastructure for a Cluster ... succeeded
4. 检查CRS配置正常。
- [/grid/11.2/product]#crsctl status res -t
- --------------------------------------------------------------------------------
- NAME TARGET STATE SERVER STATE_DETAILS
- --------------------------------------------------------------------------------
- Local Resources
- --------------------------------------------------------------------------------
- ora.CRS.dg
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.asm
- ONLINE ONLINE cntl069 Started
- ONLINE ONLINE cntl070 Started
- ora.gsd
- OFFLINE OFFLINE cntl069
- OFFLINE OFFLINE cntl070
- ora.net1.network
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.ons
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.registry.acfs
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- --------------------------------------------------------------------------------
- Cluster Resources
- --------------------------------------------------------------------------------
- ora.LISTENER_SCAN1.lsnr
- 1 ONLINE ONLINE cntl069
- ora.cntl069.vip
- 1 ONLINE ONLINE cntl069
- ora.cntl070.vip
- 1 ONLINE ONLINE cntl070
- ora.cvu
- 1 ONLINE ONLINE cntl069
- ora.oc4j
- 1 ONLINE ONLINE cntl069
- ora.scan1.vip
- 1 ONLINE ONLINE cntl069
5. 确实LISTENER资源,netca 创建。
6. 如果已经有数据库,可使用下面命令加入资源
- 添加数据库(oracle用户下执行)
- [oracle@node1 ~]$ srvctl add database -d racdb -o /u01/product/oracle/11.2.0/db_1
- [oracle@node1 ~]$
-
- 添加实例(oracle用户下执行)
- [oracle@node1 ~]$ srvctl add instance -d racdb -i racdb1 -n node1
- [oracle@node1 ~]$ srvctl add instance -d racdb -i racdb2 -n node2
-
- 添加之前数据库的服务(oracle用户下执行)
- [oracle@node1 ~]$ srvctl add service -d racdb -s kettle -r racdb1 -a racdb2 -P BASIC
再次检查CRS状态
- [/grid/11.2/product]#crsctl status res -t
- --------------------------------------------------------------------------------
- NAME TARGET STATE SERVER STATE_DETAILS
- --------------------------------------------------------------------------------
- Local Resources
- --------------------------------------------------------------------------------
- ora.CRS.dg
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.DATA.dg
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.LISTENER.lsnr
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.asm
- ONLINE ONLINE cntl069 Started
- ONLINE ONLINE cntl070 Started
- ora.gsd
- OFFLINE OFFLINE cntl069
- OFFLINE OFFLINE cntl070
- ora.net1.network
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.ons
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- ora.registry.acfs
- ONLINE ONLINE cntl069
- ONLINE ONLINE cntl070
- --------------------------------------------------------------------------------
- Cluster Resources
- --------------------------------------------------------------------------------
- ora.LISTENER_SCAN1.lsnr
- 1 ONLINE ONLINE cntl069
- ora.cntl069.vip
- 1 ONLINE ONLINE cntl069
- ora.cntl070.vip
- 1 ONLINE ONLINE cntl070
- ora.cvu
- 1 ONLINE ONLINE cntl069
- ora.ecard2pt.db
- 1 ONLINE ONLINE cntl069 Open
- 2 ONLINE ONLINE cntl070 Open
- ora.oc4j
- 1 ONLINE ONLINE cntl069
- ora.scan1.vip
- 1 ONLINE ONLINE cntl069
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/17086096/viewspace-1982031/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/17086096/viewspace-1982031/