11GRAC CRS启动失败

最新推荐文章于 2022-06-23 17:28:34 发布

客家族_Shark曾_小凡仙

最新推荐文章于 2022-06-23 17:28:34 发布

阅读量5.9k

点赞数

分类专栏： Oracle 管理

本文链接：https://blog.csdn.net/ZengMuAnSha/article/details/46310413

版权

Oracle 管理专栏收录该内容

184 篇文章 1 订阅

订阅专栏

[root@racnode2 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

又发生了失败!

[root@racnode2 ~]# crsctl check cssd
CRS-272: This command remains for backward compatibility only
Cluster Synchronization Services is online

[root@racnode2 ~]# crsctl check crsd
CRS-272: This command remains for backward compatibility only
Cannot communicate with Cluster Ready Services

[root@racnode2 ~]# crsctl check evmd
CRS-272: This command remains for backward compatibility only
Cannot communicate with Event Manager

启动CRS 报告说已经激活状态
[root@racnode2 ~]# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

[oracle@racnode1 ~]$ olsnodes -n
racnode1 1
racnode2 2
[oracle@racnode1 ~]$ olsnodes -i
PRCO-19: 检索集群中节点的列表时失败
PRCO-4: OCR 初始化失败
PROC-32: 本地节点上的集群准备服务尚未运行消息传送错误

OCR 坏了?

重导入不行!

[root@racnode1 ~]# ocrconfig -local -import /u01/crs_home/product/cdata/racnode1/backup_20150413_193124.olr
PROTL-19: Cannot proceed while the Oracle High Availability Service is running

[root@racnode1 ~]# ps -ef |grep d.bin
root 2489 1 0 09:37 ? 00:00:01 /u01/crs_home/product/bin/ohasd.bin reboot
grid 2636 1 0 09:38 ? 00:00:00 /u01/crs_home/product/bin/gipcd.bin
grid 2643 1 0 09:38 ? 00:00:00 /u01/crs_home/product/bin/mdnsd.bin
grid 2655 1 0 09:38 ? 00:00:00 /u01/crs_home/product/bin/gpnpd.bin
grid 2717 1 0 09:38 ? 00:00:01 /u01/crs_home/product/bin/ocssd.bin
root 2833 1 0 09:41 ? 00:00:00 /u01/crs_home/product/bin/octssd.bin reboot
grid 2848 1 0 09:41 ? 00:00:00 /u01/crs_home/product/bin/evmd.bin
grid 3072 1 0 09:42 ? 00:00:00 /u01/crs_home/product/bin/oclskd.bin
root 3608 2788 0 09:49 pts/1 00:00:00 grep d.bin

[root@racnode1 ~]# crsctl stat res -t -init
----------------------------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS Cluster Resources
--------------------------------------------------------------------------------------------------------------------------------------------
ora.asm 1 ONLINE ONLINE racnode1 Started
ora.crsd 1 ONLINE OFFLINE
ora.cssd 1 ONLINE ONLINE racnode1
ora.cssdmonitor 1 ONLINE ONLINE racnode1
ora.ctssd 1 ONLINE ONLINE racnode1 ACTIVE:0
ora.diskmon 1 ONLINE ONLINE racnode1
ora.drivers.acfs 1 ONLINE ONLINE racnode1
ora.evmd 1 ONLINE ONLINE racnode1
ora.gipcd 1 ONLINE ONLINE racnode1
ora.gpnpd 1 ONLINE ONLINE racnode1
ora.mdnsd 1 ONLINE ONLINE racnode1

就ORA.CRSD 没启动

[grid@racnode1 crsd]$ tail -500 crsd.log

[ OCRAPI][3010044496]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2015-06-01 09:42:37.584: [ OCRAPI][3010044496]a_init:13!: Clusterware init unsuccessful : [44]
2015-06-01 09:42:37.584: [ CRSOCR][3010044496] OCR context init failure. Error: PROC-44: 缃.??板.?.?缁..?ｆ.浣.腑?洪. 缃.??板.?.?缁..?ｆ.浣..璇.[7]
2015-06-01 09:42:37.584: [ CRSD][3010044496][PANIC] CRSD exiting: Could not init OCR, code: 44
2015-06-01 09:42:37.584: [ CRSD][3010044496] Done.

查看日志 OCR PROC-44

表决磁盘可以啊

[root@racnode1 crsd]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 2c284f8178974f76bf16baabb5920203 (/dev/asm-diska) [ORC]
2. ONLINE 438ef4b51ebd4ffabfbf9153db6d90d1 (/dev/asm-diskb) [ORC]
3. ONLINE a9dfb2767c3e4fc7bfe1168f675262ad (/dev/asm-diskc) [ORC]

ASM 实例启动了

[grid@racnode1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on 星期一 6月 1 10:36:15 2015

Copyright (c) 1982, 2009, Oracle. All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>
GROUP_NUMBER HEADER_STATUS STATE PATH UDID
------------ ------------------------ ---------------- -------------------------------------------------- ----------
3 MEMBER NORMAL /dev/asm-diskc
3 MEMBER NORMAL /dev/asm-diska
1 MEMBER NORMAL /dev/asm-diske
2 MEMBER NORMAL /dev/asm-diskf
3 MEMBER NORMAL /dev/asm-diskb

表决磁盘组在线

只有个日志磁盘组没MOUNT

SQL> Select group_number,name,state,type,total_mb,free_mb From v$asm_diskgroup;

GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB
------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ----------
1 ARCHI MOUNTED EXTERN 5120 4688
2 DATA MOUNTED EXTERN 10240 8061
3 ORC MOUNTED NORMAL 3072 2146
0 REDO DISMOUNTED 0 0

可以 mount 成功

ALTER DISKGROUP redo mount
Diskgroup altered. redo mount
Diskgroup altered.

难道是因为CRS无法访问ASM吗? 查看监听,启动监听, 等待动态注册.......

[grid@racnode1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 01-6月 -2015 10:48:28

Copyright (c) 1991, 2009, Oracle. All rights reserved.

正在连接到 (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12541: TNS: 无监听程序
TNS-12560: TNS: 协议适配器错误
TNS-00511: 无监听程序
Linux Error: 2: No such file or directory
[grid@racnode1 ~]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 01-6月 -2015 10:48:39

Copyright (c) 1991, 2009, Oracle. All rights reserved.

启动/u01/crs_home/product/bin/tnslsnr: 请稍候...

TNSLSNR for Linux: Version 11.2.0.1.0 - Production
系统参数文件为/u01/crs_home/product/network/admin/listener.ora
写入/u01/crs_home/base/diag/tnslsnr/racnode1/listener/alert/log.xml的日志信息
监听: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

正在连接到 (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
LISTENER 的 STATUS
------------------------
别名 LISTENER
版本 TNSLSNR for Linux: Version 11.2.0.1.0 - Production
启动日期 01-6月 -2015 10:48:39
正常运行时间 0 天 0 小时 0 分 1 秒
跟踪级别 off
安全性 ON: Local OS Authentication
SNMP OFF
监听程序参数文件 /u01/crs_home/product/network/admin/listener.ora
监听程序日志文件 /u01/crs_home/base/diag/tnslsnr/racnode1/listener/alert/log.xml
监听端点概要...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
监听程序不支持服务
命令执行成功

SQL> alter system register;

System altered.

lsnrctl status 等了好久哦!!

动态注册读的是parameter file
静态注册读的是listener.ora

SQL> show parameter spfile

NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
spfile string +ORC/racnode-cluster/asmparame
terfile/registry.253.876943459

SQL> show parameter lis

NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
listener_networks string
local_listener string
remote_listener string

SQL> alter system set LOCAL_LISTENER='(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.4.100)(PORT = 1521))';

System altered.

这样也等了好久

[grid@racnode1 racnode1]$ vi alertracnode1.log
[grid@racnode1 racnode1]$ pwd
2015-06-01 11:34:13.433
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:15.469
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:17.498
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:19.546
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:21.618
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:23.689
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:25.739
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:27.784
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:29.824
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:31.851
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:33.892
[ohasd(2489)]CRS-2765:资源 'ora.crsd' 已失败 (在服务器 'racnode1' 上)?B
2015-06-01 11:34:33.892
[ohasd(2489)]CRS-2771:已达到资源 'ora.crsd' 的最大重新启动尝试次数; 将不会重新启动。

再查看CRSD.LOG

2015-06-01 11:34:16.514: [ OCRAPI][2567017040]a_init:13!: Clusterware init unsuccessful : [44]
2015-06-01 11:34:16.515: [ CRSOCR][2567017040] OCR context init failure. Error: PROC-44: 网络地址和网络接口操作中出错网络地址和网络接口操作错误 [7]
2015-06-01 11:34:16.515: [ CRSD][2567017040][PANIC] CRSD exiting: Could not init OCR, code: 44
2015-06-01 11:34:16.515: [ CRSD][2567017040] Done.

这次用GRID看清了具体错误!

grid@racnode1 crsd]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

#node1
192.168.4.100 racnode1.localdomain racnode1
192.168.4.110 racnode1-vip.localdomain racnode1-vip
192.168.5.100 racnode1-priv.localdomain racnode1-priv
#node2
192.168.4.101 racnode2.localdomain racnode2
192.168.4.111 racnode2-vip.localdomain racnode2-vip
192.168.5.101 racnode2-priv.localdomain racnode2-priv

192.168.4.121 scan-cluster.localdomain scan-cluster

看下是否能PING 通结果私有网卡没通

grid@racnode1 cssd]$ oifcfg iflist
eth0 192.168.4.0

网卡1 接入系统却没有激活

操作系统图形界面激活网卡1
[grid@racnode1 cssd]$ oifcfg iflist
eth0 192.168.4.0
eth1 192.168.5.0

等了10分钟后一切都好了

为什么网卡在停机和重启后无法激活有待进步考证