CRS 启动故障一例

最新推荐文章于 2021-04-03 14:54:16 发布

cmumbk5598

最新推荐文章于 2021-04-03 14:54:16 发布

阅读量170

点赞数

文章标签：数据库操作系统

机器启动后,使用crs_stat -t报错如下:
CRS-0184

--查看 /var/log/messages
May 16 16:41:51 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4507.
May 16 16:41:52 rac1 su(pam_unix)[5704]: session opened for user oracle by (uid=0)
May 16 16:41:52 rac1 su(pam_unix)[5704]: session closed for user oracle
May 16 16:41:52 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4710.
May 16 16:42:12 rac1 sshd(pam_unix)[5739]: session opened for user oracle by (uid=0)
May 16 16:42:51 rac1 su(pam_unix)[5772]: session opened for user oracle by (uid=0)
May 16 16:42:51 rac1 su(pam_unix)[5772]: session closed for user oracle

[root@rac1 ~]# more /tmp/crsctl.4710
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

--OCR 存储在OCFS 文件系统中,
[root@rac1 ~]# mount /ocfs/
ocfs2_hb_ctl: Device name specified was not found while reading uuid
mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: "Operation not permitted"

--发现是/dev/sdb11 不存在: ls /dev/sdb11 看得出.
[root@rac1 ~]# vi /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                 /                       ext3    defaults        1 1
none                    /dev/pts                devpts gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/sdb11              /ocfs ocfs2 _netdev,datavolume,nointr 0 0
LABEL=/u01              /u01                    ext3    defaults        1 2
LABEL=SWAP-sda2         swap                    swap    defaults        0 0
/dev/hdc                /media/cdrom            auto    pamconsole,fscontext=system_u:object_r:removable_t,exec,noauto,managed 0 0
/dev/fd0                /media/floppy           auto    pamconsole,fscontext=system_u:object_r:removable_t,exec,noauto,managed 0 0

继续查询:fdisk -l 看出,设备应该是/dev/sdb1

所以修改 /etc/fstab, 把/dev/sdb11 改为/dev/sdb1

[root@rac1 ~]# mount /ocfs/
[root@rac1 ~]# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda1             7.9G 6.2G 1.4G 83% /
none                  506M     0 506M   0% /dev/shm
/dev/sda3             9.9G 4.1G 5.4G 43% /u01
/dev/sdb1             512M   82M 431M 16% /ocfs

--此过程观察操作系统日志文件/var/log/message,可以看到 Cluster Ready Services 会启动成功
当/ocfs没有mount上时,不断出现如下的log信息:Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4710.
May 16 16:41:51 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4507.
May 16 16:41:52 rac1 su(pam_unix)[5704]: session opened for user oracle by (uid=0)
May 16 16:41:52 rac1 su(pam_unix)[5704]: session closed for user oracle
May 16 16:41:52 rac1 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4710.
May 16 16:42:12 rac1 sshd(pam_unix)[5739]: session opened for user oracle by (uid=0)
May 16 16:42:51 rac1 su(pam_unix)[5772]: session opened for user oracle by (uid=0)
May 16 16:42:51 rac1 su(pam_unix)[5772]: session closed for user oracle

当/ocfs mount成功后:Cluster Ready Services completed waiting on dependencies, 并不在产生log信息
May 16 16:57:53 rac1 su(pam_unix)[7400]: session opened for user oracle by (uid=0)
May 16 16:57:53 rac1 su(pam_unix)[7431]: session opened for user oracle by (uid=0)
May 16 16:57:54 rac1 su(pam_unix)[7400]: session closed for user oracle
May 16 16:57:54 rac1 logger: Cluster Ready Services completed waiting on dependencies.
May 16 16:57:54 rac1 su(pam_unix)[7465]: session opened for user oracle by (uid=0)
May 16 16:57:55 rac1 su(pam_unix)[7431]: session closed for user oracle
May 16 16:57:55 rac1 logger: Cluster Ready Services completed waiting on dependencies.
May 16 16:57:56 rac1 su(pam_unix)[7465]: session closed for user oracle
May 16 16:57:56 rac1 logger: Cluster Ready Services completed waiting on dependencies.
May 16 16:58:54 rac1 su(pam_unix)[4190]: session opened for user oracle by (uid=0)
May 16 16:58:57 rac1 logger: Running CRSD with TZ =
May 16 16:58:57 rac1 su(pam_unix)[7626]: session opened for user oracle by (uid=0)

--log 中open 的 session 对应于以下的进程:
[root@rac1 ~]# ps -elf | grep -i 7626
4 S root      7626 7536 0 80   0 -   717 wait   16:58 ?        00:00:00 /bin/su -l oracle -c /bin/sh -c 'ulimit -c unlimited; cd /u01/app/oracle/product/10.2.0/crs_1/log/rac1/cssd; /u01/app/oracle/product/10.2.0/crs_1/bin/ocssd || exit $?'
4 S oracle    7635 7626 0 82   0 - 1157 wait   16:58 ?        00:00:00 /bin/sh -c ulimit -c unlimited; cd /u01/app/oracle/product/10.2.0/crs_1/log/rac1/cssd; /u01/app/oracle/product/10.2.0/crs_1/bin/ocssd || exit $?
0 R root     16725 5877 0 77   0 - 1422 -      17:05 pts/1    00:00:00 grep -i 7626
[root@rac1 ~]# ps -elf | grep -i 4190
4 S root      4190     1 0 78   0 -   579 wait   16:34 ?        00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /u01/app/oracle/product/10.2.0/crs_1/log/rac1/evmd; exec /u01/app/oracle/product/10.2.0/crs_1/bin/evmd '
4 S oracle    7503 4190 0 76   0 - 45256 322565 16:58 ?        00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/evmd.bin
0 S root     17193 5877 0 76   0 - 1146 pipe_w 17:06 pts/1    00:00:00 grep -i 4190

--运行 crs_stat 不再报错 CRS-0184
rac1-> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....t1.inst application    ONLINE    OFFLINE
ora....t2.inst application    ONLINE    OFFLINE
ora.d1_st.db   application    OFFLINE   OFFLINE
ora.devdb1.db application    ONLINE    ONLINE    rac1
ora....11.inst application    ONLINE    ONLINE    rac1
ora....12.inst application    ONLINE    OFFLINE
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    ONLINE    ONLINE    rac1
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   application    ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    OFFLINE
ora....C2.lsnr application    ONLINE    OFFLINE
ora.rac2.gsd   application    ONLINE    OFFLINE
ora.rac2.ons   application    ONLINE    OFFLINE
ora.rac2.vip   application    ONLINE    ONLINE    rac1