RAC删除 /tmp/.oracle目录导致故障

最新推荐文章于 2021-04-13 01:44:04 发布
最新推荐文章于 2021-04-13 01:44:04 发布
阅读量379 收藏
点赞数
文章标签： 数据库 操作系统 运维

故障问题描述

客户技术人员电话反映应用出现有张表无法访问的问题（其实是数据库已经无法进行写入操作），同时，也无法远程登录检查，数据库日志等信息。根据当时情况，基本无法判断确定具体是什么原因导致表无法访问。
     
   第一时间，我们建议客户相关技术人员重启数据库，但是发现无法正常关闭数据库，使用shutdown abort强制关闭数据库然后重启。

到现场后经过同现场工程师沟通了解到在凌晨2点删除了 /tmp目录下临时文件。

经过分析确认这次数据库故障是删除 /tmp/.oracle目录导致故障。

 

故障分析

1、根据数据库报警日志确认问题：

Sun Nov 15 04:42:48 2015

WARNING: ASM communication error: op 11 state 0x50 (3113)

ERROR: slave communication error with ASM

Unable to create archive log file '+ARDATA/arch/1_15180_854962843.dbf'

ARC0:Error19504 Creating archive log file to

'+ARDATA/arch/1_15180_854962843.dbf'

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance szps1 - Archival Error

ORA-16038: log 3 sequence# 15180 cannot be archived

ORA-19504: failed to create file ""

ORA-00312: online log 3 thread 1:

'+DATA/szonline/onlinelog/group_3.263.854962845'

Sun Nov 15 04:42:48 2015

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance szps1 - Archival Error

ORA-16014: log 3 sequence# 15180 not archived, no available destinations

ORA-00312: online log 3 thread 1:

'+DATA/szonline/onlinelog/group_3.263.854962845'

Sun Nov 15 04:47:49 2015

WARNING: ASM communication error: op 11 state 0x50 (3113)

ERROR: slave communication error with ASM

Unable to create archive log file '+ARDATA/arch/1_15180_854962843.dbf'

ARC3: Error 19504 Creating archive log file to

'+ARDATA/arch/1_15180_854962843.dbf'

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance szps1 - Archival Error

ORA-16038: log 3 sequence# 15180 cannot be archived

ORA-19504: failed to create file ""

ORA-00312: online log 3 thread 1:

'+DATA/szonline/onlinelog/group_3.263.854962845'

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance szps1 - Archival Error

ORA-16014: log 3 sequence# 15180 not archived, no available destinations

ORA-00312: online log 3 thread 1:

'+DATA/szonline/onlinelog/group_3.263.854962845'

 

从上面的信息，我们看到，从2015年11月15日凌晨4:42 开始到库被重启前一直都要报无法连接ASM实例，导致了无法写日志写归档错误 。

2. ASM 告警日志提示错误

Sun Nov 15 04:42:48 2015

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 15204714

Sun Nov 15 04:47:49 2015

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 5832778

Sun Nov 15 04:52:50 2015

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 5701650

Sun Nov 15 04:57:50 2015

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 14221586

Sun Nov 15 05:02:50 2015

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 15204858

 

从上面的信息看到，ASM也从2015年11月15日开始报错，结合之前无法写日志写归档的报错，我们基本可以确认数据库不正常是由于ASM问题引发的。

3、grid 错误日志：

2015-11-15 02:00:51.685:

[/oracle/grid/bin/oraagent.bin(4325770)]CRS-5016:Process "/oracle/grid/bin/lsnrctl" spawned by agent "/oracle/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in

"/oracle/grid/log/szdb01/agent/crsd/oraagent_grid/oraagent_grid.log"

2015-11-15 02:03:51.823:

[/oracle/grid/bin/oraagent.bin(4325770)]CRS-5818:Aborted command 'start' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {0:1:8} in

/oracle/grid/log/szdb01/agent/crsd/oraagent_grid/oraagent_grid.log.

2015-11-15 02:05:55.825:

[/oracle/grid/bin/oraagent.bin(4325770)]CRS-5818:Aborted command 'check' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {0:1:8} in

/oracle/grid/log/szdb01/agent/crsd/oraagent_grid/oraagent_grid.log.

2015-11-15 02:06:01.721:

[/oracle/grid/bin/oraagent.bin(4325770)]CRS-5016:Process "/oracle/grid/bin/lsnrctl" spawned by agent "/oracle/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in

"/oracle/grid/log/szdb01/agent/crsd/oraagent_grid/oraagent_grid.log"

2015-11-15 02:15:37.519:

[ctssd(5505096)]CRS-2409:The clock on host szdb01 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

2015-11-15 02:27:12.538:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:12.621:

[/oracle/grid/bin/orarootagent.bin(5374128)]CRS-5822:Agent '/oracle/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:41827} in

/oracle/grid/log/szdb01/agent/crsd/orarootagent_root/orarootagent_root.log.

2015-11-15 02:27:12.648:

[/oracle/grid/bin/oraagent.bin(4325770)]CRS-5822:Agent '/oracle/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:1:68} in

/oracle/grid/log/szdb01/agent/crsd/oraagent_grid/oraagent_grid.log.

2015-11-15 02:27:12.649:

[/oracle/grid/bin/oraagent.bin(6160520)]CRS-5822:Agent '/oracle/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:9:52067} in

/oracle/grid/log/szdb01/agent/crsd/oraagent_oracle/oraagent_oracle.log.

2015-11-15 02:27:14.712:

[crsd(7012528)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

 /oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:15.718:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:16.818:

[crsd(6160522)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:17.817:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:18.920:

[crsd(15794626)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:19.922:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:21.009:

[crsd(6094920)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

 /oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:22.009:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:23.091:

[crsd(6160526)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

 /oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:24.099:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:25.196:

[crsd(6160528)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:26.196:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:27.288:

[crsd(6160530)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:28.291:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:29.366:

[crsd(12779904)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:30.371:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:31.465:

[crsd(5832832)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:32.466:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:33.545:

[crsd(5832834)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in

/oracle/grid/log/szdb01/crsd/crsd.log.

2015-11-15 02:27:34.548:

[ohasd(5177356)]CRS-2765:Resource 'ora.crsd' has failed on server 'szdb01'.

2015-11-15 02:27:34.548:

[ohasd(5177356)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

2015-11-15 02:27:34.555:

[ohasd(5177356)]CRS-2769:Unable to failover resource 'ora.crsd'.

 

 

根据GRID的报何错信息，我们基本可以推出：

1)   GRID 日志记录，在11月15日凌晨2:00出现监听故障，2:27出现 CRS故障，2:00删除了 /tmp/.oracle的文件夹，GRID 马上就出现了监听器故障，后续有出现了CRS故障；

2)   ORACLE数据库实例通过监听器连接ASM实例，在监听器故障之前已经建立的连接，当监听器故障时仍然可以正常使用，而数据库实例的启动归档日志进程进行归档时需要与ASM 实例建立新的连接，这个时候因为监听器已经故障了，导致数据库实例新建的连接无法连接到ASM实例，导致归档失败；

3)   由于数据库实例有多个日志组，刚开始的时候只有一个日志组被写满无法归档，后来随着时间推移所有的日志组都被写满，但所有的日志组都没有完成归档，导致无日志组可用来写入 redo 条目，阻塞了应用的SQL。

4、删除 /tmp/.oracle目录导致故障的案例（该案来源于ORALCE metalink文档 ID 370605.1）

Clusterware Intermittently Hangs And Commands Fail With CRS-184 as Network Socker Files in /tmp/.oracle or /var/tmp/.oracle Gets Deleted (文档 ID 370605.1)

 

 In this Document


Symptoms

 


Cause

 


Solution

 

Applies to:

 

Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.1.0.7

[Release 10.1 to 11.1]
Information in this document applies to any platform.

SYMPTOMS

 

CRS hangs intermittently

crs_stat -treturns

CRS-0184: Cannot communicate with the CRS daemon.

node1 [crs]> crsctl check crsd
Cannot communicate with CRS
node1 [crs]> crsctl check css
Failure 1 contacting CSS daemon


ps -ef |grep d.bin will give you the pid of the process

for example

ps -ef |grep d.bin
oracle 19703 192810 Apr10 ? 00:01:03 /home/oracle/oracle/product/10.2.0/crs/bin/evmd.bin
oracle 19976 19950 0 Apr10 ? 00:06:47 /home/oracle/oracle/product/10.2.0/crs/bin/ocssd.bin
root 19323 1 0 Apr10? 00:08:47 /home/oracle/oracle/product/10.2.0/crs/bin/crsd.bin

 

CAUSE

This is caused bya cron job that cleans up the /tmp directory which also removes the Oracle socket files in /tmp/.oracle

 

SOLUTION

Do not remove /tmp/.oracle or /var/tmp/.oracle or its files while Oracle Clusterware is up.

 

 

结论：根据当前的实际情况，再通过ORACLE 官方METALINK的实际案例，我们最终确认这次数据库突然无法正常运行是由于相关操作系统运维人员在一些日常运维的管理中，清理一些日志引发的，和数据库无关。为了防范于未然，为了下次再出现类似的问题，我们建议对数据库服务器相关的操作系统层面的运维，需要加强数据库管理员和操作系统管理员之间的协商和合作。

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/21582653/viewspace-2128357/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/21582653/viewspace-2128357/

congjiong3047
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
/tmp/.oracle，/var/tmp/.oracle如果删除了，该怎么办？
elvis的Oracle Lab
04-28 1918
在官方文档有如下的描述： 3 Installing Oracle Grid Infrastructure and Oracle Real Application Clusters Caution: After installation is complete, do not remove manually or run cron jobs that remove /tmp/.orac
误删/tmp目录，不能启动ubuntu
王润青
09-04 2456
有这样一系列操作： 双系统，开机自动挂载windows硬盘 cp -a /tmp windows下tmp rm -r /tmp ln -s windows下tmp /tmp 结果重启ubuntu不能启动。 分析：曾经把/tmp在/home下做过链接，结果是正常的。所以，这个地方应该是ubuntu启动时首先检查/tmp，然后才是挂载windows。当检查到/tmp时，发现找不到真正的/tm
oracle tmp目录权限,Oracle rac下/tmp/bootstrap权限问题
weixin_30160343的博客
04-04 125
rac下/tmp/bootstrap权限问题环境：CentOS5.6 64bit、linux.x64_11gR2_grid.zip1、报错[grid@node1 grid]$ ./runcluvfy.sh stage -pre crsinst -n node1,node2 -fixup -verbose/bin/rm: cannot remove directory `/tmp/bootstrap...
oracle var/tmp,坑人的/tmp/.oracle
weixin_39757040的博客
04-02 377
看似是个临时目录，但是删除它会导致RAC运行出错，只能重启RAC或server解决；另外，若启动RAC时该目录仍残留有关闭前的各种文件，也会导致RAC无法正常启动，此时需要将其删除才能成功；The hidden directory ‘/var/tmp/.oracle’ was removed while instances & the CRS stack were up and runni...
/var/tmp/目录无.oracle导致listener启动失败
北漂IT一卒的专栏
12-04 1585
在测试设备上，原先错误操作，对/var/tmp/目录的权限做过修改，今天在这个测试设备上恢复启动一个oracle数据库，启动监听时报错：    [oracle@localhost admin]$ lsnrctl start     LSNRCTLfor Linux: Version 10.2.0.2.0 - Production on 04-JAN-201210:10:11     Copy
Oracle数据库下载、安装、卸载教程
12-15
第一步：去Oracle官网https://www.oracle.com/database/technologies/oracle-database-software-downloads.html 其中包括两个压缩包：win64_11gR2_database_1of2.zip，win64_11gR2_database_2of2.zip 当出现如上...
oracle19c cdb rac to rac adg部署20200307.pdf
03-07
oracle 19c容器多租户 rac to rac的adg测试部署方案，完全实测步骤。 使用了DG_BROCKER管理，方便switchover。
Oracle 19C CDB创建与删除
12-14
一.通过以下链接方法安装ORACLE19C。...cp /etc/init.d/oracledb_ORCLCDB-19c /etc/init.d/oracledb_ORCLABC-19c #利用ORCLCDB的配置文件，创建ORCLABC的配置文件 2.修改内容 export ORACLE_HOME=/opt/oracle/
Oracle RAC 12c运维最佳实践指南.pdf
10-14
Oracle RAC 12c运维最佳实践指南.pdf
oracle rac 不停机调整时间.txt
04-12
oracle rac 不停机调整时间.txt
坑人的/tmp/.oracle
congdiku5262的博客
11-21 123
看似是个临时目录，但是删除它会导致RAC运行出错，只能重启RAC或server解决；另外，若启动RAC时该目录仍残留有关闭前的各种文件，也会导致RAC无法正常启动，此时需要将其删除才能成功； The hidden direct...
/tmp/.oracle目录导致的crs不能正常启动
cjh621117的博客
06-29 179
今天解决了一个小问题，心情很好 故障原因： 重启了crs以后，cssd就无法启动 上去看了看日志，没有特殊的提示，只是报 c...
tmp .oracle,【翻译自mos文章】在/tmp/.oracle目录下的文件是什么？
weixin_30404219的博客
04-10 146
在/tmp/.oracle目录下的文件是什么？来源于：What Are The Files In The /tmp/.oracle Directory? (文档 ID 1050601.6)问题描述：在/tmp/.oracle目录下有很多临时文件，比如:s#1571.1s#1753.1s#3246.1用操作系统的file命令来看这些文件的类型：[oracle@rhel63single .oracle...
oracle var/tmp,请教/tmp/.oracle & /var/tmp/.oracle 目录下文件的用途及清理w
weixin_36051940的博客
04-02 317
[root@mlab2 ~]# cd /var/tmp/.oracle/[root@mlab2 .oracle]# ls -ltrtotal 0srwxrwxrwx 1 oracle oinstall 0 Dec72009 s#13153.2srwxrwxrwx 1 oracle oinstall 0 Dec72009 s#13153.1prw-r--r-- 1 oracle oi...
oracle下.tmp后缀文件,【翻譯自mos文章】在/tmp/.oracle目錄下的文件是什么？
weixin_30695203的博客
04-13 184
在/tmp/.oracle目錄下的文件是什么？來源於：What Are The Files In The /tmp/.oracle Directory? (文檔 ID 1050601.6)問題描述：在/tmp/.oracle目錄下有很多臨時文件，比如:s#1571.1s#1753.1s#3246.1用操作系統的file命令來看這些文件的類型：[oracle@rhel63single .oracle...
【翻译自mos文章】Clusterware间歇性的hang,命令报CRS-184并且Network Socket Files in /tmp/.oracle or /var/tmp/.oracle被删
msdnchina的专栏@JiNan,ShanDong
12-23 1435
APPLIES TO: Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.1.0.7 [Release 10.1 to 11.1] Information in this document applies to any platform. SYMPTOMS CRS hangs intermittently crs_
/tmp/.oracle 和/var/tmp/.oracle文件的四篇经典文章
msdnchina的专栏@JiNan,ShanDong
12-14 1874
/tmp/.oracle 和/var/tmp/.oracle文件的四篇经典文章
【翻译自mos文章】在/tmp/.oracle目录下的文件是什么？
msdnchina的专栏@JiNan,ShanDong
12-23 1792
【翻译自mos文章】在/tmp/.oracle目录下的文件是什么？
oracle var/tmp,必须吐槽一下 dd /var/tmp/.oracle/npohasd
weixin_28864057的博客
04-02 935
安装oracle 11g RAC 时执行root.sh无法成功dd一会儿也就罢了，下次重启crs时还起不来(主机重启也一样)。类似如下：[root@ol6-112-rac1 ~]# ps -ef |grep crsroot 1983 1 0 10:39 ? 00:00:00 /u01/app/11.2.0.3/grid/bin/crsctl.bin start h...
PRCR-1079 : 无法启动资源 ora.scan1.vip CRS-5017: The resource action "ora.scan1.vip start" encountered the following error: CRS-5005: IP Address: 192.168.16.7 is already in use in the network . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/orarootagent_root/orarootagent_root.log". CRS-2674: Start of 'ora.scan1.vip' on 'rac2' failed CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy start scan ... failed FirstNode configuration failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 9379. /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
最新发布
06-10
这个错误出现在 Oracle RAC（Real Application Clusters）环境中，通常是由于虚拟 IP 地址（VIP）已经在网络中被使用了导致的。建议您先检查网络配置，确保该 IP 地址没有被其他设备占用。如果没有其他设备使用该 IP...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值