OCR自动备份文件目录cdata撑满磁盘目录

收到告警短信，一台oracle 11204 rac的磁盘目录达到80%。

登录机器，检查目录的磁盘使用情况，发现如下目录比较异常$GRID_HOME/cdata，这是ocr的自动备份目录，不应该这么大。

检查文件，如下：

-rw------- 1 root root     7254016 2016-05-09 02:03:24.354188445 +0800 99455096.ocr
-rw------- 1 root root     7254016 2016-04-05 10:01:36.127187229 +0800 99503628.ocr
-rw------- 1 root root     7254016 2016-09-21 02:02:06.974046491 +0800 99505384.ocr
-rw------- 1 root root     7254016 2017-05-21 01:27:53.204534421 +0800 99669812.ocr
-rw------- 1 root root     7254016 2016-06-07 18:05:47.388857746 +0800 99900344.ocr
-rw------- 1 root root     7254016 2017-01-16 13:20:46.046229638 +0800 99991638.ocr
-rw------- 1 grid oinstall 7254016 2015-12-17 09:03:05.881826383 +0800 backup00.ocr
-rw------- 1 grid oinstall 7254016 2015-12-17 05:03:05.445686297 +0800 backup01.ocr
-rw------- 1 grid oinstall 7254016 2015-12-17 01:03:04.910579534 +0800 backup02.ocr
-rw------- 1 grid oinstall 7254016 2015-12-17 01:03:05.150579534 +0800 day_.ocr
-rw------- 1 grid oinstall 7254016 2015-12-16 01:03:02.517976741 +0800 day.ocr
-rw------- 1 grid oinstall 7254016 2015-12-16 01:03:02.594976741 +0800 week_.ocr
-rw------- 1 grid oinstall 7254016 2015-12-09 21:02:46.266978418 +0800 week.ocr

按道理应该只有下面的几个字母开头的文件，上面的数字开头的文件是不应该有的。

检查crs的log日志，发现：

2017-09-01 01:34:19.437: [  OCRRAW][526362368]th_delete_backupfile: Failed to delete the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup02.ocr] Retval:[-2]
2017-09-01 01:34:19.437: [  OCRSRV][526362368]th_delete_backupfile: Failed to delete the backup file:[backup02.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]
2017-09-01 01:34:19.681: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup01.ocr] Retval:[1]
2017-09-01 01:34:19.681: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup01.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 01:34:19.697: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup00.ocr] Retval:[1]
2017-09-01 01:34:19.697: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup00.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 01:34:19.712: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/26396436.ocr] Retval:[1]
2017-09-01 01:34:19.712: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[26396436.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 01:34:19.726: [  OCRSRV][526362368]th_manipulate_backups: Failed to rename the temporary backup file [26396436.ocr].
2017-09-01 05:34:20.005: [  OCRRAW][526362368]th_delete_backupfile: Failed to delete the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup02.ocr] Retval:[-2]
2017-09-01 05:34:20.005: [  OCRSRV][526362368]th_delete_backupfile: Failed to delete the backup file:[backup02.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]
2017-09-01 05:34:20.234: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup01.ocr] Retval:[1]
2017-09-01 05:34:20.234: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup01.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 05:34:20.247: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup00.ocr] Retval:[1]
2017-09-01 05:34:20.247: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup00.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 05:34:20.261: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/42213858.ocr] Retval:[1]
2017-09-01 05:34:20.261: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[42213858.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 05:34:20.275: [  OCRSRV][526362368]th_manipulate_backups: Failed to rename the temporary backup file [42213858.ocr].
2017-09-01 09:34:20.563: [  OCRRAW][526362368]th_delete_backupfile: Failed to delete the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup02.ocr] Retval:[-2]
2017-09-01 09:34:20.563: [  OCRSRV][526362368]th_delete_backupfile: Failed to delete the backup file:[backup02.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]
2017-09-01 09:34:20.794: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup01.ocr] Retval:[1]
2017-09-01 09:34:20.794: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup01.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 09:34:20.809: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/backup00.ocr] Retval:[1]
2017-09-01 09:34:20.809: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[backup00.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 09:34:20.823: [  OCRRAW][526362368]proprbkp_rename: Failed to rename the backup file [/app/grid/product/11.2.0/cdata/gw-db-cluster/32981792.ocr] Retval:[1]
2017-09-01 09:34:20.823: [  OCRSRV][526362368]th_rename_backupfile: Failed to rename the backup file:[32981792.ocr] Location:[/app/grid/product/11.2.0/cdata/gw-db-cluster]. Retval:[49]
2017-09-01 09:34:20.837: [  OCRSRV][526362368]th_manipulate_backups: Failed to rename the temporary backup file [32981792.ocr].

因为这些数字开头的文件是因为不能rename成字母开头的文件而生成的。

因为文件的权限是不对的。

解决方法：

（1）将用户和组是grid:oinstall的改成root:root。

（2）手动删除数字开头的ocr文件，最好留下3天的，防止无备份可用。

另外还有个bug跟这个情况有点像，但应该不是：

MOS上的BUG，ID 1191067.1，BUG7313884，文档上介绍fixed版本11.2.0.2，文档内容介绍如下：
Due to bug 9446443, automatic OCR backups are incorrectly owned which is preventing CRSD from overwriting them.

bug 9446443 is fixed in 11.2.0.2, 12.1.

It's recommended to apply patch to fix the issue, but if patch is unavailable, workaround is to change ownership and permission of all 7 automatic backup files manually. OCR should be owned by root, but depend on platform, group may or may not be root - you can check any randomly named backup file to identify what ownership and permission it should have; in example below:
-rw------- 1 root root     7143424 Aug 30 09:40 38455890.ocr
With this, please change all 7 automatic backup files to be owned  by root:root with permission "-rw-------"