EM 12c Read-errors ORA-27603 ORA-27626报错总结

EM 12C报数据库报磁盘read errors的错误：

EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46) 

oracle官方分析认为是触发bug导致：

当cell节点出现内存溢出，ORA-27626: Exadata error: 215 (Cell failed to allocate memory)错误就会出在数据库的alert日志中产生。      这个错误发生在大量的smart scan操作需要分配更多的内存用于smart scan。这不是说cell节点需增加更多的内存用于smart scan，而是说在这一点上的smart scan需要内存比较高。      这是一个与I/O相关的错误。所以在trace文件中可以看到很多类似的以下信息  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  所以在相同的时间里，在EM12c中可以看到以下的read-error相关的信息。  EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46)  这个错误可以打patch 18169273来解决，这样ORA-27603和ORA-27626就不会再报出来，EM12c中报出的错误也不会报出来了。

=== ODM Cause Determination ===  The issue is caused by  Bug 18169273  - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY === ODM Cause Justification ===  On 11.2.0.3 Exadata, ORA-27626: Exadata error: 215 (Cell failed to allocate memory) generated in db alert log.  when the cell runs out of memory, for that scan it returns error 215 (OSS_ERRCODE_NOCELLMEM) to RDBMS. Once received the IO result, we print that in alert.log. Those errors mean that some smart scans didn't actually go through smart scan but did get serviced through Block IO. This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan.  It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point.  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  Above version, symptom matches  Bug 18169273.

估计和sga比较小也是有关系的


下面是oracle的分析思路

1、出现这个问题可能需要分析的日志

+ 所有数据库节点ASM和所有RDBMS实例的alert log  + GI/RDBMS的opatch输出 - ‘opatch lsinventory -detail'  + Exadata image版本 - imageinfo  + 所有cell节点sundiag输出  + ASM中的以下输出：  select name, read_errs from v$asm_disk;

2、结合em的报错信息，查看alert日志时发现db节点1 alert日志有报错

FileName = alert_reportdb1.log  Wed Nov 25 11:28:33 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_90598.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  ...  Wed Nov 25 11:43:49 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_95186.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  ...  Wed Nov 25 11:43:50 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_94153.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  ...  Wed Nov 25 12:30:21 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_127407.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  ...  Wed Nov 25 12:52:51 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_124095.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  ...  Wed Nov 25 12:52:52 2015  Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_71945.trc:  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)

3、分析trace文件

FileName = reportdb1_ora_90598.trc  Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production  With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,  Data Mining and Real Application Testing options  ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1  System name: Linux  Node name: fd02db01.deppon.com  Release: 2.6.39-400.128.17.el5uek  Version: #1 SMP Tue May 27 13:20:24 PDT 2014  Machine: x86_64  Instance name: reportdb1  Redo thread mounted by this instance: 1  Oracle process number: 359  Unix process pid: 90598, image: oracle@fd02db01.deppon.com  *** 2015-11-25 09:56:43.270  *** SESSION ID:(4162.57941) 2015-11-25 09:56:43.270  *** CLIENT ID:(192.168.4.221) 2015-11-25 09:56:43.270  *** SERVICE NAME:(report) 2015-11-25 09:56:43.270  *** MODULE NAME:(JDBC Thin Client) 2015-11-25 09:56:43.270  *** ACTION NAME:() 2015-11-25 09:56:43.270  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  *** 2015-11-25 09:56:46.861  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 1: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 2: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 3: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 4: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 5: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 7: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 8: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 9: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 10: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 11: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 12: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  *** 2015-11-25 09:56:47.636  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 6: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 8: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 9: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675  Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 8: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  *** 2015-11-25 09:56:48.971  Start of KCFIS reconnect diagnostics      ---KCFIS is responsible for ExaData-optimized I/O. It is related to smart scan. Appstate: 0x7f7e9e0c82d8 Cell: 172.16.247.110/, Device: /box/predicate1600118  Printing reconnect diagnostics: app_type 1 tsn 16 objd 1674375 objn 1674375  Dumping sqltext for sqlid: 5dmxfk16wnkav  kxsDumpSqlTextFromSQLID SQL=  select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,  FBASE_CODE,  FORG_NAME,  FORG_CODE,  FBIG_NAME,  FBIG_CODE,  ...  +++++++++++++++++ Dumping sqltext in chunks of 2048 bytes ++++++++  select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,  FBASE_CODE,  FORG_NAME,  FORG_CODE,  FBIG_NAME,  FBIG_CODE,  FSMALL_NAME,  FSMALL_CODE,  FDEPT_NAME,  ... ... TO_CHAR(T.FTOTAL_AMOUNT, 'fm999999990.00') FTOTAL_AMOUNT FROM t_etl_papers t LEFT JOIN crm.t_cust_custbasedata CUS ON t.fnumber = cus.fcustnumber AND cus.fcuststatus <> '2' WHERE 1 = 1 and t.FDEPT_CODE IN (SELECT T.FDEPTCODE CODE FROM DPRP.V_BUSINESS_ORGANIZATION T WHERE T.FDEPTCODE = 'W011302040502' OR T.FSMALLAREACODE = 'W011302040502' OR T.FBIGAREACODE = 'W011302040502' OR T.FORGCODE = 'W011302040502' OR T.FBASECODE ='W011302040502' OR t.fbosscode='W011302040502' OR T.fex_boss_code='W011302040502' union all select w.FDEPTCODE from DPRP.v_operation_organization w WHERE w.FDEPTCODE = 'W011302040502' OR w.FSMALLAREACODE = 'W011302040502' OR w.FBIGAREACODE = 'W011302040502' OR w.FORGCODE = 'W011302040502' OR w.FBASECODE = 'W011302040502' OR w.fbosscode = 'W011302040502' OR w.fex_boss_code='W011302040502' ) and t.FDATE = to_date('2015-11-24','yyyy-mm-dd') ORDER BY t.FDEPT_CODE,T.FNUMBER,T.FCHARGE_TYPE,T.FPAYMENT_TYPE,T.FBILL_TYPE) m ) )  +++++++++++++ End of SQl dump +++++++++++++  Dumping session  sid: 4162 ser: 57941 audsid: 231250857 user: 52/DPRP  flags: (0x100045) USR/- flags_idl: (0x1) BSY/-/-/-/-/-  flags2: (0x40009) -/-/INC  pid: 359 O/S info: user: grid, term: UNKNOWN, ospid: 90598  image: oracle@fd02db01.deppon.com  client details:  O/S info: user: root, term: unknown, ospid: 1234  machine: fspbaobiao-221 program: JDBC Thin Client  client info: 192.168.4.221  application name: JDBC Thin Client, hash value=2546894660  current SQL:  select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,  FBASE_CODE,  FORG_NAME,  FORG_CODE,  FBIG_NAME,  FBIG_CODE,  FSMALL_NAME,  FSMALL_CODE,  FDEPT_NAME,  ...  *** 2015-11-25 09:56:49.621  =======Dumping Fail Data blocks=======  Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987  Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353  Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141  Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 4: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729  Fail Block 5: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098  ====End of Dumping Fail Data blocks===  kcfis: Exadata error: Device 'o/172.16.247.111//box/predicate' error code 215 (Cell failed to allocate memory)  *** 2015-11-25 11:28:33.613  ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0  ORA-27626: Exadata error: 215 (Cell failed to allocate memory)

3、然后通过相关报错查看文档确定是bug

-- Search --  Note: This is INTERNAL ONLY research. No action should be taken by the customer on this information. This is research only, and may NOT be applicable to your specific situation.  Searched on: Cell failed to allocate memory  Bug 20310116 - ORA-27603 AND ORA-27626 IN ALERT LOG  Bug 18169273 - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY

4、采用打one-off-patch的方式来避免这个告警，或者忽略

=== ODM Proposed Solution ===  Apply one-off patch 18169273 on 11.2.0.3.28 Exadata Database. === ODM Proposed Solution Justif ===  Bug 18169273 is fixed in 12.1.0.2 and above.  There is a one-off patch on 11.2.0.3.28 Exadata Database.  These messages should not be considered buggy but only informative. === ODM Solution / Action Plan ===  This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan.  It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point.  To resolve the issue, please apply the patch 18169273.  Download and review the readme and pre-requisites for Patch 18169273.  Download link:  https://updates.oracle.com/download/10098816.html  Select a Release: Oracle 11.2.0.3.28 Exadata Database

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29953799/viewspace-1973100/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/29953799/viewspace-1973100/