oracle 10gORA-07445 RAC环境下一个案例

oracle 10gORA-07445

1.数据库使用的软件及其版本

序号	名称	内容
1	操作系统	CentOS release 5.6 (Final) x86-64
2	数据库版本	Oracle Database 10g Release 2
3	数据库安装包： 补丁包：	10201_database_linux_x86_64.cpio.gz p6810189_10204_Linux-x86-64.zip
4	OEMbug解决补丁	p8350262_10204_Generic.zip

2.数据库结构

3.现象

现在出现的报表系统goldengate同步进程异常中断，还有修改部分product表内容rollback。数据库后台报严重的错误。

主要排查以下错误日志：tomcat日志、数据库alter日志、数据表日志错误、数据库转储文件、goldengate进程、goldengate同步日志。

3.1 tomcat日志错误

jdbc版本：ojdbc14-10.2.0.4.0.jar

2012-05-14 16:15:36,755 [http-8083-3] ERROR net.sf.hibernate.transaction.JDBCTransaction - Rollback failed java.sql.SQLException: No more data to read from socket at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"] at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"]

3.2 数据库alter日志

Mon May 14 14:37:33 2012 Errors in file /oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc: ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] [] Mon May 14 14:48:14 2012 Process 0x0x2d8367dc0 appears to be hung while dumping Current time = 485605312, process death time = 485541208 interval = 60000

从上面的日志可以看出进程hung住，让我查看/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件

3.3 查看转存文件

/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件比较长，大概有5M，我们就看关键部分

Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x2b09d44be000, PC: [0x2b09d26ee608, _intel_fast_memcpy.A()+10] *** 2012-05-14 14:37:33.145 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] [] Current SQL statement for this session: update product set type=:1, ship_type=:2, stock_warning2=:3, stock_warning=:4, store_id=:5, PRODUCT_NAME=:6, UNIT_PRICE=:7, LIST_PRICE=:8, QUANTITY=:9, BRAND=:10, MODEL=:11, Spec=:12, UNIT=:13, MANUFACTORY=:14, PRODUCINGAREA=:15, RESELLER=:16, supplier_id=:17, CREATE_TIME=:18, UPDATE_TIME=:19, DESCRIPTION=:20, BIG_PICTURE=:21, SMALL_PICTURE=:22, CENTER_PICTURE=:23, POP_PICTURE=:24, POP_PICTURE1=:25, POP_PICTURE2=:26, POP_PICTURE3=:27, POP_PICTURE4=:28, POP_PICTURE5=:29, note=:30, TEMPLATE_ID=:31, CHECKSTATUS=:32, check_man=:33, check_time=:34, STATUS=:35, SEARCHKEY=:36, MAIN_CATEGORY=:37, PRODUCT_NO=:38, sid=:39, displayId=:40, SORTINDEX=:41, corpid=:42, LINKURL=:43, spec_name1=:44, spec_name2=:45, spec_value1=:46, spec_value2=:47, POP_TYPE=:48, tag=:49, visitCount=:50, startTime=:51, endTime=:52, refer_price=:53, base_price=:54, parent_id=:55, isconfig=:56, short_name=:57, custom_image=:58, app_usercount=:59, app_totalCount=:60, buy_totalCount=:61, weight=:62, integral=:63, integral_Count=:64, is_userPrice=:65, short_name_color=:66, short_name_bold=:67, p_color=:68, p_color_pic=:69, charges_flag=:70, EXT_FIELD1=:71, EXT_FIELD2=:72, extFileld3=:73, extFileld4=:74, ext_fileld5=:75, ext_fileld6=:76, ext_fileld7=:77, ext_fileld8=:78, ext_fileld9=:79, ext_fileld10=:80, ext_fileld11=:81, ext_fileld12=:82, ext_fileld13=:83, ext_fileld14=:84, ext_fileld15=:85, ext_fileld16=:86, ext_fileld17=:87, ext_fileld18=:88, ext_fileld19=:89, ext_fileld20=:90, esalesno=:91, classtype=:92, game_type=:93, each_buy_count=:94, limit_buy_count=:95, start_limit_time=:96, end_limit_time=:97, channel_flag=:98, YDPRICE=:99, TQYD_DAYS=:100, SCOPE=:101, VD_ID1=:102, VD_ID2=:103, VD_ID3=:104, VD_ID4=:105, VD_ID5=:106, AUTO_GTCARD=:107, TV_NOTE=:108, MOBILE_NOTE=:109, IS_YYPRODUCT=:110, EXT_FILELD41=:111, WARRANTY=:112, AGENT_ID=:113, min_buynum=:114, MODEL_NO=:115, tp_spec_value1=:116, tp_spec_value2=:117 where PRODUCT_ID=:118 ----- Call Stack Trace -----

ORA-07445这个是个内部错误，查看oracle官方文档解释

ORA-07445:exception encountered: core dump [string] [string] [string] [string]

[string][string]

Cause: An OS exception occurred which should resultin the creation of a core file.

This is aninternal error.

Action: Contact your customer support representative.

从这个解释让我们联系oracle的技术支持。

3.4 报表数据库goldengate同步日志

查看oracle goldengate进程

GGSCI (DB-87) 1> info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING REPLICAT ABORT REPL 10:43:32 00:00:01

查看goldengate日志

2012-05-24 18:03:19 WARNING OGG-00869 Oracle GoldenGate Delivery for Oracle, repl.prm: OCI Error calling OCITransCommit (status = 3114-ORA-03114: not connected to ORACLE), SQL<UPDATE "JSCNBI"."PRODUCT" SET "PRODUCT_NAME" = :a1,"UNIT_PRICE" = :a2,"LIST_PRICE" = :a3,"QUANTITY" = :a4,"BRAND" = :a5,"MODEL" = :a6,"SPEC" = :a7,"UNIT" = :a8,"MANUFACTORY" = :a9,"RESELLER" = :a10,"P>. 2012-05-24 18:03:19 WARNING OGG-01003 Oracle GoldenGate Delivery for Oracle, repl.prm: Repositioning to rba 7146675 in seqno 1671. 2012-05-24 18:03:19 ERROR OGG-00665 Oracle GoldenGate Delivery for Oracle, repl.prm: OCI Error calling OCITransCommit (status = 3114-ORA-03114: not connected to ORACLE), SQL<UPDATE "JSCNBI"."PRODUCT" SET "PRODUCT_NAME" = :a1,"UNIT_PRICE" = :a2,"LIST_PRICE" = :a3,"QUANTITY" = :a4,"BRAND" = :a5,"MODEL" = :a6,"SPEC" = :a7,"UNIT" = :a8,"MANUFACTORY" = :a9,"RESELLER" = :a10,"P>. 2012-05-24 18:03:19 ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, repl.prm: PROCESS ABENDING.

4.解决问题

这个oracle 10.2.0.4 在rac环境下的一个bug，有以下两种方法来解决

l 升级到10.2.0.5版本

l 尽量避免行迁移

4.1尽量避免行迁移

4.1.1 避免新插入的数据行迁移

通过配置pctfree和pctused参数来使表减少行迁移

SQL> alter table product pctfree 20; Table altered. SQL> alter table product pctfree 40; Table altered.

4.1.2 消除现有数据的行迁移

BEGIN

SYS.DBMS_STATS.GATHER_TABLE_STATS (

OwnName => 'JSCNBI'

,TabName => 'PRODUCT'

,Estimate_Percent => 0

,Method_Opt => 'FOR ALLINDEXED COLUMNS SIZE 1 '

,Degree => 4

,Cascade => TRUE

,No_Invalidate => FALSE);

END;

/

1)重新move表

SQL> alter table product move tablespace JSCNBIDATA01;

2)重新rebuild索引

执行sql语句查询出所有的索引 select 'alter index '|| I.INDEX_NAME || ' rebuild tablespace JSCNBIINDEX01;' from user_indexes i where I.TABLE_NAME='PRODUCT'; alter index VISICOUNT2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_STATUS rebuild tablespace JSCNBIINDEX01; alter index SID_INDEX2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_NAME2 rebuild tablespace JSCNBIINDEX01; alter index CREATE_TIME2 rebuild tablespace JSCNBIINDEX01; alter index CREATE_SID2 rebuild tablespace JSCNBIINDEX01; alter index PRODUCT_ID rebuild tablespace JSCNBIINDEX01;

3)重新制定clob存储空间

alter table PRODUCT move tablespace jscndata lob (EXT_FILELD41) store as(tablespace JSCNBIINDEX01); alter table PRODUCT move tablespace jscndata lob (DESCRIPTION) store as(tablespace JSCNBIINDEX01);

4)结束以后收集统计信息

BEGIN SYS.DBMS_STATS.GATHER_TABLE_STATS ( OwnName => 'JSCNBI' ,TabName => 'PRODUCT' ,Estimate_Percent => 0 ,Method_Opt => 'FOR ALL INDEXED COLUMNS SIZE 1 ' ,Degree => 4 ,Cascade => TRUE ,No_Invalidate => FALSE); END; /

此方法，以及在测试环境验证通过。

4.2 升级数据库到10.2.0.5 版本，或者11g