KETTLE 内存溢出错误

最新推荐文章于 2024-04-28 08:00:00 发布

这个有点吓人啊

最新推荐文章于 2024-04-28 08:00:00 发布

阅读量2.5k

点赞数

本文链接：https://blog.csdn.net/qq_43685346/article/details/84035609

版权

分享一下我老师大神的人工智能教程！零基础，通俗易懂！http://blog.csdn.net/jiangjunshow

也欢迎大家转载本篇文章。分享知识，造福人民，实现我们中华民族伟大复兴！

原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。

深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831

kettle内存溢出错误解决

环境：
源端数据库：oracle 10G R2
目标端数据库：oracle 11G R2
kettle版本：5.0.1-stable

报错：
抽取大数据量表时，报错，日志信息如下：

2015/01/05 11:27:42 - Spoon - 转换已经打开.2015/01/05 11:27:42 - Spoon - 正在打开转换 [TABLE_BWWN]...2015/01/05 11:27:42 - Spoon - 开始执行转换.2015/01/05 11:27:42 - TABLE_BWWN - 为了转换解除补丁开始  [TABLE_BWWN]2015/01/05 11:27:44 - max.0 - Finished reading query, closing connection.2015/01/05 11:27:44 - max.0 - 完成处理 (I=1, O=0, R=0, W=1, U=0, E=02015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : UnexpectedError:2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 一个未预期的错误发生在Spoon: probable cause:在停止Spoon前，请先关闭其它spoon窗口! 2015/01/05 12:25:08 - Spoon - Java heap space2015/01/05 12:25:08 - 插入 / 更新.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.bindBytesAsStream(OraclePreparedStatement.java:6713)2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.setBytesInternal(OraclePreparedStatement.java:6700)2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatement.setBytes(OraclePreparedStatement.java:6660)2015/01/05 12:25:08 - 插入 / 更新.0 -  at oracle.jdbc.driver.OraclePreparedStatementWrapper.setBytes(OraclePreparedStatementWrapper.java:150)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.row.value.ValueMetaBase.setPreparedStatementValue(ValueMetaBase.java:4704)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValue(Database.java:928)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValues(Database.java:943)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.core.database.Database.setValuesInsert(Database.java:888)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:113)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:301)2015/01/05 12:25:08 - 插入 / 更新.0 -  at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)2015/01/05 12:25:08 - 插入 / 更新.0 -  at java.lang.Thread.run(Unknown Source)2015/01/05 12:25:08 - Spoon - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: Java heap space2015/01/05 12:25:08 - Spoon -  at java.util.Arrays.copyOf(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.safeTrim(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.access$100(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding$StringDecoder.decode(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.decode(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.StringCoding.decode(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.String.<init>(Unknown Source)2015/01/05 12:25:08 - Spoon -  at java.lang.String.<init>(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.value.ValueMetaBase.convertBinaryStringToString(ValueMetaBase.java:1105)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.value.ValueMetaBase.getString(ValueMetaBase.java:1452)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.core.row.RowMeta.getString(RowMeta.java:230)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.showPreviewGrid(TransPreviewDelegate.java:302)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.refreshView(TransPreviewDelegate.java:268)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.trans.TransPreviewDelegate$1.widgetSelected(TransPreviewDelegate.java:188)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder.setSelection(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder.onMouse(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.custom.CTabFolder$1.handleEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1227)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7368)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8673)2015/01/05 12:25:08 - Spoon -  at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:625)2015/01/05 12:25:08 - Spoon -  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)2015/01/05 12:25:09 - 插入 / 更新.0 - 完成处理 (I=20575, O=20574, R=20575, W=20574, U=0, E=12015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Unexpected error2015/01/05 12:25:09 - 表输入.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException: 2015/01/05 12:25:09 - 表输入.0 - Couldn't get row from result set2015/01/05 12:25:09 - 表输入.0 - ORA-03111: 通信通道收到中断2015/01/05 12:25:09 - 表输入.0 - 2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2302)2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2270)2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:153)2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)2015/01/05 12:25:09 - 表输入.0 -  at java.lang.Thread.run(Unknown Source)2015/01/05 12:25:09 - 表输入.0 - Caused by: java.sql.SQLException: ORA-03111: 通信通道收到中断2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1082)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:369)2015/01/05 12:25:09 - 表输入.0 -  at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:273)2015/01/05 12:25:09 - 表输入.0 -  at org.pentaho.di.core.database.Database.getRow(Database.java:2290)2015/01/05 12:25:09 - 表输入.0 -  ... 4 more2015/01/05 12:25:09 - 表输入.0 - Finished reading query, closing connection.2015/01/05 12:25:09 - 表输入.0 - 完成处理 (I=20580, O=0, R=1, W=20579, U=0, E=12015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!2015/01/05 12:25:09 - Spoon - 转换完成!!2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!2015/01/05 12:25:09 - TABLE_BWWN - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : 错误被检测到!2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN2015/01/05 12:25:09 - TABLE_BWWN - TABLE_BWWN

解决方法：
找到kettle目录下的Spoon.bat文件，找到如下部分：

REM ******************************************************************REM ** Set java runtime options                                     **REM ** Change 512m to higher values in case you run out of memory   **REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **REM ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx512m" "-XX:MaxPermSize=256m"set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

修改参数，将内存值调大（256的倍数），调整后如下所示：

REM ******************************************************************REM ** Set java runtime options                                     **REM ** Change 512m to higher values in case you run out of memory   **REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **REM ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx1024m" "-XX:MaxPermSize=2048m"set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

重新执行kettle抽取数据，不再报内存溢出错误。

小结：
使用kettle时，如果遇到数据量大、大对象时很容易出现内存溢出现象，此时需要根据实际的硬件配置对预设的java堆栈大小进行调整。

原创作品，出自 “深蓝的blog” 博客，欢迎转载，转载时请务必注明以下出处，否则追究版权法律责任。

深蓝的blog：http://blog.csdn.net/huangyanlong/article/details/42453831