Kettle_内存溢出【已解决】

1.问题描述

   最近，收到异常邮件，提醒某个Job在执行的时候出现异常,就到Azkaban查一下日志，如下:

20-02-2020 05:22:52 CST job_all_table_nutnpo INFO - 2020/02/20 05:22:52 - NUT_INFO_AUTH.0 - Finished processing (I=50000, O=0, R=0, W=50000, U=0, E=0)
20-02-2020 05:22:54 CST job_all_table_nutnpo INFO - 2020/02/20 05:22:54 - INSERT/UPDATE NUT_INFO_AUTH.0 - Finished processing (I=50000, O=1603, R=50000, W=50000, U=0, E=0)
20-02-2020 05:23:35 CST job_all_table_nutnpo INFO - 2020/02/20 05:23:35 - NUT_LOG.0 - linenr 150000
20-02-2020 05:23:44 CST job_all_table_nutnpo INFO - 2020/02/20 05:23:44 - INSERT/UPDATE NUT_LOG.0 - linenr 150000
20-02-2020 05:24:49 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:02 CST job_all_table_nutnpo INFO - Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1777493319-59"
20-02-2020 05:25:04 CST job_all_table_nutnpo INFO - Exception in thread "qtp1777493319-80" Exception in thread "Thread-6" 	at java.text.MergeCollation.<init>(MergeCollation.java:224)
20-02-2020 05:25:05 CST job_all_table_nutnpo INFO - Exception in thread "Timer-2" Exception in thread "qtp1777493319-65" Exception in thread "qtp1777493319-162" 05:25:05,021 ERROR [2] In main loop, we have serious trouble
20-02-2020 05:25:05 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:05 CST job_all_table_nutnpo INFO - Exception in thread "qtp1777493319-63" 	at java.text.RBTableBuilder.build(RBTableBuilder.java:117)
20-02-2020 05:25:06 CST job_all_table_nutnpo INFO - Exception in thread "qtp1777493319-60 Selector3" 	at java.text.RBCollationTables.<init>(RBCollationTables.java:83)
20-02-2020 05:25:06 CST job_all_table_nutnpo INFO - 	at java.text.RuleBasedCollator.<init>(RuleBasedCollator.java:300)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 05:25:05,021 ERROR [2] In main loop, we have serious trouble
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at java.text.RuleBasedCollator.<init>(RuleBasedCollator.java:281)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at sun.util.locale.provider.CollatorProviderImpl.getInstance(CollatorProviderImpl.java:106)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at java.text.Collator.getInstance(Collator.java:242)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaBase.<init>(ValueMetaBase.java:239)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaBase.<init>(ValueMetaBase.java:223)
20-02-2020 05:25:07 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaBase.<init>(ValueMetaBase.java:215)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaString.<init>(ValueMetaString.java:37)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaString.<init>(ValueMetaString.java:33)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown Source)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at java.lang.Class.newInstance(Class.java:442)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:507)
20-02-2020 05:25:08 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta(ValueMetaFactory.java:50)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta(ValueMetaFactory.java:57)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.row.value.ValueMetaBase.getValueFromSQLType(ValueMetaBase.java:4920)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getValueFromSQLType(Database.java:2068)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getRowInfo(Database.java:2029)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getRow(Database.java:2741)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getLookup(Database.java:3113)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getLookup(Database.java:3103)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.core.database.Database.getLookup(Database.java:3099)
20-02-2020 05:25:09 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues(InsertUpdate.java:89)
20-02-2020 05:25:10 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.processRow(InsertUpdate.java:299)
20-02-2020 05:25:10 CST job_all_table_nutnpo INFO - 	at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
20-02-2020 05:25:10 CST job_all_table_nutnpo INFO - 	at java.lang.Thread.run(Thread.java:748)
20-02-2020 05:25:11 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:12 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:13 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:13 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:15 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:17 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:18 CST job_all_table_nutnpo INFO - 	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
20-02-2020 05:25:18 CST job_all_table_nutnpo INFO - 	at java.lang.StringCoding.encode(StringCoding.java:344)
20-02-2020 05:25:19 CST job_all_table_nutnpo INFO - 	at java.lang.String.getBytes(String.java:918)
20-02-2020 05:25:20 CST job_all_table_nutnpo INFO - 	at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
20-02-2020 05:25:20 CST job_all_table_nutnpo INFO - 	at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
20-02-2020 05:25:20 CST job_all_table_nutnpo INFO - 	at java.io.File.isDirectory(File.java:849)
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - 	at org.apache.karaf.main.Main.doMonitor(Main.java:282)
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - 	at org.apache.karaf.main.Main.access$100(Main.java:65)
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - 	at org.apache.karaf.main.Main$1.run(Main.java:207)
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space
20-02-2020 05:25:21 CST job_all_table_nutnpo INFO - java.lang.OutOfMemoryError: Java heap space

  从日志内容中，由java.lang.OutOfMemoryError: Java heap space 可以看出是内存不足导致的报错，所以需要将java运行内存调大一点。

  原因:  Kettle的运行内存受到限制

2. 解决方案

 a. 首先找到Kettle目录下的Spoon.bat文件

 

 b. 找到如下截图位置,进行修改,保存，退出

     

       主要修改了三个参数(256的倍数)，我解释一下:

• -Xms：设置JVM初始内存
• -Xmx：设置JVM最大可用内存
•  -XX:MaxPermSize：设置JVM最大允许分配内存，按需分配

 

  c. 重新运行Kettle,即可

 

(注: 仔细看一下上面的截图中的标题,可以看到, 人家也是建议  防止运行内存溢出, 将值改的大一点)