问题描述:
在执行spark程序时,需要读取200w数据作为缓存。在利用.broadcast广播这些数据时,遇到Exception in thread "main" java.lang.OutOfMemoryError: Java heap space问题。
报错信息如下:
15/09/15 05:26:09 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on ip-172-31-10-136.ec2.internal:34472 in memory (size: 2.0 KB, free: 397.3 MB)
15/09/15 05:26:09 INFO spark.ContextCleaner: Cleaned broadcast 3
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2351)
at java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2276)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1428)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.ArrayList.writeObject(ArrayList.java:762)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.Obj