spark als java_ALS部署Spark集群入坑记

[Stage 236:> (0 + 0) / 400]17/12/04 09:45:55 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError

java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1841)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1534)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)

java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)

java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)

scala.collection.immutable.$colon$colon.writeObject(List.scala:379)

sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

栈溢出原因：由于ASL的迭代次数过多，DAG过深，导致RDD的lineage，栈溢出。

现有解决：1.迭代次数减少。

2.利用checkpoint？

参考：

其他：