六、spark集群报错:GC overhead limit exceeded & thread task-result-getter-2
难点:
集群提交spark报错代码
xxxxxx
20/09/16 03:58:51 WARN scheduler.TaskSetManager: Stage 26 contains a task of very large size (1319 KB). The maximum recommended task size is 100 KB.
20/09/16 04:03:45 ERROR util.Utils: Uncaught exception in thread task-result-getter-2
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:65)
at org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1303)
at org.apache.spark.schedu