Spark 总结项目一---报错:bigger than spark.driver.maxResultSize

七、spark集群报错：bigger than spark.driver.maxResultSize

难点
报错代码

20/09/15 15:21:32 ERROR scheduler.TaskSetManager: Total size of serialized results of 423 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 423 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler