spark maxResultSize

INFO org.apache.spark.deploy.yarn.ApplicationMaster: 
Unregistering ApplicationMaster with FAILED 
(diag message: 
	User class threw exception: 
		org.apache.spark.SparkException: 
			Job aborted due to stage failure: 
				Total size of serialized results of 19 tasks (1069.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
 
val conf = new SparkConf().setAppName("your app name")
conf.set("spark.yarn.executor.memoryOverhead", "2000")
conf.set("spark.driver.maxResultSize", "8g")

©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页