Total size of serialized results of 20 tasks (1088.8 MB) is bigger thanspark.driver.maxResultSize (1024.0 MB)
pypark 2.0 (spark python) 执行中出现上述错误,解决方案:
SparkContext.setSystemProperty('spark.driver.maxResultSize', '10g')
Total size of serialized results of 20 tasks (1088.8 MB) is bigger thanspark.driver.maxResultSize (1024.0 MB)
pypark 2.0 (spark python) 执行中出现上述错误,解决方案:
SparkContext.setSystemProperty('spark.driver.maxResultSize', '10g')