用scala写spark堆空间问题解决

 

 

19/05/27 17:23:00 INFO SecurityManager: Changing modify acls groups to: 
19/05/27 17:23:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(Administrator); groups with view permissions: Set(); users  with modify permissions: Set(Administrator); groups with modify permissions: Set()
19/05/27 17:23:01 INFO Utils: Successfully started service 'sparkDriver' on port 62158.
19/05/27 17:23:01 INFO SparkEnv: Registering MapOutputTracker
19/05/27 17:23:01 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:216)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:198)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
    at Spark1$.main(Spark1.scala:6)
    at Spark1.main(Spark1.scala)
19/05/27 17:23:01 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:216)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:198)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
    at Spark1$.main(Spark1.scala:6)
    at Spark1.main(Spark1.scala)

 

解决方法：搞定。