提交任务到spark集群,老是超时。至于超时原因,可能是计算量太大。
解决方案:提交时,加一个参数“–conf spark.network.timeout=10000000”。
spark-submit
–conf spark.network.timeout=10000000 \
2018-10-29 18:07:02 ERROR TaskSchedulerImpl:70 - Lost executor driver on localhost: Executor heartbeat timed out after 123574 ms
2018-10-29 18:07:02 ERROR TaskSetManager:70 - Task 3 in stage 110.0 failed 1 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 110.0 failed 1 times, most recent failure: Lost task 3.0 in stage 110.0 (TID 496, localhost, executor driver): ExecutorLostFailure (executor driver exi