Cluster deploy mode is not compatible with master "local"解决办法

最新推荐文章于 2024-04-18 21:03:46 发布

功夫猫熊yeah

最新推荐文章于 2024-04-18 21:03:46 发布

阅读量3.1k

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/weixin_39031707/article/details/103522552

版权

spark 专栏收录该内容

28 篇文章 0 订阅

订阅专栏

spark-submit  --deploy-mode cluster --executor-memory 4G --executor-cores 4 --conf spark.executor.memoryOverhead=1000 --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf spark.network.timeout=300s --conf spark.executor.heartbeatIntervals=100s oss://taobao/_ali.py 20191212 00 0

报错,发现忘记加–master yarn,在aws上跑没问题,但是在aliyun e-mapreduce跑就报错报错如下:
Exception in thread “main” org.apache.spark.SparkException: Cluster deploy mode is not compatible with master “local”
解决办法:
加上–master yarn
修改后的提交命令

spark-submit --master yarn --deploy-mode cluster --executor-memory 4G --executor-cores 4 --conf spark.executor.memoryOverhead=1000 --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf spark.network.timeout=300s --conf spark.executor.heartbeatIntervals=100s oss://taobao/_ali.py 20191212 00 0

正常