【spark】Cluster deploy mode is not applicable to Spark shells

前言

使用CDH5.13 集成 apche spark2.4.2 使用spark-shell命令报错。

spark配置

spark-defaults.conf文件如下：

spark.master                     yarn
spark.deploy.mode                cluster
spark.submit.deployMode          cluster
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://nameservice1/tmp/spark/log/
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.memory              1g
spark.driver.maxResultSize       2g
spark.executor.memory            2g
yarn.scheduler.maximum-allocation-mb  2048m
spark.executor.instances         600
spark.executor.extraJavaOptions  -XX:+PrintGCDetails
spark.shuffle.service.enabled    false
spark.history.fs.logDirectory    hdfs://nameservice1/tmp/spark/log/
spark.yarn.historyServer.address master:18080
#spark.executor.memoryOverhead    2900
#spark.driver.memoryOverhead      2900
#spark.yarn.executor.memoryOverhead 2900
#spark.yarn.driver.memoryOverhead 2900
spark.network.timeout 2000s
spark.executor.heartbeatInterval 800s
spark.files.fetchTimeout 1000s
spark.port.maxRetries 100
spark.sql.autoBroadcastJoinThreshold 536870912
spark.sql.shuffle.partitions 1500
spark.sql.broadcastTimeout      800000ms
spark.default.parallelism 1500
spark.executor.cores 1
spark.kryoserializer.buffer.max   256m
#spark.yarn.queue                  bi
spark.executor.extraJavaOptions  -Dfile.encoding=UTF-8
spark.driver.extraJavaOptions    -Dfile.encoding=UTF-8
spark.port.maxRetries           100

报错信息：

报错信息如下：

Exception in thread "main" org.apache.spark.SparkException: Cluster deploy mode is not applicable to Spark shells.
	at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:857)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:292)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

具体为：

解决方案

使用命令：

/bin/spark-shell --master yarn --deploy-mode client --num-executors 3 --executor-memory 2G --executor-cores 2

问题解决！

方法二：

将 spark-defaults.conf文件如下：

spark.deploy.mode                cluster
spark.submit.deployMode          cluster

改为 client模式

spark.deploy.mode                client
spark.submit.deployMode          client