前言
使用CDH5.13 集成 apche spark2.4.2
使用spark-shell
命令报错。
spark配置
spark-defaults.conf文件如下:
spark.master yarn
spark.deploy.mode cluster
spark.submit.deployMode cluster
spark.eventLog.enabled true
spark.eventLog.dir hdfs://nameservice1/tmp/spark/log/
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 1g
spark.driver.maxResultSize 2g
spark.executor.memory 2g
yarn.scheduler.maximum-allocation-mb 2048m
spark.executor.instances 600
spark.executor.extraJavaOptions -XX:+PrintGCDetails
spark.shuffle.service.enabled false
spark.history.fs.logDirectory hdfs://nameservice1/tmp/spark/log/
spark.yarn.historyServer.address master:18080
#spark.executor.memoryOverhead 2900
#spark.driver.memoryOverhead 2900
#spark.yarn.executor.memoryOverhead 2900
#spark.yarn.driver.memoryOverhead 2900
spark.network.timeout 2000s
spark.executor.heartbeatInterval 800s
spark.files.fetchTimeout 1000s
spark.port.maxRetries 100
spark.sql.autoBroadcastJoinThreshold 536870912
spark.sql.shuffle.partitions 1500
spark.sql.broadcastTimeout 800000ms
spark.default.parallelism 1500
spark.executor.cores 1
spark.kryoserializer.buffer.max 256m
#spark.yarn.queue bi
spark.executor.extraJavaOptions -Dfile.encoding=UTF-8
spark.driver.extraJavaOptions -Dfile.encoding=UTF-8
spark.port.maxRetries 100
报错信息:
报错信息如下:
Exception in thread "main" org.apache.spark.SparkException: Cluster deploy mode is not applicable to Spark shells.
at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:857)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
具体为:
解决方案
使用命令:
/bin/spark-shell --master yarn --deploy-mode client --num-executors 3 --executor-memory 2G --executor-cores 2
问题解决!
方法二:
将 spark-defaults.conf文件如下:
spark.deploy.mode cluster
spark.submit.deployMode cluster
改为 client模式
spark.deploy.mode client
spark.submit.deployMode client