apache spark3.0.2 on yarn cluster(HDP3.1.4) :bad substitution问题

肉装法师

已于 2022-02-10 15:14:00 修改

阅读量1.3k

点赞数

分类专栏： Spark Ambari 文章标签： apache yarn hadoop

于 2022-02-10 15:09:31 首次发布

本文链接：https://blog.csdn.net/weixin_41772761/article/details/122860648

版权

Spark 同时被 2 个专栏收录

25 篇文章 1 订阅

订阅专栏

Ambari

23 篇文章 4 订阅

订阅专栏

问题描述：

本人使用ambari2.7.4 hdp版本为 3.1.4.0-315
由于ambari spark版本低，所以搭建外部apache spark3.0.2集群
代码运行client模式正常，但运行cluster报错

/opt/soft/spark3/bin/spark-submit --master yarn --deploy-mode cluster --driver-memory 1g --num-executors 3 --total-executor-cores 2 --executor-memory 2g --queue spark --class streaming.SSSHudiETL --jars /opt/soft/hudi/hudi-0.9.0/packaging/hudi-spark-bundle/target/hudi-spark3-bundle_2.12-0.9.0.jar --packages org.apache.spark:spark-avro_2.12:3.0.2 streaming-1.0-SNAPSHOT-jar-with-dependencies.jar

报错如下

launch_container.sh: line 39: :$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/hdp/3.1.4.0-
315/hadoop/conf:/usr/hdp/3.1.4.0-315/hadoop/*:/usr/hdp/3.1.4.0-
315/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-
client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-
client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-
framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-
framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-
framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-
framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-
framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-
0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*:bad substitution
bad
......

解决方法：

在将yarn使用的包为1.9低版本替换复制到apache spark jars后
又在spark-default.xml后增加配置最后三行 -Dhdp.version=…

    
spark.driver.extraClassPath  
spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.eventLog.dir hdfs:///spark2-history/
spark.eventLog.enabled true
spark.executor.extraJavaOptions -XX:+UseNUMA
spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.extraListeners  
spark.history.fs.cleaner.enabled true
spark.history.fs.cleaner.interval 7d
spark.history.fs.cleaner.maxAge 90d
spark.history.fs.logDirectory hdfs:///spark2-history/
spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.history.store.path /var/lib/spark2/shs_db
spark.history.ui.port 18081
spark.io.compression.lz4.blockSize 128kb
spark.master yarn
spark.shuffle.file.buffer 1m
spark.shuffle.io.backLog 8192
spark.shuffle.io.serverThreads 128
spark.shuffle.unsafe.file.output.buffer 5m
spark.sql.autoBroadcastJoinThreshold 26214400
spark.sql.hive.convertMetastoreOrc true
spark.sql.hive.metastore.jars /usr/hdp/current/spark2-client/standalone-metastore/*
spark.sql.hive.metastore.version 3.0
spark.sql.orc.filterPushdown true
spark.sql.orc.impl native
spark.sql.queryExecutionListeners  
spark.sql.statistics.fallBackToHdfs true
spark.sql.streaming.streamingQueryListeners  
spark.sql.warehouse.dir /warehouse/tablespace/managed/hive
spark.unsafe.sorter.spill.reader.buffer.size 1m
spark.yarn.dist.files  
spark.yarn.historyServer.address xxx:18081
spark.yarn.queue spark
spark.driver.extraJavaOptions -Dhdp.version=3.1.4.0-315
spark.yarn.am.extraJavaOptions -Dhdp.version=3.1.4.0-315
spark.executor.extraJavaOptions -Dhdp.version=3.1.4.0-315

到此为止spark-sql spark-submit yarn-client都没有问题但是cluster还是报上述错误。

将 /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop下mapred-site.xml 配置更改

cd /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop
sed -i 's/${hdp.version}/3.1.4.0-315/g'  mapred-site.xml

在这里插入图片描述
所有${hdp.version}替换成具体版本即可

最后运行cluster成功

22/02/10 14:36:10 INFO SecurityManager: Changing view acls groups to: 
22/02/10 14:36:10 INFO SecurityManager: Changing modify acls groups to: 
22/02/10 14:36:10 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/02/10 14:36:10 INFO Client: Submitting application application_1644382793459_0178 to ResourceManager
22/02/10 14:36:10 INFO YarnClientImpl: Submitted application application_1644382793459_0178
22/02/10 14:36:11 INFO Client: Application report for application_1644382793459_0178 (state: ACCEPTED)
22/02/10 14:36:11 INFO Client: 
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: spark
	 start time: 1644474970298
	 final status: UNDEFINED
	 tracking URL: http://xxx/proxy/application_1644382793459_0178/
	 user: root
22/02/10 14:36:12 INFO Client: Application report for application_1644382793459_0178 (state: ACCEPTED)
22/02/10 14:36:13 INFO Client: Application report for application_1644382793459_0178 (state: ACCEPTED)
22/02/10 14:36:14 INFO Client: Application report for application_1644382793459_0178 (state: ACCEPTED)
22/02/10 14:36:15 INFO Client: Application report for application_1644382793459_0178 (state: ACCEPTED)
22/02/10 14:36:16 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:16 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: xxx
	 ApplicationMaster RPC port: 38113
	 queue: spark
	 start time: 1644474970298
	 final status: UNDEFINED
	 tracking URL: http://xxx:8088/proxy/application_1644382793459_0178/
	 user: root
22/02/10 14:36:17 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:18 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:19 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:20 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:21 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:22 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:23 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:24 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:25 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:26 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:27 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)
22/02/10 14:36:28 INFO Client: Application report for application_1644382793459_0178 (state: RUNNING)