报错：bad substitution ……Container exited with a non-zero exit code 1

最新推荐文章于 2024-07-23 21:43:17 发布

liaoyanyunde

最新推荐文章于 2024-07-23 21:43:17 发布

阅读量4.4k

点赞数

本文链接：https://blog.csdn.net/liaoyanyunde/article/details/83022487

版权

在执行Spark任务时遇到了'bad substitution'错误，问题源于缺少在conf/spark-defaults.conf中配置spark.yarn.am.extraJavaOptions参数。此参数仅在yarn client模式下用于YARN Application Master。若使用yarn cluster模式，应配置spark.driver.extraJavaOptions。为避免设置最大堆大小(-Xmx)，可以配置-Dhdp.version指定HDP版本，例如：spark.driver.extraJavaOptions -Dhdp.version=2.4.2.0-258和spark.yarn.am.extraJavaOptions -Dhdp.version=2.4.2.0-258。

摘要由CSDN通过智能技术生成

执行Spark任务报错

18/10/10 19:42:57 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1538070068357_0019_02_000004 on host: rzx09. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1538070068357_0019_02_000004
Exit code: 1
Exception message: /u08/hadoop/yarn/local/usercache/root/appcache/application_1538070068357_0019/container_1538070068357_0019_02_000004/launch_container.sh: line 14: carbondata.tar.gz/carbonlib/*:$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*