环境描述
spark 版本: spark-2.4.0-bin-hadoop2.7
hdp 版本: 2.6.3.0-235
Spark整合Hadoop
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
HADOOP_CONF_DIR=/usr/hdp/current/hadoop-client/etc/hadoop
YARN_CONF_DIR=/usr/hdp/current/hadoop-client/etc/hadoop
提交任务时的异常解决
异常
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
YARN LOG: java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map;
命令行错误: Container exited with a non-zero exit code 1 && YARN LOG: ... bad substitution
解决问题1
原因: 复制jersey依赖,统一spark和hadoop的jersey-client和jersey-core版本
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
cp /usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar .
cp /usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar .
解决问题2
原因: hadoop依赖不统一, 替换hadoop-yarn相关依赖
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
rm -f ./hadoop-yarn*
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.3.0-235.jar .
解决问题3
原因: jsr305依赖版本不统一, 替换jsr305依赖
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
rm -f ./jsr305-1.3.9.jar
cp /usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar .
解决问题4
原因: hdp版本变量未设置
查看hdp版本
hdp-select status hadoop-client
image.png
添加变量
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/conf
cp spark-defaults.conf.template spark-defaults.conf
vim spark-defaults.conf
spark.driver.extraJavaOptions -Dhdp.version=2.6.3.0-235
spark.executor.extraJavaOptions -Dhdp.version=2.6.3.0-235
spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.3.0-235
vim java-opts
-Dhdp.version=2.6.3.0-235