10.spark on yarn

最新推荐文章于 2024-07-19 16:04:56 发布
最新推荐文章于 2024-07-19 16:04:56 发布
阅读量386 收藏 9
点赞数 8
分类专栏： spark 文章标签： spark 大数据 yarn spark on yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/2301_79691134/article/details/136236222
版权
本文详细描述了在Spark3.4.2环境中使用YARN进行部署时遇到的问题，包括错误提示和如何配置SparkHistoryServer以查看详细的日志记录。通过配置`spark-env.sh`和`spark-defaults.conf`文件，以及启动HistoryServer，解决了运行SparkPi示例时的错误并确保了日志管理。
摘要由CSDN通过智能技术生成

目录

概述

spark 版本 3.4.2, hadoop 版本 3.3.6

官方文档
在这里插入图片描述

实践

SparkPi

官方案例实践，直接执行报错。

[root@hadoop01 spark-3.4.2]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
>     --master yarn \
>     --deploy-mode cluster \
>     --driver-memory 4g \
>     --executor-memory 2g \
>     --executor-cores 1 \
>     --queue thequeue \
>     examples/jars/spark-examples*.jar \
>     10
24/02/22 15:56:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/02/22 15:56:50 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hadoop01/10.32.36.142:8032
24/02/22 15:56:51 INFO Configuration: resource-types.xml not found
24/02/22 15:56:51 INFO ResourceUtils: Unable to find 'resource-types.xml'.
24/02/22 15:56:51 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8096 MB per container)
24/02/22 15:56:51 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
24/02/22 15:56:51 INFO Client: Setting up container launch context for our AM
24/02/22 15:56:51 INFO Client: Setting up the launch environment for our AM container
24/02/22 15:56:51 INFO Client: Preparing resources for our AM container
24/02/22 15:56:51 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
24/02/22 15:56:53 INFO Client: Uploading resource file:/tmp/spark-8fe81d98-3b2f-4a03-aa5f-4665a3959e70/__spark_libs__2923631333742251680.zip -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0001/__spark_libs__2923631333742251680.zip
24/02/22 15:56:54 INFO Client: Uploading resource file:/data/hadoop/soft/spark-3.4.2/examples/jars/spark-examples_2.12-3.4.2.jar -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0001/spark-examples_2.12-3.4.2.jar
24/02/22 15:56:54 INFO Client: Uploading resource file:/tmp/spark-8fe81d98-3b2f-4a03-aa5f-4665a3959e70/__spark_conf__3172536214166465796.zip -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0001/__spark_conf__.zip
24/02/22 15:56:54 INFO SecurityManager: Changing view acls to: root
24/02/22 15:56:54 INFO SecurityManager: Changing modify acls to: root
24/02/22 15:56:54 INFO SecurityManager: Changing view acls groups to: 
24/02/22 15:56:54 INFO SecurityManager: Changing modify acls groups to: 
24/02/22 15:56:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY
24/02/22 15:56:54 INFO Client: Submitting application application_1708505130791_0001 to ResourceManager
24/02/22 15:56:55 INFO Client: Deleted staging directory hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0001
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1708505130791_0001 to YARN : Application application_1708505130791_0001 submitted by user root to unknown queue: thequeue
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:336)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:225)
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1310)
        at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1758)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/02/22 15:56:55 INFO ShutdownHookManager: Shutdown hook called
24/02/22 15:56:55 INFO ShutdownHookManager: Deleting directory /tmp/spark-8fe81d98-3b2f-4a03-aa5f-4665a3959e70
24/02/22 15:56:55 INFO ShutdownHookManager: Deleting directory /tmp/spark-33c5a540-ae1c-4f37-933c-fde2893c89a0
[root@hadoop01 spark-3.4.2]#

没有建这个队列，采取默认队列

./bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1 \
    examples/jars/spark-examples*.jar \
    10

[root@hadoop01 spark-3.4.2]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
>     --master yarn \
>     --deploy-mode cluster \
>     --driver-memory 4g \
>     --executor-memory 2g \
>     --executor-cores 1 \
>     examples/jars/spark-examples*.jar \
>     10
24/02/22 16:00:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/02/22 16:00:00 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hadoop01/10.32.36.142:8032
24/02/22 16:00:01 INFO Configuration: resource-types.xml not found
24/02/22 16:00:01 INFO ResourceUtils: Unable to find 'resource-types.xml'.
24/02/22 16:00:01 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8096 MB per container)
24/02/22 16:00:01 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
24/02/22 16:00:01 INFO Client: Setting up container launch context for our AM
24/02/22 16:00:01 INFO Client: Setting up the launch environment for our AM container
24/02/22 16:00:01 INFO Client: Preparing resources for our AM container
24/02/22 16:00:01 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
24/02/22 16:00:03 INFO Client: Uploading resource file:/tmp/spark-337dc919-faec-4ce4-8500-2a353afe9d01/__spark_libs__3444560846958053802.zip -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0002/__spark_libs__3444560846958053802.zip
24/02/22 16:00:04 INFO Client: Uploading resource file:/data/hadoop/soft/spark-3.4.2/examples/jars/spark-examples_2.12-3.4.2.jar -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0002/spark-examples_2.12-3.4.2.jar
24/02/22 16:00:04 INFO Client: Uploading resource file:/tmp/spark-337dc919-faec-4ce4-8500-2a353afe9d01/__spark_conf__9205864261457490190.zip -> hdfs://hadoop01:9000/user/root/.sparkStaging/application_1708505130791_0002/__spark_conf__.zip
24/02/22 16:00:04 INFO SecurityManager: Changing view acls to: root
24/02/22 16:00:04 INFO SecurityManager: Changing modify acls to: root
24/02/22 16:00:04 INFO SecurityManager: Changing view acls groups to: 
24/02/22 16:00:04 INFO SecurityManager: Changing modify acls groups to: 
24/02/22 16:00:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY
24/02/22 16:00:04 INFO Client: Submitting application application_1708505130791_0002 to ResourceManager
24/02/22 16:00:04 INFO YarnClientImpl: Submitted application application_1708505130791_0002
24/02/22 16:00:05 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:05 INFO Client: 
         client token: N/A
         diagnostics: [Thu Feb 22 16:00:05 +0800 2024] Scheduler has assigned a container for AM, waiting for AM container to be launched
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1708588804348
         final status: UNDEFINED
         tracking URL: http://hadoop01:8088/proxy/application_1708505130791_0002/
         user: root
24/02/22 16:00:06 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:07 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:08 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:09 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:10 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:11 INFO Client: Application report for application_1708505130791_0002 (state: ACCEPTED)
24/02/22 16:00:12 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:12 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: hadoop02
         ApplicationMaster RPC port: 41047
         queue: default
         start time: 1708588804348
         final status: UNDEFINED
         tracking URL: http://hadoop01:8088/proxy/application_1708505130791_0002/
         user: root
24/02/22 16:00:13 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:14 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:15 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:16 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:17 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:18 INFO Client: Application report for application_1708505130791_0002 (state: RUNNING)
24/02/22 16:00:19 INFO Client: Application report for application_1708505130791_0002 (state: FINISHED)
24/02/22 16:00:19 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: hadoop02
         ApplicationMaster RPC port: 41047
         queue: default
         start time: 1708588804348
         final status: SUCCEEDED
         tracking URL: http://hadoop01:8088/proxy/application_1708505130791_0002/
         user: root
24/02/22 16:00:19 INFO ShutdownHookManager: Shutdown hook called
24/02/22 16:00:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-337dc919-faec-4ce4-8500-2a353afe9d01
24/02/22 16:00:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-e095d81e-dd03-4ad7-bbb6-d84a6bd8ca62
[root@hadoop01 spark-3.4.2]#

执行如下图
在这里插入图片描述

添加日志记录

如下图，当想看详细日志时，没有
在这里插入图片描述

配置如下：

hadoop 中 yarn-site.xml 参考 hadoop一主三从安装 中的 hadoop historyserver 配置

[root@hadoop01 spark-3.4.2]# cd conf/
[root@hadoop01 conf]# ls
fairscheduler.xml.template  log4j2.properties.template  metrics.properties.template  spark-defaults.conf.template  spark-env.sh.template  workers.template
[root@hadoop01 conf]# pwd
/data/hadoop/soft/spark-3.4.2/conf
# 第一步配置 spark-env.sh
[root@hadoop01 conf]# mv spark-env.sh.template spark-env.sh
[root@hadoop01 conf]# vi spark-env.sh 

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=200 -Dspark.history.fs.logDirectory=hdfs://hadoop01:9000/spark-eventlog"

fairscheduler.xml.template  log4j2.properties.template  metrics.properties.template  spark-defaults.conf.template  spark-env.sh  workers.template
# 第二步配置 spark-defaults.conf
[root@hadoop01 conf]# mv spark-defaults.conf.template spark-defaults.conf
[root@hadoop01 conf]# vi spark-defaults.conf 
spark.master=yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoop01:9000/spark-eventlog
spark.eventLog.compress true
# 注意：在哪个节点上启动spark的historyserver进程，spark.yarn.historyServer.address的值里面就指定哪个节点的主机名信息
spark.yarn.historyServer.address hadoop01:18080
spark.history.ui.port 18080

spark.ui.retainedJobs 50
spark.ui.retainedStages 300
# 序列化
spark.serializer org.apache.spark.serializer.KryoSerializer

# 注意先建 /spark-eventlog 目录，否则执行会报错
[root@hadoop01 spark-3.4.2]# hdfs dfs -mkdir /spark-eventlog
2024-02-22 16:27:15,442 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 spark-3.4.2]# 

# 第三步 
[root@hadoop01 spark-3.4.2]# sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /data/hadoop/soft/spark-3.4.2/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-hadoop01.out
[root@hadoop01 spark-3.4.2]#

最终执行，如下
在这里插入图片描述

hadoop
yarn资源管理器

结束

spark on yarn 实践至此结束，如有疑问，欢迎评论区留言。

流月up
关注
  • 8
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
hive中执行sql后一直是ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
weixin_43230682的博客
06-04 1591
查看yarn的后台任务都是显示的ACCEPTED： 可以执行建库和建表以及select * 的操作，但是其它的操作就一直卡住了： 原因与解决： 看了网上的博客资料，先是对namenode进行格式化之后重启hive但也还是不行 看日志怀疑是没有资源可分配？ 于是又把给虚拟机的内存从2G增加到8G，并且在hive-site.xml中又增加了对调度器最小和最大资源分配的设置： <property> <name>yarn.scheduler.minimum-allocation-mb
Spark2.1.1中用各种模式运行计算圆周率的官方Demo
C/C++
04-10 1167
1使用单机local模式提交任务2使用独立的Spark集群模式提交任务3使用Spark集群 +Hadoop集群的模式提交任务        3.1用yarn-client模式执行计算程序                3.1.1操作步骤和方法                3.1.2常见错误解决                          3.1.2.1 Yarn applicationha...
错误：ACCEPTED: waiting for AM container to be allocated, launched and register with RM
qq_41672628的博客
04-18 1215
ACCEPTED: waiting for AM container to be allocated, launched and register with RM 长时间的这个，需要变更下 这里显示的localhost， 需要yarn-site.xml设置yarn.hostname
Hive3.1.2 on Spark 3.0.0报错
qq_53663722的博客
09-13 270
解决：在/opt/software/hadoop/hadoop-3.1.3/etc/hadoop目录下创建resource-types.xml文件。1、到spark的/opt/software/spark/spark-3.0.0/conf目录下修改配置文件。在/opt/software/spark/spark-3.0.0/sbin路径下启动spark。hive on spark环境下在hive中创建表成功，但是执行插入数据语句报错。3、在hive/conf/hive-site.xml中增加。
spark on yarn 的那些坑
热门推荐
跟我一起去征服
06-08 9万+
在公司6个节点的测试集群运行得好好的，结果也很正常，然后放上60个节点的预生产环境，我勒个擦，搞了我两天，主要是生产环境的那些家伙不配合，一个问题搞得拖啊拖 ，首先是安全认证问题，截取一两个有意义的吧：       1.没有认证 Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate fa
Spark on Yarn模式部署.docx
10-03
Spark on Yarn 模式部署是一种常见的 Spark 应用场景，本文将详细介绍 Spark on Yarn 模式部署的步骤和配置过程。 标题解释 Spark on Yarn 模式部署是指将 Spark 应用程序部署在 Yarn 集群上，使得 Spark 能够使用 ...
Spark实验：On Yarn模式安装部署（带答案）1
08-03
这可以通过设置两个属性`yarn.nodemanager.pmem-check-enabled`和`yarn.nodemanager.vmem-check-enabled`为`false`来实现。这个改动需要在master节点以及所有slave节点上执行，可以通过复制文件来同步配置。 4. **...
03_MapReduce 和Spark on YARN.docx
02-24
在大数据处理领域，Hadoop MapReduce 和 Apache Spark 是两种重要的计算框架，它们都在YARN（Yet Another Resource Negotiator）上运行以实现资源管理和任务调度。本文将深入探讨这两个框架以及YARN的相关概念。 ...
Spark on Yarn集群搭建手册
08-29
Spark on Yan集群搭建的详细过程，减少集群搭建的时间
【hive on spark Error】return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
01-07
当我提个任务（spark on yarn）并且看到并发及资源的分配情况正如自己先前所料的时候，我长舒了一口气，端起杯子拿上手机起身打算去接杯水消遣一下。 然而我一杯温水接完凑在嘴边，眼睛扫了一下手机刚打算痛饮一番的...
MapReduce 作业状态卡死 ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
four404的博客
05-24 6932
前言：配置好了yarn后，跑wordcount的例子，但是一直未完成。web页面查看任务状态为：ACCEPTED: waiting for AM container to be allocated, launched and register with RM. 在web页面查看其状态，如果active nodes为0，首先考虑是nodemanager没有正常启动。（下图是已经正常启动后，完成了任务的界面。如果是任务卡死，那么Apps Completed 就是0） #jps 查看当前的进程 如果没有
[YARN] 一个JDK的bug导致RM无法分配Container
张伟的专栏
09-19 4246
[YARN] 一个JDK的bug导致RM无法分配Container 一月 20, 2017 0 条评论 最近集群因为NM的OOM，然后决定把ContainerMetric给关闭了，然后采取了批量重启NM的方式，采取的步骤是先批量下线，然后在上线，后来发现集群任务越来越慢，集群的利用率越来越低 以下是集群可用内存，一直在增加 其中出现的现象是： 1.事件队列队列堆积非常严重，最高500W+...
hive执行sql后一直处在ACCEPTED: waiting for AM container to be allocated, launched and register with RM.状态
a1833255的博客
07-22 8228
hive执行sql后一直处在ACCEPTED: waiting for AM container to be allocated, launched and register with RM.状态，并且报错返回FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 查看had...
k8s部署失败常见原因
xiaoyi52的专栏
10-22 5173
1. 错误的容器镜像/非法的仓库权限 其中两个最普遍的问题是： 指定了错误的容器镜像。 使用私有镜像却不提供仓库认证信息。 这在首次使用 Kubernetes 或者绑定 CI/CD 环境时尤其棘手。 让我们看个例子。首先我们创建一个名为 fail 的 deployment，它指向一个不存在的 Docker 镜像： $ kubectl run fail --image=rosskukulinski/dne:v1.0.0 然后我们查看 Pods，可以看到有一个状态为ErrImagePull或者ImageP
Spark常见故障诊断（一）
beliefer的博客
04-01 1万+
本人维护的Spark主要运行在三个Hadoop集群上，此外还有其他一些小集群或者隐私集群。这些机器加起来有三万台左右。目前运维的Spark主要有Spark2.3和Spark1.6两个版本。用户在使用的过程中难免会发生各种各样的问题，为了对经验进行沉淀，也为了给Spark用户提供一些借鉴，这里将对各类问题如何处理进行介绍。由于故障种类繁多，所以我会开多个博客逐一进行介绍。 本文介绍一些Spark1...
Spark技术内幕：Executor分配详解
weixin_34279184的博客
10-05 605
当用户应用new SparkContext后，集群就会为在Worker上分配executor，那么这个过程是什么呢？本文以Standalone的Cluster为例，详细的阐述这个过程。序列图如下：1. SparkContext创建TaskScheduler和DAG SchedulerSparkContext是用户应用和Spark集群的交换的主要接口，用户应用一般首先要创建它。如果你使用Spar...
Spark Streaming
2301_77836489的博客
07-19 304
Spark Streaming
spark shell
最新发布
m0_74934794的博客
07-19 170
2.1.1读取linux文件系统的文件创建RDD。2.1.2读取hdfs文件系统上的文件创建RDD。--需要保证每一个worker中都有该文件。--过滤数据,保留每行中含有hello的数据。2.2使用Parallelize创建RDD。2.3从其他的RDD中创建新的RDD。--对RDD中的数据进行过滤操作。--统计RDD中的数据记录表。2.1 读取文件创建RDD。--读取RDD的第一条数据。4.实现wordcount。1.进行shell命令行。3.对RDD进行操作。--读取中的N条数据。
Spark on Yarn分布式集群搭建教程详解
本文档详细介绍了Spark On Yarn在完全分布式集群环境中的搭建过程，适用于大数据处理场景。整个搭建分为五个关键步骤： 1. **环境准备** - 软件和版本选择：文档推荐使用Java JDK 8u211、Scala 2.12.11、Zookeeper...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

流月up

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值