Spark/Spark-JobServer 执行SQL时任务超时

已于 2022-03-18 17:57:57 修改
阅读量2.4k 收藏
点赞数
文章标签： spark scala big data
于 2022-03-18 17:54:30 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/dymkkj/article/details/123581026
版权

1 问题，SQL执行时超时

2022/03/18 17:36:47 HttpRequest GET http://10.1.170.115:8080/v1/statement/20220318_093646_03065_awf2q/2
map[errorClass:java.util.concurrent.TimeoutException message:Futures timed out after [600 seconds] stack:java.util.concurrent.TimeoutException: Futures timed out after [600 seconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
	at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
	at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
	at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
	at akka.dispatch.BatchingExecutor$BlockableBatch.blockOn(BatchingExecutor.scala:106)
	at scala.concurrent.Await$.result(package.scala:190)
	at spark.jobserver.WebApi.spark$jobserver$WebApi$$getJobManagerForContext(WebApi.scala:812)
	at spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$94$$anonfun$apply$95.apply(WebApi.scala:723)
	at spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$94$$anonfun$apply$95.apply(WebApi.scala:713)
	at spray.routing.ApplyConverterInstances$$anon$17$$anonfun$apply$6.apply(ApplyConverterInstances.scala:60)
	at spray.routing.ApplyConverterInstances$$anon$17$$anonfun$apply$6.apply(ApplyConverterInstances.scala:59)
	at spray.routing.ConjunctionMagnet$$anon$3$$anon$4$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:38)
	at spray.routing.ConjunctionMagnet$$anon$3$$anon$4$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:37)
	at spray.routing.directives.BasicDirectives$$anon$2.happly(BasicDirectives.scala:79)
	at spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)
	at spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)
	at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)
	at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)
	at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)
	at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)
	at spray.routing.RequestContext.reject(RequestContext.scala:202)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)
	at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)
	at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)
	at spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirectives.scala:81)
	at spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirectives.scala:79)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
]
2022/03/18 17:37:33 [-] Plugin:ParseInputTablePlugin @ 0.0.1 -------------------------
2022/03/18 17:37:33 [-] Plugin:ContextSavePlugin @ 0.0.2 -------------------------
2022/03/18 17:37:33 [-] Plugin:AuditLogPlugin @ 0.0.1 -------------------------

定位

  • 可能性1，初步判断Spark在获取结果数据时，数据量过大导致超时
  • 可能性2，当Spark 进行broadcast join时，其中一个表数据过大，导致数据拉取时超过broadcast join的广播时间

解决

  • 调整参数
spark.sql.broadcastTimeout=3600
青山流水在深谷
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Spark调优-解决job任务运行超时或者慢的问题
lukabruce的博客
04-25 228
（3）如果 spark.shuffle.io.connectionTimeout 导致的连接失败，并且重试次数（spark.rpc.numRetries）也耗尽了，那么 spark.network.timeout 将决定在放弃任务之前等待多长时间。在任务运行过程中，发现有些task执行速度较慢，耗时数分钟级别，根据执行结果分析，发现是数据倾斜了，倾斜的原因可能是网络io,cpu,mem都有可能造成这个节点上的任务执行缓慢，可以去看该节点的性能监控来分析原因。（默认值是120s）（默认值是120s）
Spark任务偶现Task卡住很长时间导致Stage整体耗时长
独行的卡卡的博客
02-24 1万+
问题现象 提交大量Spark任务，概率性出现个别Task卡住一段时间，进而导致Stage整体耗时开销异常。 问题分析 采样Job836 异常Stage2249 -> 卡住Task8： 对应Executor日志： ... INFO | [Executor task launch worker-78] | Running task 8.0 in stage 2249.0 (T...
记一次Spark引擎执行Sql超时优化
星空的风fly
02-22 6368
前几天用spark引擎执行了一个较大的sql，涉及的表和数据量都不少，不同时间段执行了几次都超时，经过上网及分析，尝试解决了此问题，使用spark引擎测试几次大概都在半个小时左右完成，不再出现超时现象 一、问题现象 摘抄部分现场日志如下： 2022-02-01 13:02:12 INFO 22/02/01 13:02:12 [dag-scheduler-event-loop] INFO DAGScheduler: ShuffleMapStage 28 (run at ThreadPoolExecuto
Spark常见报错与问题解决方法
热门推荐
书忆江南的IT博客
04-21 4万+
1.org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow 原因：kryo序列化缓存空间不足。 解决方法：增加参数，--confspark.kryoserializer.buffer.max=2047m。 2.org.elasticsearch.hadoop.rest.EsHadoopNoN...
Spark shuffle(ExchangeExec)过多导致任务运行过慢甚至超时
monkeyboy_tech的博客
03-15 2419
背景以及现象 本文基于 spark 3.1.2 设置spark.driver.memory=2g 在调试spark sql任务的时候，发现有任务产生了200多个exchange，而且任务长期运行不出来。 分析 运行对应的sql（多个连续的join操作，且join的key都不一样），得到如下的物理计划（我们只截取了一部分）： 和之前的文章spark task过多导致任务运行过慢甚至超时 做法一样(对应的内存都是调优完后的镜像信息)，三部曲如下： 用jstat -gcutil查看一下对应的gc情况，如下：
spark任务结束时timeout问题
蓬莱人形
09-23 1807
先上错误图 这个错误是由一个临时导出需求产生的，任务需要读取两个大表进行join操作，所以耗费的时间也比较多，在Spark任务结束时产生了上图的异常。 这个任务其实在底层已经成功运行并输出结果了，且任务结果文件已经成功写入到HDFS中，再加上日志的警告部分提示的是ShutdownHook timeout，初步判断是Hadoop的ShutdownHookManager在任务结束时关闭钩子时执行FutureTask的get方法时产生了timeout异常。 不排除是Spark的原因，根据网上所说将spark的n
spark-jobserver：适用于Apache Spark的REST作业服务器
02-05
spark-jobserver提供了一个RESTful接口，用于提交和管理作业，jar和作业上下文。 此存储库包含完整的Spark作业服务器项目，包括单元测试和部署脚本。 它最初始于 ，但现在是主要的开发仓库。 其他有用的链接：，， ...
编译的spark-hive_2.11-2.3.0和 spark-hive-thriftserver_2.11-2.3.0.jar
06-23
spark-hive_2.11-2.3.0...spark-hive-thriftserver_2.11-2.3.0.jar log4j-2.15.0.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.25.jar curator-client-2.4.0.jar curator-framework-2.4.0.jar curator-recipes-2.4.0.jar
spark-sql_2.11-2.4.0-cdh6.1.1.jar
05-14
spark-sql_2.11-2.4.0-cdh6.1.1.jar
spark-job-server Build程序包
10-17
在这个“spark-job-server Build程序包”中，我们看到的是通过 sbt（Scala Build Tool）构建的 Spark JobServer 的特定版本，即 V0.7，它基于 Scala 2.10 版本。 Sbt 是 Scala 社区广泛使用的构建工具，它允许...
spark-3.4.1-bin-hadoop3.tgz - Spark 3.4.1 安装包(内置了Hadoop 3)
最新发布
07-19
文件名: spark-3.4.1-bin-hadoop3.tgz 这是 Apache Spark 3.4.1 版本的二进制文件，专为与 Hadoop 3 配合使用而设计。Spark 是一种快速、通用的集群计算系统，用于大规模数据处理。这个文件包含了所有必要的组件，...
SparkSql 一张大表被广播 问题排查
weixin_38148824的博客
03-02 941
SparkSql 一张大表被广播
spark job 异常问题
liuwei063608的专栏
03-04 1431
== spark job 异常问题 == === 1、java.io.IOException : Cannot run program “/usr/local/bin/python”=== yarn-client模式 sparrk-python 任务异常 java.io.IOException: Cannot run program “/usr/local/bin/python”: error=...
Akka源码分析-ActorSystem
weixin_34186950的博客
07-19 421
　　由于本人对Akka比较感兴趣，也用Akka开发了一些系统，但对Akka的源码还没有具体分析过，希望研究源码的同时写一点博客跟大家分享。有不当之处还请指正。我准备采取Debug的方式来研究Akka的运行过程，从入口开始，直至分析Akka是如何运转的。这样虽然会有点乱，但比较直接，大家凑合着看吧。 　　使用Akka首先要创建的一个对象就是ActorSys...
Spark排错与优化
weixin_30383279的博客
10-15 702
一. 运维 1. Master挂掉,standby重启也失效 Master默认使用512M内存，当集群中运行的任务特别多时，就会挂掉，原因是master会读取每个task的event log日志去生成spark ui，内存不足自然会OOM，可以在master的运行日志中看到，通过HA启动的master自然也会因为这个原因失败。 解决 增加Master的内存占用...
简单搞定Spark性能优化:常见异常解决（三）
github_36444580的博客
05-31 4950
第一部分内容链接：https://blog.csdn.net/github_36444580/article/details/117037685 die
sparksql调优之第一弹
大数据星球-浪尖
01-18 1万+
1，jvm调优这个是扯不断，理还乱。建议能加内存就加内存，没事调啥JVM，你都不了解JVM和你的任务数据。spark调优系列之内存和GC调优2，内存调优缓存表spark2.+采用：spark.catalog.cacheTable("tableName")缓存表，spark.catalog.uncacheTable("tableName")解除缓存。spark 1.+采用：采用 sqlContext
Flink任务请求一次后失败，task无限重启
Baron_ND的博客
08-04 2283
在flink上提交任务的时候，获取Oracle的数据计算后可以正常回调，但是获取hbase的数据后，只能回调一次后，task就卡住了，最后通过修改元空间 增加启动yjm和ytm搞定 jobmanage日志如下： 7:45:11.443 TKD [main-EventThread] ERROR o.a.f.s.c.o.a.c.ConnectionState - Authentication failed 17:45:20.998 TKD [flink-rest-server-netty-worker.
/export/server/spark/spark-3.2.3-bin-hadoop2.7/bin/spark-submit --master yarn predict.py")
05-13
其中，`/export/server/spark/spark-3.2.3-bin-hadoop2.7/bin/spark-submit` 是 PySpark 提交脚本的路径，`--master yarn` 表示使用 YARN 作为集群管理器，`predict.py` 是要提交的 PySpark 应用程序的文件名。...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值