org.apache.spark.SparkException: Exception thrown in awaitResult （Spark报错）

最新推荐文章于 2022-05-25 12:18:48 发布
最新推荐文章于 2022-05-25 12:18:48 发布
阅读量2.7w 收藏 9
点赞数 8
分类专栏： Spark 文章标签： Exception thrown in awaitResult
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq403580298/article/details/83210683
版权 
WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node1:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96)
        at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node1/192.168.25.121:7077
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused: node1/192.168.25.121:7077
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        ... 1 more
18/10/18 22:17:32 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node1:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96)
        at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node1/192.168.25.121:7077
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused: node1/192.168.25.121:7077
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        ... 1 more
18/10/18 22:17:52 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node1:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96)
        at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to connect to node1/192.168.25.121:7077
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
        ... 4 more

出现原因：
未启动Spark。需要先启动Spark:

cd /usr/local/spark/sbin/
./start-all.sh
hxhh
关注
专栏目录
【Spark】Spark cache 报错 Exception thrown in awaitResult
九师兄
05-13 2339
写了一段代码，如下，但是运行的时候，时不时报错错误如下java.lang.OutOfMemoryError: Java heap space 错误，这通常意味着在 Spark 应用程序执行过程中，JVM 的堆内存不足以容纳所有数据，导致了内存溢出错误。你调用了 cache() 方法试图缓存 DataFrame 或 RDD 的结果，但是由于数据集的大小超出了分配给 Spark Executor 的内存，导致了内存溢出错误。
六（1）、spark遇到的问题
Angular_need的博客
04-12 4328
1、意外退出spark-shell，而不是quit，然后再输入spark-shell命令的时候，报错： 19/04/11 13:42:32 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. jps查看，SparkSubmit在，查看4040端口也是被SparkSubmit占用...
spark报org.apache.spark.SparkException: Exception thrown in awaitResult异常
森屿旅人996的博客
07-13 8774
运行spark程序时，报错如下 20/07/13 16:09:39 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://DESKTOP-27NPJ0G:4040 20/07/13 16:09:39 INFO SparkContext: Added JAR D:\Software\java\WorkSpace\hrbuspark\target\SparkWordCount.jar at spark://DESKTOP-27NPJ0G:53
sparksql Exception thrown in awaitResult
liu82327114的博客
01-21 6382
Caused by: java.sql.SQLException: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:279) at com.lenovo.lps.farseer.priest2.ext.SparkExecDao.executeOneSql(SparkExecDao.java.
小记-bug解决:Exception thrown in awaitResult/Connection reset by peer
xinji
07-03 9130
org.apache.spark.SparkException: Exception thrown in awaitResult "time":"2020-07-02 11:10:15","logtype":"ERROR","loginfo":"Executor self-exiting due to : Driver masterba17fc94-f00c-477a-b1d4-4c165c7893d9cupid-11-194-251-15:45755 disassociated! Shutting dow
spark报错处理
weixin_30216561的博客
10-27 2604
Spark报错处理 1、问题：org.apache.spark.SparkException: Exception thrown in awaitResult 分析：出现这个情况的原因是spark启动的时候设置的是hostname启动的，导致访问的时候DNS不能解析主机名导致。 问题解决： 第一种方法：确保URL是spark://服务器ip:7077，而不是spark://hostname...
Spark各种问题集锦[持续更新]
strongyoung的专栏
08-13 1万+
1、Initial job has not accepted any resources 16/08/13 17:05:42 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 16/08/13 17:05:57 WARN TaskSchedulerImpl: Initial job has not accepted any re
Spark启动失败：slave节点无法连接到master
机器学习，大数据
04-20 4971
1. 问题描述 启动spark时报错，使用start_all.sh或在slave节点上start_slave.sh都报相同的错。 报错信息如下： 19/04/20 04:35:49 INFO Utils: Successfully started service 'sparkWorker' on port 45265. 19/04/20 04:35:49 INFO Worker: Starti...
idea中Exception in thread “main“ java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.Native
qq_56572867的博客
04-02 807
这个困扰了我三天的问题，之前已经不想解决了，直到再一次遇到，决定彻底干它，之前也在网上搜了一些解决它的办法，什么改变环境还有加入NativeIO，都没有用，不过最后还是使用了NativeIO的方法解决了，之前一直没有用的原因是，没有把文件放对位置 正确的位置应该是放在java包下，和项目包并行，而且必须是这个包名org.apache.hadoop.io.nativeio和NativeIO这个类名，不然不能覆盖原来的这个类 下面附上NativeIO的源码，有的包可能要自己引一下，细节上应该不用说了
关于Spark报错不能连接到Server的解决办法（Failed to connect to master master_hostname:7077）
热门推荐
ybdesire的专栏
04-24 7万+
如何彻底解决 Failed to connect to master master_hostname:7077
Spark IDEA本地提交到集群运行job
江湖峰哥的博客
04-03 538
场景:运行IDEA本地测试代码,实现提交job到本地虚拟机Spark集群环境运行 前提:本地宿主机和虚拟机网路互通 def main(args: Array[String]): Unit = { val sparkSession = SparkSession.builder .master("spark://hostname:7077") .appName("countBy...
StandaloneAppClient$ClientEndpoint: Failed to connect to master master:7077
qq_33465047的博客
09-24 5241
出现这种问题时， 21/09/09 09:38:31 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master master:7077 org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
idea链接不上spark主机模式
qq_39518915的博客
08-23 767
报错:Exception thrown in awaitResult: 原因：版本问题 解决办法：更改 <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version&...
spark任务shell运行_java调用shell脚本执行spark任务错误
weixin_42405503的博客
01-30 604
java调用shell脚本执行spark任务使用java通过 Runtime.getRuntime().exec(); 调用shell 脚本，在脚本中执行 spark 任务会报错，如下：19/05/15 15:46:47 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.9.2:7077org....
Spark程序运行常见错误解决方法以及优化
AcceptedLin的博客
08-19 4385
Spark程序运行常见错误解决方法及优化 一.org.apache.spark.shuffle.FetchFailedException 1.问题描述 这种问题一般发生在有大量shuffle操作的时候,task不断的failed,然后又重执行，一直循环下去，非常的耗时。 2.报错提示 (1) missing output loc...
spark 提交任务到yarn 时报错 Caused by: java.sql.SQLException: No suitable driver
剑雨江湖
05-25 974
org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:227) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:471) at org.apache.spark.dep.
flume发送端报错org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: 140.210.65.42, port: 56660 }: Exception thrown from remote handler
最新发布
06-06
这个错误提示表明，在使用Flume发送数据时，NettyAvroRpcClient连接远程主机（IP地址为140.210.65.42，端口号为56660）时，出现了异常。 可能的原因包括： 1. 网络连接问题：检查网络连接是否正常，确保Flume可以...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值