Spark常见异常: java.util.concurrent.TimeoutException: Futures timed out

大数据学习僧

已于 2022-05-09 14:14:06 修改

阅读量2.8k

点赞数

分类专栏： Spark 文章标签： spark java big data

于 2021-12-06 21:38:38 首次发布

本文链接：https://blog.csdn.net/yu7888/article/details/121756870

版权

Spark 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

执行spark on yarn任务时报错:

Caused by : java.util.concurrent.TimeoutException: Futures timed out after 1000s

一般这个报错发生在大表join小表broadcast join模式的时候，会将小于spark.sql.autoBroadcastJoinThreshold值（默认为10M）的表广播到其他计算节点，不走shuffle过程，但是而其中一个 DataFrame 非常大，因此发送它会消耗很多时间。
参考此网站

This happens because Spark tries to do Broadcast Hash Join and one of the DataFrames is very large, so sending it consumes much time.
You can:
Set higher spark.sql.broadcastTimeout to increase timeout - spark.conf.set(“spark.sql.broadcastTimeout”, newValueForExample36000)
persist() both DataFrames, then Spark will use Shuffle Join

所以可以:

增大spark.sql.broadcastTimeout的值;
持久化两个DataFrames;
另外，还可以考虑将BroadcastJoin禁用掉，以及增加spark.driver.memory的值。

In addition to increasing spark.sql.broadcastTimeout or persist() both DataFrames,
You may try:
1.disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1
2.increase the spark driver memory by setting spark.driver.memory to a higher value.
https://blog.csdn.net/weixin_44455388/article/details/101286428