sparksql广播设置

最新推荐文章于 2024-05-24 00:30:00 发布

kehan_c

最新推荐文章于 2024-05-24 00:30:00 发布

阅读量4.2k

点赞数

分类专栏：基础知识文章标签： spark

本文链接：https://blog.csdn.net/kehan_c/article/details/93516774

版权

基础知识专栏收录该内容

13 篇文章 0 订阅

订阅专栏

spark sql broadcast join 配置：
–conf spark.sql.autoBroadcastJoinThreshold=31457280 \

一个比较不错的介绍广播的博文：
https://blog.csdn.net/lsshlsw/article/details/48662669
https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-joins-broadcast.html

// Force BroadcastHashJoin using SQL’s BROADCAST hint
// Supported hints: BROADCAST, BROADCASTJOIN or MAPJOIN
val qBroadcastLeft = “”"
SELECT /*+ BROADCAST (lf) */ *
FROM range(100) lf, range(1000) rt
WHERE lf.id = rt.id
“”"
scala> sql(qBroadcastLeft).explain
== Physical Plan ==
*BroadcastHashJoin [id#34L], [id#35L], Inner, BuildRight
:- *Range (0, 100, step=1, splits=8)
± BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))
± *Range (0, 1000, step=1, splits=8)

val qBroadcastRight = “”"
SELECT /*+ MAPJOIN (rt) */ *
FROM range(100) lf, range(1000) rt
WHERE lf.id = rt.id
“”"
scala> sql(qBroadcastRight).explain
== Physical Plan ==
*BroadcastHashJoin [id#42L], [id#43L], Inner, BuildRight
:- *Range (0, 100, step=1, splits=8)
± BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))
± *Range (0, 1000, step=1, splits=8)