先自我介绍一下，小编浙江大学毕业，去过华为、字节跳动等大厂，目前阿里P7
深知大多数程序员，想要提升技能，往往是自己摸索成长，但自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！
因此收集整理了一份《2024年最新大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。
既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！
由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新
如果你需要这些资料，可以添加V获取：vip204888 （备注大数据）
正文
tryInjectRuntimeFilter使用核心的处理流程，尝试应用Runtime Filter，整体代码如下
private def tryInjectRuntimeFilter(plan: LogicalPlan): LogicalPlan = {
var filterCounter = 0
val numFilterThreshold = conf.getConf(SQLConf.RUNTIME\_FILTER\_NUMBER\_THRESHOLD)
plan transformUp {
case join @ ExtractEquiJoinKeys(joinType, leftKeys, rightKeys, _, _, left, right, hint) =>
var newLeft = left
var newRight = right
(leftKeys, rightKeys).zipped.foreach((l, r) => {
// Check if:
// 1. There is already a DPP filter on the key
// 2. There is already a runtime filter (Bloom filter or IN subquery) on the key
// 3. The keys are simple cheap expressions
if (filterCounter < numFilterThreshold &&
!hasDynamicPruningSubquery(left, right, l, r) &&
!hasRuntimeFilter(newLeft, newRight, l, r) &&
isSimpleExpression(l) && isSimpleExpression(r)) {
val oldLeft = newLeft
val oldRight = newRight
if (canPruneLeft(joinType) && filteringHasBenefit(left, right, l, hint)) {
newLeft = injectFilter(l, newLeft, r, right)
}
// Did we actually inject on the left? If not, try on the right
if (newLeft.fastEquals(oldLeft) && canPruneRight(joinType) &&
filteringHasBenefit(right, left, r, hint)) {
newRight = injectFilter(r, newRight, l, left)
}
if (!newLeft.fastEquals(oldLeft) || !newRight.fastEquals(oldRight)) {
filterCounter = filterCounter + 1
}
}
})
join.withNewChildren(Seq(newLeft, newRight))
}
}
过程中有很多的条件判断，应用Runtime Filter的基本条件：
- 插入的Runtime Filter没超过阈值（默认10）
- 等值条件的Key上不能有DPP、Runtime Filter
- 等值条件的Key是一个简单表达式（即没有套上UDF等）
之后根据条件，选择将Runtime Filter应用到左子树还是右子树，条件为
- Join类型支持下推（比如RightOuter只能用于左子树）
- Application端支持通过joins、aggregates、windows下推过滤条件
- Creation端有过滤条件
- 当前join是shuffle join或者是一个子结构中包含shuffle的broadcast join
- Application端的扫描数据大于阈值（默认10G）
提到的两个阈值的配置项
val RUNTIME\_FILTER\_NUMBER\_THRESHOLD =
buildConf("spark.sql.optimizer.runtimeFilter.number.threshold")
.doc("The total number of injected runtime filters (non-DPP) for a single " +
"query. This is to prevent driver OOMs with too many Bloom filters.")
.version("3.3.0")
.intConf
.checkValue(threshold => threshold >= 0, "The threshold should be >= 0")
.createWithDefault(10)
val RUNTIME\_BLOOM\_FILTER\_APPLICATION\_SIDE\_SCAN\_SIZE\_THRESHOLD =
buildConf("spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold")
.doc("Byte size threshold of the Bloom filter application side plan's aggregated scan " +
"size. Aggregated scan byte size of the Bloom filter application side needs to be over " +
"this value to inject a bloom filter.")
.version("3.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10GB")
2.2 injectFilter
injectFilter是核心进行Runtime Filter规则应用的地方，在此处，bloomFilter和Semi Join是互斥的，只能有一个执行
if (conf.runtimeFilterBloomFilterEnabled) {
injectBloomFilter(
filterApplicationSideExp,
filterApplicationSidePlan,
filterCreationSideExp,
filterCreationSidePlan
)
} else {
injectInSubqueryFilter(
filterApplicationSideExp,
filterApplicationSidePlan,
filterCreationSideExp,
filterCreationSidePlan
)
2.3 injectBloomFilter
2.3.1 执行条件
首先进行一个判断，在Creation端的数据不能大于阈值（Creation端数据量大会导致bloomFilter的误判率高，最终过滤效果差）
// Skip if the filter creation side is too big
if (filterCreationSidePlan.stats.sizeInBytes > conf.runtimeFilterCreationSideThreshold) {
return filterApplicationSidePlan
}
阈值配置默认10M
val RUNTIME\_BLOOM\_FILTER\_CREATION\_SIDE\_THRESHOLD =
buildConf("spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold")
.doc("Size threshold of the bloom filter creation side plan. Estimated size needs to be " +
"under this value to try to inject bloom filter.")
.version("3.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10MB")
Creation端的数据是一个预估数据，是LogicalPlan中的属性LogicalPlanStats获取的，分是否开启CBO，具体获取方式待研究
def stats: Statistics = statsCache.getOrElse {
if (conf.cboEnabled) {
statsCache = Option(BasicStatsPlanVisitor.visit(self))
} else {
statsCache = Option(SizeInBytesOnlyStatsPlanVisitor.visit(self))
}
statsCache.get
}
2.3.2 创建Creation端的聚合
就是创建一个bloomFilter的聚合函数BloomFilterAggregate，是AggregateFunction的子类，属于Expression。根据统计信息中是否存在行数，会传入不同的参数
val rowCount = filterCreationSidePlan.stats.rowCount
val bloomFilterAgg =
if (rowCount.isDefined && rowCount.get.longValue > 0L) {
new BloomFilterAggregate(new XxHash64(Seq(filterCreationSideExp)), rowCount.get.longValue)
} else {
new BloomFilterAggregate(new XxHash64(Seq(filterCreationSideExp)))
}
2.3.3 创建Application端的过滤条件
根据1.3中的描述，此处就是把上节中Creation端创建的bloomFilter过滤条件构建成Application端的条件
Alias就是一个别名的效果；ColumnPruning就是进行列裁剪，后续不需要的列不读取；ConstantFolding就是进行常量折叠；ScalarSubquery是标量子查询，标量子查询的查询结果是一行一列的值（单一值）
BloomFilterMightContain就是一个内部标量函数，检查数据是否由bloomFilter包含，继承自Predicate，返回boolean值
val alias = Alias(bloomFilterAgg.toAggregateExpression(), "bloomFilter")()
val aggregate =
ConstantFolding(ColumnPruning(Aggregate(Nil, Seq(alias), filterCreationSidePlan)))
val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
val filter = BloomFilterMightContain(bloomFilterSubquery,
new XxHash64(Seq(filterApplicationSideExp)))
最终结果是在原Application端的计划树上加一个filter，如下就是最终的返回结果
Filter(filter, filterApplicationSidePlan)
2.4 injectInSubqueryFilter
injectInSubqueryFilter整体流程与injectBloomFilter差不多，差异应该是在Application端生成的过滤条件变成in
val actualFilterKeyExpr = mayWrapWithHash(filterCreationSideExp)
val alias = Alias(actualFilterKeyExpr, actualFilterKeyExpr.toString)()
val aggregate =
ColumnPruning(Aggregate(Seq(filterCreationSideExp), Seq(alias), filterCreationSidePlan))
if (!canBroadcastBySize(aggregate, conf)) {
// Skip the InSubquery filter if the size of `aggregate` is beyond broadcast join threshold,
// i.e., the semi-join will be a shuffled join, which is not worthwhile.
return filterApplicationSidePlan
}
val filter = InSubquery(Seq(mayWrapWithHash(filterApplicationSideExp)),
ListQuery(aggregate, childOutputs = aggregate.output))
Filter(filter, filterApplicationSidePlan)
这里有一个小优化就是mayWrapWithHash，当数据类型的大小超过int时，就是把数据转为hash
// Wraps `expr` with a hash function if its byte size is larger than an integer.
private def mayWrapWithHash(expr: Expression): Expression = {
if (expr.dataType.defaultSize > IntegerType.defaultSize) {
new Murmur3Hash(Seq(expr))
} else {
expr
}
}
3 BloomFilterAggregate
类有三个核心参数：
- child：子表达式，就是InjectRuntimeFilter里传的XxHash64，目前看起来数据先经过XxHash64处理成long再放入BloomFilter
- estimatedNumItemsExpression：估计的数据量，如果InjectRuntimeFilter没拿到统计信息，就用配置的默认值
- numBitsExpression：要使用的bit数
case class BloomFilterAggregate(
child: Expression,
estimatedNumItemsExpression: Expression,
numBitsExpression: Expression,
estimatedNumItemsExpression和numBitsExpression对应的配置如下
val RUNTIME\_BLOOM\_FILTER\_EXPECTED\_NUM\_ITEMS =
buildConf("spark.sql.optimizer.runtime.bloomFilter.expectedNumItems")
.doc("The default number of expected items for the runtime bloomfilter")
.version("3.3.0")
.longConf
.createWithDefault(1000000L)
val RUNTIME\_BLOOM\_FILTER\_NUM\_BITS =
buildConf("spark.sql.optimizer.runtime.bloomFilter.numBits")
.doc("The default number of bits to use for the runtime bloom filter")
.version("3.3.0")
.longConf
.createWithDefault(8388608L)
BloomFilter用的是Spark自己实现的一个类BloomFilterImpl，BloomFilterAggregate的createAggregationBuffer接口中创建
override def createAggregationBuffer(): BloomFilter = {
BloomFilter.create(estimatedNumItems, numBits)
}
参数就是前面的estimatedNumItemsExpression和numBitsExpression，是懒加载的参数（应该在处理过程会被改变，所以实际跟前面的值之间还加了一层与默认值的比较赋值）
// Mark as lazy so that `estimatedNumItems` is not evaluated during tree transformation.
private lazy val estimatedNumItems: Long =
Math.min(estimatedNumItemsExpression.eval().asInstanceOf[Number].longValue,
SQLConf.get.getConf(RUNTIME\_BLOOM\_FILTER\_MAX\_NUM\_ITEMS))
处理数据的接口应该是update，把数据用XxHash64处理后加入BloomFilter
override def update(buffer: BloomFilter, inputRow: InternalRow): BloomFilter = {
val value = child.eval(inputRow)
// Ignore null values.
if (value == null) {
return buffer
}
buffer.putLong(value.asInstanceOf[Long])
buffer
}
对象BloomFilterAggregate有对应的序列化和反序列化接口
object BloomFilterAggregate {
final def serialize(obj: BloomFilter): Array[Byte] = {
// BloomFilterImpl.writeTo() writes 2 integers (version number and num hash functions), hence
// the +8
val size = (obj.bitSize() / 8) + 8
require(size <= Integer.MAX\_VALUE, s"actual number of bits is too large $size")
val out = new ByteArrayOutputStream(size.intValue())
obj.writeTo(out)
out.close()
out.toByteArray
}
final def deserialize(bytes: Array[Byte]): BloomFilter = {
val in = new ByteArrayInputStream(bytes)
val bloomFilter = BloomFilter.readFrom(in)
in.close()
bloomFilter
}
}
4 BloomFilterMightContain
有两个参数
- bloomFilterExpression：是上节BloomFilter的二进制数据
- valueExpression：应该跟上节的child一致，对输入数据做处理的表达式，XxHash64
网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。
需要这份系统化的资料的朋友，可以添加V获取：vip204888 （备注大数据）
一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！
Contain
有两个参数
- bloomFilterExpression：是上节BloomFilter的二进制数据
- valueExpression：应该跟上节的child一致，对输入数据做处理的表达式，XxHash64
网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。
需要这份系统化的资料的朋友，可以添加V获取：vip204888 （备注大数据）
[外链图片转存中…(img-EoDXGnSv-1713153091791)]
一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！
693
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
