[spark-hash学习]minhash算法实现细节

最新推荐文章于 2024-09-05 09:15:40 发布

皮卡圈快使用十万伏特

最新推荐文章于 2024-09-05 09:15:40 发布

阅读量4.6k

点赞数

分类专栏： spark 文章标签： spark-hash

本文链接：https://blog.csdn.net/hu_oo/article/details/61616538

版权

本文详细介绍了Spark-Hash库中Minhash算法的实现步骤，包括多次minhash计算，将签名分块，按顺序排列并进行哈希分组，以及可选的过滤操作，以提高相似向量的检测效率。

摘要由CSDN通过智能技术生成

Implementation of LSH follows the rough steps

minhash each vector some number of times. The number of times to hash is an input parameter. The hashing function is defined in com.invincea.spark.hash.Hasher. Essentially each element of the input vector is hashed and the minimum hash value for the vector is returned. Minhashing produces a set of signatures for each vector.
Chop up each vector's minhash signatures into bands where each band contains an equal number of signatures. Bands with a greater number of signatures will produce clusters withgreater similarity. A greater number of bands will increase the probabilty that similar vector signatures hash to the same value.
Order each of the vector bands such that for each band the vector's data for that band are grouped tog