我不会首先使用Spark,但如果你真的承诺特定的堆栈,你可以结合一堆ml变压器来获得最佳匹配。你需要Tokenizer(或split):
import org.apache.spark.ml.feature.RegexTokenizer
val tokenizer = new RegexTokenizer().setPattern("").setInputCol("text").setMinTokenLength(1).setOutputCol("tokens")
NGram(例如3克)
import org.apache.spark.ml.feature.NGram
val ngram = new NGram().setN(3).setInputCol("tokens").setOutputCol("ngrams")
Vectorizer(例如CountVectorizer或HashingTF):
import org.apache.spark.ml.feature.HashingTF
val vectorizer = new HashingTF().setInputCol("ngrams").setOutputCol("vectors")
和LSH:
import org.apache.spark.ml.feature.{MinHashLSH, MinHashLSHModel}
// Increase numHashTables in practice.
val lsh = new MinHashLSH().setInputCol("vectors").setOutputCol("lsh")
与Pipeline
import org.apache.spark.ml.Pipeline
val pipeline = new Pipeline().setStages(Array(tokenizer, ngram, vectorizer, lsh))
适合于例如数据合并:
val query = Seq("Hello there 7l | real|y like Spark!").toDF("text")
val db = Seq(
"Hello there ! I really like Spark ❤️!",
"Can anyone suggest an efficient algorithm"
).toDF("text")
val model = pipeline.fit(db)
变换两者:
val dbHashed = model.transform(db)
val queryHashed = model.transform(query)
并加入
model.stages.last.asInstanceOf[MinHashLSHModel]
.approxSimilarityJoin(dbHashed, queryHashed, 0.75).show
+--------------------+--------------------+------------------+
| datasetA| datasetB| distCol|
+--------------------+--------------------+------------------+
|[Hello there ! ...|[Hello there 7l |...|0.5106382978723405|
+--------------------+--------------------+------------------+