Word2Vec简介
Word2Vec是一个词嵌入方法,可以计算每个单词在给定的语料库环境下的分布式向量,如果两个单词的语义相近,那么词向量在向量空间中也相互接近,判断向量空间的接近程度来判断来两个单词是否相似。
首先导入Word2Vec所需要的包,并创建可以代表文档的词语序列
from pyspark.ml.feature import Word2Vec
from pyspark.sql import SparkSession
spark= SparkSession\
.builder \
.appName("dataFrame") \
.getOrCreate()
# Input data: Each row is a bag of words from a sentence or document.
documentDF = spark.createDataFrame([
("Hi I heard about Spark".split(" "), ),
("I wish Java could use case classes".split(" "), ),
("Logistic regression models are neat".split(" "), )
], ["text"])
# Learn a mapping from words to Vectors.
word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol="text", outputCol="result")
model = word2Vec.fit(documentDF)
result = model.transform(documentDF)
for row in result.collect():
text, vector = row
print("Text: [%s] => \nVector: %s\n" % (", ".join(text), str(vector)))
输出:
Text: [Hi, I, heard, about, Spark] =>
Vector: [-0.05760560743510723,-0.03687768429517746,0.053699607402086263]
Text: [I, wish, Java, could, use, case, classes] =>
Vector: [-0.06942265214664595,-0.07444838913423674,0.029864686142121042]
Text: [Logistic, regression, models, are, neat] =>
Vector: [0.025776204053545373,0.06013465970754624,-0.0191410340834409]
可以看到,文档被转变为了一个3维的特征向量,这些特征向量就可以被应用到相关的机器学习方法中。