TFIDF将文本向量化,用于后期的特征使用
1.创建DF
// $example on$ val sentenceData = sqlContext.createDataFrame(Seq( (0, "Hi I heard about Spark"), (0, "I wish Java could use case classes"), (1, "Logistic regression models are neat") )).toDF("label", "sentence")
JSON格式:
{"label":0,"sentence":"I wish Java could use case classes"}
{"label":1,"sentence":"Logistic regression models are neat"}
{"label":0,"sentence":"Hi I heard about Spark"}
2.由于我们需要处理的是term词条,所以先tokenizer
val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words") val wordsData = tokenizer.transform(sentenceData)
3.接着求TF即词条的频率
val hashingTF = new HashingTF() .setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20) val featurizedData = hashingTF.transform(wordsData)
格式:
scala> featurizedData.select("rawFeatures").show
+--------------------+
| rawFeatures|
+--------------------+
|(20,[5,6,9],[2.0,...|
|(20,[3,5,12,14,18...|
|(20,[5,12,14,18],...|
+--------------------+
格式2:
scala> res36.take(5).foreach(println)
[(20,[5,6,9],[2.0,1.0,2.0])]
[(20,[3,5,12,14,18],[2.0,2.0,1.0,1.0,1.0])]
[(20,[5,12,14,18],[1.0,2.0,1.0,1.0])]
4.获取iDF
val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features") val idfModel = idf.fit(featurizedData)
5.得到TF-iDF
val rescaledData = idfModel.transform(featurizedData) rescaledData.select("features", "label").take(3).foreach(println)scala>rescaledData.select("features", "label").take(3).foreach(println) [(20,[5,6,9],[0.0,0.6931471805599453,1.3862943611198906]),0] [(20,[3,5,12,14,18],[1.3862943611198906,0.0,0.28768207245178085,0.28768207245178085,0.28768207245178085]),0] [(20,[5,12,14,18],[0.0,0.5753641449035617,0.28768207245178085,0.28768207245178085]),1]