sparl ML使用TFIDF

TFIDF将文本向量化,用于后期的特征使用

1.创建DF

// $example on$
val sentenceData = sqlContext.createDataFrame(Seq(
  (0, "Hi I heard about Spark"),
  (0, "I wish Java could use case classes"),
  (1, "Logistic regression models are neat")
)).toDF("label", "sentence")

 

JSON格式:

{"label":0,"sentence":"I wish Java could use case classes"}
{"label":1,"sentence":"Logistic regression models are neat"}
{"label":0,"sentence":"Hi I heard about Spark"}

 

2.由于我们需要处理的是term词条,所以先tokenizer

val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")
val wordsData = tokenizer.transform(sentenceData)

3.接着求TF即词条的频率

val hashingTF = new HashingTF()
  .setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)
val featurizedData = hashingTF.transform(wordsData)

格式:

scala> featurizedData.select("rawFeatures").show
+--------------------+
|         rawFeatures|
+--------------------+
|(20,[5,6,9],[2.0,...|
|(20,[3,5,12,14,18...|
|(20,[5,12,14,18],...|
+--------------------+

格式2:
scala> res36.take(5).foreach(println)
[(20,[5,6,9],[2.0,1.0,2.0])]
[(20,[3,5,12,14,18],[2.0,2.0,1.0,1.0,1.0])]
[(20,[5,12,14,18],[1.0,2.0,1.0,1.0])]

4.获取iDF

val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")
val idfModel = idf.fit(featurizedData)

5.得到TF-iDF

val rescaledData = idfModel.transform(featurizedData)
rescaledData.select("features", "label").take(3).foreach(println)
 
scala>rescaledData.select("features", "label").take(3).foreach(println)
[(20,[5,6,9],[0.0,0.6931471805599453,1.3862943611198906]),0]
[(20,[3,5,12,14,18],[1.3862943611198906,0.0,0.28768207245178085,0.28768207245178085,0.28768207245178085]),0]
[(20,[5,12,14,18],[0.0,0.5753641449035617,0.28768207245178085,0.28768207245178085]),1]