mahout学习(三)--基于reuters数据的聚类分析入门

# 数据准备
$ cd /home/hadoop/data/examples/mahout
$ wget http://www.daviddlewis.com/resources/testcollections/reuters21578/reuters21578.tar.gz
$ tar -xzvf reuters21578.tar.gz


#1 运行处理reuters数据的 org.apache.lucene.benchmark.utils.ExtractReuters 类
$ mahout org.apache.lucene.benchmark.utils.ExtractReuters /home/hadoop/data/examples/mahout/reuters-extracted reuters-text

#2 将处理后的数据复制到HDFS
$ hadoop fs -mkdir mahoutdata/reuters
$ hadoop fs -copyFromLocal /home/hadoop/mahout-distribution-0.7/examples/reuters-text/ mahoutdata/reuters/

#3 生成 sequence-file
$ mahout seqdirectory -i /user/hadoop/mahoutdata/reuters/ -o /user/hadoop/mahoutdata/reuters-seqfiles -c UTF-8

#4 检查生成的 sequence-file（注意是否存在文件为空的现象）
$ mahout seqdumper -i mahoutdata/reuters-seqfiles/chunk-0 | less

#5 使用 sequence-file 生成 vector file
$ mahout seq2sparse -i mahoutdata/reuters-seqfiles -o mahoutdata/reuters-vectors -ow
# -ow → overwrite

#6 查看向量文件是否存在以下7个目录
$ hadoop fs -ls mahoutdata
$ hadoop fs -ls mahoutdata/reuters-vectors
Found 7 items
reuters-vectors/df-count
reuters-vectors/dictionary.file-0
reuters-vectors/frequency.file-0
reuters-vectors/tf-vectors
reuters-vectors/tfidf-vectors
reuters-vectors/tokenized-documents
reuters-vectors/wordcount


#7 检查: reuters-vectors/tf-vectors/part-r-00000
$ hadoop fs -ls mahoutdata/reuters-vectors/tf-vectors

#8 运行 canopy clustering 算法生成初始的中心点
$ mahout canopy -i mahoutdata/reuters-vectors/tf-vectors -o mahoutdata/reuters-canopy-centroids -dm org.apache.mahout.common.distance.CosineDistanceMeasure -t1 1500 -t2 2000

-dm → specifying the distance measure to be used while clustering (here it is cosine distance measure)

#9 运行 k-means clustering 算法
$ mahout kmeans -i mahoutdata/reuters-vectors/tfidf-vectors -c mahoutdata/reuters-canopy-centroids -o mahoutdata/reuters-kmeans-clusters -cd 0.1 -ow -x 20 -k 10

-i → input
-o → output
-c → initial centroids for k-means (not defining this parameter will trigger k-means to generate random initial centroids)
-cd → convergence delta parameter
-ow → overwrite
-x → specifying number of k-means iterations
-k → specifying number of clusters

#10 查看 k-means 聚类结果
$ mahout clusterdump -dt sequencefile -d mahoutdata/reuters-vectors/dictionary.file-* -i mahoutdata/reuters-kmeans-clusters/clusters-*-final -o clusters.txt -b 15
$ cat clusters.txt

-dt → dictionary type

-b → specifying length of each word

参考：

（1）reuters数据聚类分析的shell脚本

（2）mahout 0.7 k-means聚类