参照
https://cwiki.apache.org/confluence/display/MAHOUT/Twenty+Newsgroups的步骤,其中关于hadoop与mahout的配置启动之类的以前的文章都记录过,此处直接在mahout-0.4上运行例子过程(当然也在mahot-0.3上进行了测试,一切正常)。
首先,下载数据集20news-bydate.tar.gz,在$MAHOUT_HOME下的examples/bin/下建立work文件夹(mkdir /home/hadoop/mahout-distribution-0.4/examples/bin/work/)将下载的数据集解压tar 20news-bydate.tar.gz,之后放到work文件夹下。
其次,产生Input数据集,即对训练数据集进行预处理,数据准备阶段,将各类中的数据进行分词处理,去掉标点及副词等,同时将各类中的文件读入到一个大文件中,使得每类最后只有一个文件包含起初所有的文件,mahout下处理的文件必须是SequenceFile格式的,还需要把txtfile转换成sequenceFile。等处理完看结果时就明白了,命令为$MAHOUT_HOME/bin/./mahout org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups \
-p /home/hadoop/mahout-distribution-0.4/examples/bin/work/20news-bydate-train \
-o /home/hadoop/mahout-distribution-0.4/examples/bin/work/bayes-train-input \
-a org.apache.mahout.vectorizer.DefaultAnalyzer \
-c UTF-8
完毕后将work下的bayes-train-input放到hadoop的分布式文件系统上的 20news-input,在hadoop/hadoop-0.20.2/bin输入命令hadoop dfs -put /home/hadoop/mahout-distribution-0.4/examples/bin/work/bayes-train-input 20news-input
第三,用处理好的训练数据集进行训练得出分类模型即中间结果,模型保存在分布式文件系统上,在mahout的目录下输入命令$MAHOUT_HOME/bin/ ./mahout trainclassifier \
-i 20news-input \
-o newsmodel \
-type bayes \
-ng 3 \
-source hdfs
当然可以查看newsmodel里的内容时,先查看其里面都有什么,命令:hadoop fs -lsr /user/hadoop/newsmodel
将其导入到本地txt格式,进行查看,例如命令:./mahout seqdumper -s /user/hadoop/newsmodel/trainer-tfIdf//trainer-tfIdf/part-00000 -o /home/hadoop/out/part-1
最后,用模型进行测试,输入命令$MAHOUT_HOME/bin/ ./mahout testclassifier \
-m newsmodel \
-d 20news-input \
-type bayes \
-ng 3 \
-source hdfs \
-method mapreduce
这里说明下自己理解的参数ng的设置,ng就是作为属性的单词个数,默认为1,就是单个单词作为属性,这里设置为3,但是训练和测试时此参数的值要设置一样。
我运行测试时抛出了异常,如下:
[hadoop@sc706-26 bin]$ ./mahout testclassifier -m newsmodel -d 20news-input -type bayes -ng 3 -source hdfs -method mapreduce
Running on hadoop, using HADOOP_HOME=/home/hadoop/hadoop-0.20.2
HADOOP_CONF_DIR=/home/hadoop/hadoop-0.20.2/conf
10/12/04 11:47:44 INFO common.HadoopUtil: Deleting 20news-input-output
10/12/04 11:47:44 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/12/04 11:47:45 INFO mapred.FileInputFormat: Total input paths to process : 20
10/12/04 11:47:46 INFO mapred.JobClient: Running job: job_201012040906_0022
10/12/04 11:47:47 INFO mapred.JobClient: map 0% reduce 0%
10/12/04 11:52:13 INFO mapred.JobClient: Task Id : attempt_201012040906_0022_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 10 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:39)
at java.nio.CharBuffer.allocate(CharBuffer.java:312)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:760)
at org.apache.hadoop.io.Text.decode(Text.java:350)
at org.apache.hadoop.io.Text.decode(Text.java:327)
at org.apache.hadoop.io.Text.toString(Text.java:254)
at org.apache.mahout.common.StringTuple.readFields(StringTuple.java:143)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1836)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at org.apache.mahout.classifier.bayes.io.SequenceFileModelReader.loadFeatureWeights(SequenceFileModelReader.java:99)
at org.apache.mahout.classifier.bayes.io.SequenceFileModelReader.loadModel(SequenceFileModelReader.java:52)
at org.apache.mahout.classifier.bayes.datastore.InMemoryBayesDatastore.initialize(InMemoryBayesDatastore.java:80)
at org.apache.mahout.classifier.bayes.model.ClassifierContext.initialize(ClassifierContext.java:47)
at org.apache.mahout.classifier.bayes.mapreduce.bayes.BayesClassifierMapper.configure(BayesClassifierMapper.java:135)
... 18 more
出错原因:java.lang.OutOfMemoryError: Java heap space内存溢出
解决方法:在hadoop-0.20.2/conf/mapred-site.xml里设置一下hadoop的mapred.child.java.opts,将其设定为你计算机内存的一半就可以啦,(linux查看内存的命令是:free -m)我的是1G的,设成512M,<name>mapred.child.java.opts</name>
<value>-Xmx512M</value>,没设之前查看日志我的默认是200M。再有种方法是换个大点的内存,设置完后测试正常,在终端直接就可以看见结果,如下图是我测试时的结果:
当然我们可以用web接口方式查看job运行时的情况有无异常等
转自:http://blog.163.com/jiayouweijiewj@126/blog/static/171232177201011411534050/