Mahout0.8中贝叶斯分类器的使用方法

1.1 Mahout0.8中贝叶斯分类器的使用方法

在《京东大数据计算平台-Mahout0.6应用开发》文档的第三部分第一节中介绍了贝叶斯分类器对文本的分类，即Twenty Newsgroups例子；实际上还存在另一种运行方式，在0.6版本的安装目录下./examples/bin下有个脚本文件classifu-20newsgroups.sh，这个脚本中的逻辑和《京东大数据计算平台-Mahout0.6应用开发》是一致的，只不过将各个API用shell脚本封装到一起了。从0.7版本开始，Mahout移除了命令行调用的API：prepare20newsgroups，trainclassifier，testclassifier，只能通过shell脚本执行了。下面将详细介绍shell脚本的使用方法，以及其中的逻辑过程等信息。

1.1.1 环境配置

按照https://cwiki.apache.org/confluence/display/MAHOUT/Twenty+Newsgroups说明需要设置环境变量

JAVA_HOME：JAVA的安装路径

HADOOP_HOME：HADOOP的安装路径

MAHOUT_HOME：MAHOUT的安装路径

同时将HADOOP和MAHOUT的bin目录加入到path路径中。在Ubuntu（Release12.04(precise) 64-bit）上，早当前用户目录内执行gedit .profile，在打开的.profile文件中加入如下内容：

exportJAVA_HOME=/java/home

exportHADOOP_HOME=/hadoop/home

exportMAHOUT_HOME=/mahout/home

exportPATH=$PATH:$JAVA_HOME/bin:$MAHOUT_HOME/bin:$HADOOP_HOME/bin

由于从http://mirrors.hust.edu.cn/apache/mahout/0.8/直接下载了编译好的Mahout0.8mahout-distribution-0.8.tar.gz，所以不用执行编译过程。

MAHOUT_LOCAL与HADOOP_CONF_DIR

以上的连个参数是控制Mahout是在本地运行还是在Hadoop上运行的关键。

$MAHOUT_HOME/bin/mahout文件指出，只要设置MAHOUT_LOCAL的值为一个非空（not empty string）值，则不管用户有没有设置HADOOP_CONF_DIR和HADOOP_HOME这两个参数，Mahout都以本地模式运行；换句话说，如果要想Mahout运行在Hadoop上，则MAHOUT_LOCAL必须为空。

HADOOP_CONF_DIR参数指定Mahout运行Hadoop模式时使用的Hadoop配置信息，这个文件目录一般指向的是$HADOOP_HOME目录下的conf目录。

所以要想本地运行MAHOUT0.8还必须在.profile文件中设置环境变量MAHOUT_LOCAL为

export MAHOUT_LOCAL=true

如果不设置该变量将无法执行或者报错

1.1.2 通过shell执行Twenty Newsgroups例子

cd $MAHOUT_HOME

./examples/bin/classify-20newsgroups.sh

在执行脚本的过程中如果出现网络问题导致无法下载20news-bydate.tar.gz，导致后续过程无法进行，可以手动下载到执行目录（/tmp/mahout-work-${USER}，USER为计算机名）再重新执行shell脚本即可。

1.1.3 shell脚本逻辑解析

#!/bin/bash

#

# Licensed tothe Apache Software Foundation (ASF) under one or more

# contributorlicense agreements.  See the NOTICE filedistributed with

# this work foradditional information regarding copyright ownership.

# The ASFlicenses this file to You under the Apache License, Version 2.0

# (the"License"); you may not use this file except in compliance with

# theLicense.  You may obtain a copy of theLicense at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unlessrequired by applicable law or agreed to in writing, software

# distributedunder the License is distributed on an "AS IS" BASIS,

# WITHOUTWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See theLicense for the specific language governing permissions and

# limitationsunder the License.

#

 

#

# Downloads the20newsgroups dataset, trains and tests a classifier.

#

# To run:  change into the mahout directory and type:

#examples/bin/classify-20newsgroups.sh

 

if ["$1" = "--help" ] || [ "$1" = "--?" ];then

  echo "This script runs SGD and Bayesclassifiers over the classic 20 News Groups."

  exit

fi

 

SCRIPT_PATH=${0%/*}

if ["$0" != "$SCRIPT_PATH" ] && ["$SCRIPT_PATH" != "" ]; then

  cd $SCRIPT_PATH

fi

START_PATH=`pwd`

 

WORK_DIR=/tmp/mahout-work-${USER}

algorithm=(cnaivebayes naivebayes sgd clean)

if [ -n"$1" ]; then

  choice=$1

else

  echo "Please select a number to choosethe corresponding task to run"

  echo "1. ${algorithm[0]}"

  echo "2. ${algorithm[1]}"

  echo "3. ${algorithm[2]}"

  echo "4. ${algorithm[3]} -- cleans upthe work area in $WORK_DIR"

  read -p "Enter your choice : "choice

fi

 

echo "ok.You chose $choice and we'll use ${algorithm[$choice-1]}"

alg=${algorithm[$choice-1]}

 

echo"creating work directory at ${WORK_DIR}"

 

mkdir -p${WORK_DIR}

if [ ! -e${WORK_DIR}/20news-bayesinput ]; then

  if [ ! -e ${WORK_DIR}/20news-bydate ]; then

    if [ ! -f ${WORK_DIR}/20news-bydate.tar.gz]; then

      echo "Downloading20news-bydate"

      curl http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz-o ${WORK_DIR}/20news-bydate.tar.gz

    fi

    mkdir -p ${WORK_DIR}/20news-bydate

    echo "Extracting..."

    cd ${WORK_DIR}/20news-bydate && tarxzf ../20news-bydate.tar.gz && cd .. && cd ..

  fi

fi

#echo$START_PATH

cd $START_PATH

cd ../..

 

set -e

 

if ["x$alg" == "xnaivebayes" -o  "x$alg" =="xcnaivebayes" ]; then

  c=""

 

  if [ "x$alg" =="xcnaivebayes" ]; then

    c=" -c"

  fi

 

  set -x

  echo "Preparing 20newsgroups data"

  rm -rf ${WORK_DIR}/20news-all

  mkdir ${WORK_DIR}/20news-all

  cp -R ${WORK_DIR}/20news-bydate/*/*${WORK_DIR}/20news-all

#将20newsgroups数据转化为序列化格式的文件

  echo "Creating sequence files from20newsgroups data"

  ./bin/mahout seqdirectory \

    -i ${WORK_DIR}/20news-all \

    -o ${WORK_DIR}/20news-seq -ow

#将序列化格式的文本文件转化为向量

  echo "Converting sequence files tovectors"

  ./bin/mahout seq2sparse \

    -i ${WORK_DIR}/20news-seq \

    -o ${WORK_DIR}/20news-vectors  -lnorm -nv -wt tfidf

#将向量数据随机拆分成两份80-20，分别用于训练集合测试集

  echo "Creating training and holdout setwith a random 80-20 split of the generated vector dataset"

  ./bin/mahout split \

    -i ${WORK_DIR}/20news-vectors/tfidf-vectors\

    --trainingOutput${WORK_DIR}/20news-train-vectors \

    --testOutput ${WORK_DIR}/20news-test-vectors  \

    --randomSelectionPct 40 --overwrite--sequenceFiles -xm sequential

#训练贝叶斯网络

  echo "Training Naive Bayes model"

  ./bin/mahout trainnb \

    -i ${WORK_DIR}/20news-train-vectors -el \

    -o ${WORK_DIR}/model \

    -li ${WORK_DIR}/labelindex \

    -ow $c

#用训练数据作为测试集，产生的误差为训练误差

  echo "Self testing on training set"

  ./bin/mahout testnb \

    -i ${WORK_DIR}/20news-train-vectors\

    -m ${WORK_DIR}/model \

    -l ${WORK_DIR}/labelindex \

    -ow -o ${WORK_DIR}/20news-testing $c

#用测试集测试，产生的误差为测试误差

  echo "Testing on holdout set"

  ./bin/mahout testnb \

    -i ${WORK_DIR}/20news-test-vectors\

    -m ${WORK_DIR}/model \

    -l ${WORK_DIR}/labelindex \

    -ow -o ${WORK_DIR}/20news-testing $c

 

elif ["x$alg" == "xsgd" ]; then

  if [ ! -e "/tmp/news-group.model"]; then

    echo "Training on${WORK_DIR}/20news-bydate/20news-bydate-train/"

    ./bin/mahoutorg.apache.mahout.classifier.sgd.TrainNewsGroups${WORK_DIR}/20news-bydate/20news-bydate-train/

  fi

  echo "Testing on ${WORK_DIR}/20news-bydate/20news-bydate-test/with model: /tmp/news-group.model"

  ./bin/mahoutorg.apache.mahout.classifier.sgd.TestNewsGroups --input${WORK_DIR}/20news-bydate/20news-bydate-test/ --model /tmp/news-group.model

elif ["x$alg" == "xclean" ]; then

  rm -rf ${WORK_DIR}

  rm -rf /tmp/news-group.model

fi

# Remove thework directory

#