Mahout 安装配置及一个简单测试

原创 2016年11月24日 16:57:05

Mahout 简介

Mahout 是一个很强大的数据挖掘工具,是一个分布式机器学习算法的集合,包括:被称为Taste的分布式协同过滤的实现、分类、聚类等。Mahout最大的优点就是基于hadoop实现,把很多以前运行于单机上的算法,转化为了MapReduce模式,这样大大提升了算法可处理的数据量和处理性能。

Hadoop

http://blog.csdn.net/fenglailea/article/details/53318459
风.fox

环境

Centos7 服务器
当前最新版 0.12.2

Mahout下载地址

http://archive.apache.org/dist/mahout/
http://archive.apache.org/dist/mahout/0.12.2/

wget http://archive.apache.org/dist/mahout/0.12.2/apache-mahout-distribution-0.12.2.tar.gz
tar -zxvf apache-mahout-distribution-0.12.2.tar.gz

这里放到 Hadoop 目录里

mv apache-mahout-distribution-0.12.2 /home/hadoop/hadoop/mahout

Mahout环境变量设置

设置全局/etc/bashrc,当前用户~/.bashrc
这里使用当前用户

vim ~/.bashrc

mahout环境变量

export MAHOUT_HOME=/home/hadoop/hadoop/mahout
export MAHOUT_CONF_DIR=$MAHOUT_HOME/conf
export PATH=$MAHOUT_HOME/conf:$MAHOUT_HOME/bin:$PATH

hadoop环境变量

export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_HOME_WARN_SUPPRESS=not_null

应用环境变量

source  ~/.bashrc  #推荐
或
. ~/.bashrc

source在当前shell 里运行,点号是在子shell里运行

查询是否安装成功,

mahout

若出现一下,表示安装成功

arff.vector: : Generate Vectors from an ARFF file or directory
  baumwelch: : Baum-Welch algorithm for unsupervised HMM training
  canopy: : Canopy clustering
  cat: : Print a file or resource as the logistic regression models would see it
  cleansvd: : Cleanup and verification of SVD output
  clusterdump: : Dump cluster output to text
  clusterpp: : Groups Clustering Output In Clusters
  cmdump: : Dump confusion matrix in HTML or text formats
  cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx)
  cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally.
  describe: : Describe the fields and target variable in a data set
  evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes
  fkmeans: : Fuzzy K-means clustering
  hmmpredict: : Generate random sequence of observations by given HMM
  itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering
  kmeans: : K-means clustering
  lucene.vector: : Generate Vectors from a Lucene index
  matrixdump: : Dump matrix in CSV format
  matrixmult: : Take the product of two matrices
  parallelALS: : ALS-WR factorization of a rating matrix
  qualcluster: : Runs clustering experiments and summarizes results in a CSV
  recommendfactorized: : Compute recommendations using the factorization of a rating matrix
  recommenditembased: : Compute recommendations using item-based collaborative filtering
  regexconverter: : Convert text files on a per line basis based on regular expressions
  resplit: : Splits a set of SequenceFiles into a number of equal splits
  rowid: : Map SequenceFile<Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text>}
  rowsimilarity: : Compute the pairwise similarities of the rows of a matrix
  runAdaptiveLogistic: : Score new production data using a probably trained and validated AdaptivelogisticRegression model
  runlogistic: : Run a logistic regression model against CSV data
  seq2encoded: : Encoded Sparse Vector generation from Text sequence files
  seq2sparse: : Sparse Vector generation from Text sequence files
  seqdirectory: : Generate sequence files (of Text) from a directory
  seqdumper: : Generic Sequence File dumper
  seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives
  seqwiki: : Wikipedia xml dump to sequence file
  spectralkmeans: : Spectral k-means clustering
  split: : Split Input data into test and train sets
  splitDataset: : split a rating dataset into training and probe parts
  ssvd: : Stochastic SVD
  streamingkmeans: : Streaming k-means clustering
  svd: : Lanczos Singular Value Decomposition
  testnb: : Test the Vector-based Bayes classifier
  trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model
  trainlogistic: : Train a logistic regression using stochastic gradient descent
  trainnb: : Train the Vector-based Bayes classifier
  transpose: : Take the transpose of a matrix
  validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold-out data set
  vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors
  vectordump: : Dump vectors from a sequence file to text
  viterbi: : Viterbi decoding of hidden states from given output states sequence

Mahout 和Hadoop 集成测试

首先,hadoop 要安装完成及启动

http://blog.csdn.net/fenglailea/article/details/53318459

下载测试数据

http://archive.ics.uci.edu/ml/databases/synthetic_control/

wget http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data

hadoop 上传测试数据

hadoop fs -mkdir -p ./testdata
hadoop fs -put synthetic_control.data ./testdata

查看目录及文件

hadoop fs -ls
hadoop fs -ls ./testdata

使用Mahout中的kmeans聚类算法进行测试

mahout -core  org.apache.mahout.clustering.syntheticcontrol.kmeans.Job

XX执行完成,最后几行如下

        1.0 : [distance=55.039831561905785]: [33.67,38.675,39.742,41.989,37.291,43.975,31.909,25.878,31.08,15.858,13.95,23.097,19.983,21.692,31.579,38.57,33.376,38.843,41.936,33.534,39.195,32.897,25.343,18.523,15.089,17.771,22.614,25.313,23.687,29.01,41.995,35.712,40.872,41.669,32.156,25.162,24.98,23.705,18.413,20.975,14.906,26.171,30.165,27.818,35.083,39.514,37.851,33.967,32.338,34.977,26.589,28.079,19.597,24.669,23.098,25.685,28.215,34.94,36.91,39.749]
16/11/24 16:47:52 INFO ClusterDumper: Wrote 6 clusters
16/11/24 16:47:52 INFO MahoutDriver: Program took 22175 ms (Minutes: 0.3695833333333333) 

查看输出

hadoop fs -ls ./output
Found 15 items
-rw-r--r--   1 hadoop supergroup        194 2016-11-24 16:47 output/_policy
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusteredPoints
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-0
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-1
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-10-final
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-2
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-3
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-4
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-5
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-6
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-7
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-8
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/clusters-9
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/data
drwxr-xr-x   - hadoop supergroup          0 2016-11-24 16:47 output/random-seeds

查看数据

mahout vectordump -i ./output/data/part-m-00000

查看
http://itindex.net/detail/51681-mahout
http://blog.csdn.net/wind520/article/details/38851367

版权声明:原创文章欢迎转载,不过要记得加出处哦

mahout0.9的安装和配置(1)

Mahout是Hadoop的一种高级应用。运行Mahout需要提前安装好Hadoop。Hadoop的安装网上很多,并不复杂,这里不再讲述,接下来阐述怎么安装Mahout。 本人使用的环境是window...

Mahout的安装配置与使用

Mahout 是Apache旗下的一个机器学习和数据挖掘的分布式框架,包括聚类,分类,协同过滤,关联规则挖掘等 经典的算法。      1. 安装Maven      wget http:/...

MAHOUT入门(一)——环境变量的配置

mahout环境变量的配置 包括了jdk的配置 hadoop的配置 maven的配置 mahout的配置

如何在Linux下安装配置Apache Mahout

刚刚安装在Linux下安装配置成功了Apache Mahout,决定和大家分享下,因为我花了六七个小时才完成这样一个工作,我希望你——这篇文章的阅读者能在一个小时内愉快的安装成功。我觉得分享才是王道,...

Centos下mahout安装与配置

Centos下mahout安装与配置

Linux上安装Mahout的方法步骤

在安装Mahout之前,要先安装Hadoop,本文的前提是Hadoop已安装完成
  • Olaking
  • Olaking
  • 2014年10月07日 13:58
  • 1364

CentOS 6.7安装Mahout 0.11.2

之前已经在CentOS 6.7上安装部署Hadoop 2.7.2(http://blog.csdn.net/noob_f/article/details/53366756)、Spark 1.5.2(h...
  • Noob_f
  • Noob_f
  • 2016年12月01日 19:52
  • 368

Mahout初体验——配置+HelloWorld

最近在看机器学习方面的内容,不过一直在看理论方面未免略枯燥,正好发现了一个Apache下的开源项目——Mahout,是一个Java的机器学习算法库,遂看之。 Mahout的主页Apache Ma...

在win7下Myeclipse导入Mahout

由于想要研究mahout源码,尝试过在win7下配置maven或者在eclipse下新建maven项目,但最后都以版本不符合或者配置错误而告终。后来使用了一个简便方法: 1、https://githu...

mahout安装测试

Mahout 是 Apache Software Foundation(ASF) 旗下的一个开源项目,提供一些可扩展的机器学习领域经典算法的实现,旨在帮助开发人员更加方便快捷地创建智能应用程序。Apa...
  • wind520
  • wind520
  • 2014年08月26日 18:02
  • 7543
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:Mahout 安装配置及一个简单测试
举报原因:
原因补充:

(最多只允许输入30个字)