基于hadoop2.7集群的Spark2.0,Sqoop1.4.6,Mahout0.12.2完全分布式安装

标签: spark2.0Sqoop1.4.6Mahout0.12.2分布式部署hadoop2.7
6769人阅读 评论(7) 收藏 举报
分类:

写在前边的话

       hadoop2.7完全分布式安装请参考:点击阅读,继任该篇博客之后,诞生了下面的这一篇博客

       基本环境:

              CentOS 6.5,Hadoop 2.7,Java 1.7

              Hive 2.0.0,Zookeeper 3.4.8, Hbase 1.2.2

       预安装

              Scala  2.11

              Spark 2.0

              Sqoop 1.4.6

              Mahout 0.12.2 

一:安装 Scala 2.11.X

      下载:点击进入下载  (我使用的是Scala 2.11.8)

    1:解压到指定目录,并重命名文件夹

            sudo tar -zxvf /home/master/下载/scala-2.11.8.tar.gz -C /opt/

            sudo mv scala-2.11.8/ /opt/scala

    2:修改环境变量

            sudo vim /etc/profile 加入如下代码:

#scala
export SCALA_HOME=/opt/scala
export PATH=$PATH:$SCALA_HOME/bin
           source /etc/profile

     3:每台机器上都部署scala

           执行 sudo scp -r /opt/scala/ slave1:/opt/scala

                   sudo scp -r /opt/scala/ slave2:/opt/scala

           分别在各个节点上修改环境变量即可

     4:运行scala

           终端直接输入scala即可

二:安装 Spark 2.0

       下载:点击进入下载 (这里建议不要安装最新版Spark,具体看评论)

     1:解压到指定目录,并重命名文件夹

            [master@master1 opt]$ sudo tar -zxvf /home/master/下载/spark-2.0.0-bin-hadoop2.7.tgz -C .

            [master@master1 opt]$ sudo mv spark-2.0.0-bin-hadoop2.7/ spark

     2:配置环境变量

            sudo vim /etc/profile ,加入

#spark home  
export SPARK_HOME=/opt/spark  
export PATH=$SPARK_HOME/bin:$PATH 

    3:配置spark-env.sh

            复制 :sudo cp spark-env.sh.template spark-env.sh

            加入以下:

export SCALA_HOME=/opt/scala
export JAVA_HOME=/opt/java
export SPARK_MASTER_IP=192.168.48.130
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

     4:将 slaves.template 拷贝到 slaves, 编辑内容为

            master1

            slave1

            slave2

      5:将spark目录拷贝到各个节点

           sudo scp -r /opt/spark/ slave1:/opt/spark

           sudo scp -r /opt/spark/ slave2:/opt/spark

           并修改各个节点的环境变量

      6:启动 spark

            启动master: sbin/start-master.sh

            启动salve: sbin/start-slaves.sh

            如遇到权限不足问题,直接给每台机器上的spark目录赋予 777 的权限即可

      7:web界面

            http://192.168.48.130:8080/

   

      8:shell 界面

               bin/spark-shell

   

三:安装  Sqoop 1.4.6

        下载:点击进入下载  

     1:解压到指定目录,并重命名

 sudo tar -zxvf /home/master/下载/sqoop-1.4.6.tar.gz -C 

 sudo mv sqoop-1.99.6/ sqoop

     2:配置环境变量

           sudo vim /etc/profile

#sqoop
export SQOOP_HOME=/opt/sqoop
export PATH = $SQOOP_HOME/bin:$PATH
        保存生效:source /etc/profile

     3:复制Mysql-jdbc 包到sqoop/lib目录下

               sudo cp /home/master/下载/mysql-connector-java-5.1.39-bin.jar /opt/sqoop/lib/

     4:修改bin/configure-sqoop文件

              此时如果没有启用hbase,zookeeper等组件,将相应的信息注释,如果启用了,就pass,直接进入下一步

      5:sqoop help 查看帮助

            

四:安装  Mahout 0.12.2

        下载:点击进入下载

       1:解压到指定目录,并重命名

               注意路径问题

[master@master1 opt]$ sudo tar -zxvf /home/master/桌面/apache-mahout-distribution-0.12.0.tar.gz -C .

[master@master1 opt]$ sudo mv apache-mahout-distribution-0.12.0/ mahout

        2:配置环境变量

                 sudo vim  /etc/profile ,加入以下内容:

<span style="font-size:14px;">#mahout home
export MAHOUT_HOME=/opt/mahout
export PATH=$MAHOUT_HOME/bin:$PATH
</span>
               保存生效:source /etc/profile

        3:启动mahout

              进入mahout安装目录,执行:bin/mahout

[master@master1 mahout]$ bin/mahout
Running on hadoop, using /opt/hadoop/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: /opt/mahout/mahout-examples-0.12.0-job.jar
An example program must be given as the first argument.
Valid program names are:
  arff.vector: : Generate Vectors from an ARFF file or directory
  baumwelch: : Baum-Welch algorithm for unsupervised HMM training
  canopy: : Canopy clustering
  cat: : Print a file or resource as the logistic regression models would see it
  cleansvd: : Cleanup and verification of SVD output
  clusterdump: : Dump cluster output to text
  clusterpp: : Groups Clustering Output In Clusters
  cmdump: : Dump confusion matrix in HTML or text formats
  cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx)
  cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally.
  describe: : Describe the fields and target variable in a data set
  evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes
  fkmeans: : Fuzzy K-means clustering
  hmmpredict: : Generate random sequence of observations by given HMM
  itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering
  kmeans: : K-means clustering
  lucene.vector: : Generate Vectors from a Lucene index
  matrixdump: : Dump matrix in CSV format
  matrixmult: : Take the product of two matrices
  parallelALS: : ALS-WR factorization of a rating matrix
  qualcluster: : Runs clustering experiments and summarizes results in a CSV
  recommendfactorized: : Compute recommendations using the factorization of a rating matrix
  recommenditembased: : Compute recommendations using item-based collaborative filtering
  regexconverter: : Convert text files on a per line basis based on regular expressions
  resplit: : Splits a set of SequenceFiles into a number of equal splits
  rowid: : Map SequenceFile<Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text>}
  rowsimilarity: : Compute the pairwise similarities of the rows of a matrix
  runAdaptiveLogistic: : Score new production data using a probably trained and validated AdaptivelogisticRegression model
  runlogistic: : Run a logistic regression model against CSV data
  seq2encoded: : Encoded Sparse Vector generation from Text sequence files
  seq2sparse: : Sparse Vector generation from Text sequence files
  seqdirectory: : Generate sequence files (of Text) from a directory
  seqdumper: : Generic Sequence File dumper
  seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives
  seqwiki: : Wikipedia xml dump to sequence file
  spectralkmeans: : Spectral k-means clustering
  split: : Split Input data into test and train sets
  splitDataset: : split a rating dataset into training and probe parts
  ssvd: : Stochastic SVD
  streamingkmeans: : Streaming k-means clustering
  svd: : Lanczos Singular Value Decomposition
  testnb: : Test the Vector-based Bayes classifier
  trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model
  trainlogistic: : Train a logistic regression using stochastic gradient descent
  trainnb: : Train the Vector-based Bayes classifier
  transpose: : Take the transpose of a matrix
  validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold-out data set
  vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors
  vectordump: : Dump vectors from a sequence file to text
  viterbi: : Viterbi decoding of hidden states from given output states sequence

五:额外补充

      1:出现 sudo: command not found时

           执行 export PATH=$PATH:/bin:/usr/bin:/usr/local/bin 即可

      2:这里我们发现除了spark的分布式安装以外要把安装包拷贝到各个节点之外,sqoop和mahout并不需要,只需要在master主机上部署即可,我的理解是sqoop只是进行数据传输的,数据可以是HDFS上的,也可以是Hive,或者Hbase上的,而本身他们已经是分布式的了,所以这里自然不需要将其拷贝到各个节点,而mahout也一样吧,只要运行在分布式的平台上即可,其所依赖的数据也是在hdfs或者hive上,故也不需要将其拷贝到各个节点

      3:那么问题来了,我们是否可以将sqoop或者mahout部署到slave节点上呢?答案是肯定的吧,因为每台机器之间是可以互相通过ssh访问的,sqoop使用时可以直接加上对应的IP地址即可,而mahout就可以直接使用了


大数据全新视频教程,博主亲自整理,点击查看

0
4

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    Thinkgamer微博
    微信公众号【码农故事多】
    个人资料
    • 访问:799119次
    • 积分:9819
    • 等级:
    • 排名:第1877名
    • 原创:212篇
    • 转载:22篇
    • 译文:2篇
    • 评论:279条
    个人简介
    姓名:Thinkgamer

    Github:https://github.com/thinkgamer

    主攻:云计算/python/数据分析

    程度:熟悉/熟悉/熟悉

    微信:gyt13342445911

    Email:thinkgamer@163.com

    工作状态:在职ing

    心灵鸡汤:只要努力,你就是下一个大牛...

    hadoop/spark/机器学习群:279807394(大神建的群,蹭个管理员)

    欢迎骚扰........
    博客专栏
    最新评论