编译mahout使之适合hadoop2

最新推荐文章于 2024-07-23 10:48:19 发布

weixin_33682790

最新推荐文章于 2024-07-23 10:48:19 发布

阅读量78

点赞数

文章标签：大数据 java python

原文链接：https://my.oschina.net/goudingcheng/blog/1560744

版权

2019独角兽企业重金招聘Python工程师标准>>>

在mahout父类maven工程中，将pom.xml的<hadoop.version>1.2.1</hadoop.version>修改为<hadoop.version>2.6.0</hadoop.version>

去掉hadoop-core这个依赖替换成hadoop-hdfs

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>

</dependency>

一个完整的hadoop程序依赖应该包括：hadoop-core(1.x版本)或者hadoop-hdfs(2.x版本)，hadoop-common,hadoop-mapreduce-client-core,>hadoop-mapreduce-client-common.

替换之后会发现报错，错误信息是jdk mising这样的信息，这个时候只要加入jdk依赖即可

<dependency>
   <groupId>jdk.tools</groupId>
   <artifactId>jdk.tools</artifactId>
   <version>1.6</version>
   <scope>system</scope>
   <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>

2.修改其子工程mahout-core，将pom.xml的
<profiles>
<profile>
<id>hadoop-0.20</id>
<activation>
<property>
<name>!hadoop.version</name>
</property>
</activation>
<dependencies>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-hdfs</artifactId>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
        </dependency>
       <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
        </dependency>
           <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-common</artifactId>
        </dependency>
</dependencies>
    </profile>

    <profile>
      <id>hadoop-0.23</id>
      <activation>
        <property>
          <name>hadoop.version</name>
        </property>
      </activation>
<dependencies>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
        </dependency>
       <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
        </dependency>
           <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-common</artifactId>
        </dependency>
</dependencies>
    </profile>
</profiles>

export JAVA_HOME=/root/soft/jdk1.7.0_79
export MAHOUT_HOME=/root/soft/mahout-distribution-0.9
export HADOOP_HOME=/root/soft/hadoop-2.6.0
export SCALA=/root/soft/scala-2.11.0
export HADOOP_CONF_DIR=/root/soft/hadoop-2.6.0/etc/hadoop
export M2HOME=/etc/maven/apache-maven-3.5.2
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MAHOUT_HOME/bin:$SCALA/bin:$M2HOME/bin

转载于:https://my.oschina.net/goudingcheng/blog/1560744