关闭

Hadoop 2.4源码编译和导入eclipse

260人阅读 评论(0) 收藏 举报
分类:


Hadoop源码编译


Linux 环境下编译源代码之前,需进行以下准备工作:

安装3.0.2及 以上版本的Maven

Protocol Buffers 安装版本为 2.5.0

安装findbugs

安装2.6及以上版本的CMake

确认可以连接互联网(Maven 要从代码库中下载依赖包)


Maven国内镜像配置
进入安装目录 ~/soft/apache-maven-3.0.5/conf,编辑 settings.xml 文件

修改settings.xml:

<mirror> 
 <id>nexus-osc</id> 
  <mirrorOf>*</mirrorOf> 
  <name>Nexus osc</name> 
  <url>http://maven.oschina.net/content/groups/public/</url> 
</mirror> 

修改<profiles>内容:

<profile> 
 <id>jdk-1.7</id> 
 <activation> 
     <jdk>1.7</jdk> 
 </activation> 
 <repositories> 
  <repository> 
   <id>nexus</id> 
   <name>local private nexus</name> 
   <url>http://maven.oschina.net/content/groups/public/</url> 
   <releases> 
    <enabled>true</enabled> 
   </releases> 
   <snapshots> 
    <enabled>false</enabled> 
   </snapshots> 
  </repository> 
 </repositories>
 <pluginRepositories> 
  <pluginRepository> 
   <id>nexus</id> 
   <name>local private nexus</name> 
   <url>http://maven.oschina.net/content/groups/public/</url> 
   <releases> 
    <enabled>true</enabled> 
   </releases> 
   <snapshots> 
    <enabled>false</enabled> 
   </snapshots> 
  </pluginRepository> 
 </pluginRepositories> 
</profile>

$ cp ~/soft/apache-maven-3.0.5/conf/settings.xml ~/.m2/

修改: vi /etc/resolv.conf     
nameserver 8.8.8.8
nameserver 8.8.4.4    

 

编译源码

进入源码目录:

cd ~/workspace/hadoop-2.4.0-src

tar –xzvf Hadoop-2.4.0-src.tar.gz

mvn clean

mvn package -Pdist,native,docs-DskipTests –Dtar

Hadoop导入eclipse

进入源码目录:

cd ~/workspace/hadoop-2.4.0-src

cd hadoop-maven-plugins/

mvn install

 

cd ~/workspace/hadoop-2.4.0-src

mvn eclipse:eclipse –DeskipTests


elipse-->import-->exist project from workspace导入源码 


eclipse设置Mavenwindow->preference->maven->{Installations...;userSettingsmaven\conf\settings.xml}

 

配置eclispe M2_REPO变量

Window/Preference/java/Build Path,M2_REPO默认路径修改为:

~/.m2/repository


导入eclipse报错的解决方法:

1.hadoop-streaming里面的build path有问题,显示/root/workspace/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf(missing)

删除Link Source: hadoop-yarn-server-resourcemanager/conf
再添加Link Source:
源码根目录/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf,随便起个名字;
inclusion patterns:capacity-scheduler.xml;exclusion patters:**/*.java

2.AvroRecord cannot be resolved to a type TestAvroSerialization.java

解决方法如下:
首先,下载 avro-tools-1.7.4.jar(点击下载)。存放于 hadoop-2.4.0-src 目录中。
然后执行下述命令:
$ cd ~/workspace/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/src/test/avro
$ java -jar ~/workspace/hadoop-2.4.0-src/avro-tools-1.7.4.jar compile schema avroRecord.avsc ../java
最后,刷新Eclipse工程。

3. org.apache.hadoop.ipc.protobuf.× 无法解析
解决方法如下,依次执行下述命令。
$ cd ~/workspace/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/src/test/proto
//注释:进入该目录
$ protoc --java_out=../java *.proto

 

 

0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:2101次
    • 积分:63
    • 等级:
    • 排名:千里之外
    • 原创:4篇
    • 转载:0篇
    • 译文:0篇
    • 评论:0条
    文章分类
    文章存档