Centos 6.7下集成Hadoop的Eclipse编译环境
一.软件准备
整个环境需要的软件如下：
1）apache-maven-3.5.2-bin.tar.gz
2）eclipse-java-juno-SR2-linux-gtk-x86_64.tar.gz
3）jdk-7u79-linux-x64.tar.gz
4）openssl-1.0.2a.tar.gz
5）protobuf-2.5.0.tar.gz
6）findbugs-3.0.1.tar.gz
7）hadoop-2.7.0-src.tar.gz
二.软件安装
1.JDK安装
1）执行命令tar zxvf jdk-7u79-linux-x64.tar.gz
2）执行命令mv jdk1.7.0_79 /usr/local
3）执行命令chown -R hudd /usr/local/jdk1.7.0_79
4）执行命令chgrp -R hudd /usr/local/jdk1.7.0_79
5）执行命令vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
6）执行命令source /etc/profile
7）执行命令java -version
2.Maven安装
1）执行命令tar zxvf apache-maven-3.5.2-bin.tar.gz
2）执行命令mv apache-maven-3.5.2 maven
3）执行命令mv maven /usr/local
4）执行命令chown -R hudd /usr/local/maven
5）执行命令chgrp -R hudd /usr/local/maven
6）执行命令vim /etc/profile
export M2_HOME=/usr/local/maven
export PATH=$PATH:$M2_HOME/bin
export PATH=$PATH:$M2_HOME/bin
7）执行命令source /etc/profile
8）执行命令mvn -v
3.protobuf安装
1）执行命令tar zxvf protobuf-2.5.0.tar.gz
2）执行命令chown -R hudd /home/hudd/protobuf-2.5.0
3）执行命令chgrp -Rhudd /home/hudd/protobuf-2.5.0
4）执行命令cd/home/hudd/protobuf-2.5.0
5）执行命令./configure --prefix=/usr/local
6）执行命令yum -y install glibc-headers gcc-c++
7）执行命令./configure --prefix=/usr/local
8）执行命令make
9）执行命令make install
10）执行命令protoc --version
4.findbugs安装
1）执行命令tar zxvf findbugs-3.0.1.tar.gz
2）执行命令mv findbugs-3.0.1 findbugs
3）执行命令mv findbugs /usr/local
4）执行命令chown -R hudd /usr/local/findbugs
5）执行命令chgrp -R hudd /usr/local/findbugs
6）执行命令vim /etc/profile
export FINDBUGS_HOME=/usr/local/findbugsexport PATH=$PATH:$FINDBUGS_HOME/bin
7）执行命令source /etc/profile
8）执行命令findbugs -version
5.openssl安装
1）执行命令tar zxvf openssl-1.0.2a.tar.gz
2）执行命令chown -R hudd /home/hudd/openssl-1.0.2a
3）执行命令chgrp -R hudd /home/hudd/openssl-1.0.2a
4）执行命令cd /home/hudd/openssl-1.0.2a
5）执行命令./config --prefix=/usr/local
6）执行命令make
7）执行命令make install
8）执行命令openssl version
6.eclipse安装
1）执行命令tar zxvf eclipse-java-juno-SR2-linux-gtk-x86_64.tar.gz
2）执行命令mv eclipse /usr/local
3）执行命令chown -R hudd /usr/local/eclipse
4）执行命令chgrp -R hudd /usr/local/eclipse
5）执行命令cd /usr/share/applications
6）执行命令vim eclipse.desktop
[Desktop Entry]
Encoding=UTF-8
Name=Eclipse
Comment=Eclipse IDE
Exec=/usr/local/eclipse/eclipse
Icon=/usr/local/eclipse/icon.xpm
Terminal=false
Type=Application
Categories=GNOME;Application;Development;
StartupNotify=true
Encoding=UTF-8
Name=Eclipse
Comment=Eclipse IDE
Exec=/usr/local/eclipse/eclipse
Icon=/usr/local/eclipse/icon.xpm
Terminal=false
Type=Application
Categories=GNOME;Application;Development;
StartupNotify=true
7）打开applications目录，将快捷方式拷贝到桌面即可
8）在该版本的操作系统（Centos）下，eclipse在打开类文件时会自动关闭，如下处理
在Eclipse安装目录下，编辑eclipse.ini文件，添加一行：
-Dorg.eclipse.swt.internal.gtk.cairoGraphics=false
7.Hadoop源代码处理
1）修改Maven的配置settings.xml
执行命令cd /usr/local/maven
执行命令mkdir repository
执行命令cd conf
执行命令cp settings.xml settings.xml.bak
执行命令vim settings.xml
<localRepository>/usr/local/maven/repository</localRepository>
<mirrors>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>central</id>
<name>Maven Repository Switchboard</name>
<url>http://repo1.maven.org/maven2/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>repo2</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo2.maven.org/maven2/</url>
</mirror>
<mirror>
<id>ibiblio</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://mirrors.ibiblio.org/pub/mirrors/maven2/</url>
</mirror>
<mirror>
<id>jboss-public-repository-group</id>
<mirrorOf>central</mirrorOf>
<name>JBoss Public Repository Group</name>
<url>http://repository.jboss.org/nexus/content/groups/public</url>
</mirror>
</mirrors>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>central</id>
<name>Maven Repository Switchboard</name>
<url>http://repo1.maven.org/maven2/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>repo2</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://repo2.maven.org/maven2/</url>
</mirror>
<mirror>
<id>ibiblio</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://mirrors.ibiblio.org/pub/mirrors/maven2/</url>
</mirror>
<mirror>
<id>jboss-public-repository-group</id>
<mirrorOf>central</mirrorOf>
<name>JBoss Public Repository Group</name>
<url>http://repository.jboss.org/nexus/content/groups/public</url>
</mirror>
</mirrors>
<profiles>
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
这里要注意的是，由于Maven在国内的中央仓库目前已经不可用，所以去掉或者替换maven.net.cn，否则后续编译会因为下载问题导致失败。替换的时候要注意标签之间的匹配。
执行命令cp settings.xml/home/hudd/.m2
2）预处理Hadoop源代码
执行命令tar zxvf hadoop-2.7.0-src.tar.gz
执行命令chown -R hudd /home/hudd/hadoop-2.7.0-src
执行命令chgrp -R hudd/home/hudd/hadoop-2.7.0-src
执行命令cd /home/hudd/hadoop-2.7.0-src/hadoop-maven-plugins/
执行命令mvn install
执行命令cd ..
执行命令mvn eclipse:eclipse -DskipTests
3）在eclipse中导入Hadoop源代码
File—>Import—>General—>Existing Projects into Workspace
此时会发现，仍然有很多错误：
1.hadoop-common中AvroRecord类没有找到，这个类应该位于org.apache.hadoop.io.serializer.avro下，去GrepCode官网（http://grepcode.com/）搜AvroRecord的源码，然后放到这个包下即可，注意版本要对应起来。
2.hadoop-common中缺少包org.apache.hadoop.ipc.protobuf，需要新建该包，并且去GrepCode官网搜找到该包下面的TestProtos、TestRpcServiceProtos两个类，然后放到这个包下即可。
3.hadoop-streaming中出现问题，这个是hadoop-streaming里面的build path有问题，解决办法就是remove掉引用就好，具体右键右边出错项目—>properties—>左边Java Build Path—>Source—>选定错误项—>右边Remove。
4.部分项目出现Access restriction: The method arrayBaseOffset(Class) from the type Unsafe is not accessible due to restriction on required library /usr/local/jdk1.7.0_79/jre/lib/rt.jar错误，解决办法就是右键项目Properties—>Java Build Path—> Libraries，展开JRE System Library，双击Access rules，点击Add按钮，在Resolution下拉框选择Accessible，Rule Pattern填写**/*。
8.参考资料
1）http://www.aboutyun.com/thread-12183-1-1.html
2）https://www.douban.com/note/483167324/
3）http://soige.blog.51cto.com/512568/1314599
4）https://www.zhihu.com/question/47261839?sort=created
5）http://www.cnblogs.com/yangyquin/p/5021109.html
371
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
