1.镜像配置
/etc/yum.repo.d/CentOS6-Base-163.repo
2./etc/resove.conf
添加内容:
nameserver 8.8.8.8
nameserver 8.8.4.4
3.下载Hadoop源码工程并解压
编辑文件:usr/local/hadoop-2.6.0-src/hadoop-common-project/hadoop-auth/pom.xml,第55行左右增加如下内容:
<dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty-util</artifactId> <scope>test</scope> </dependency>
4.检查jdk(JDK 1.6+)
建议用jdk1.7
5.安装配置maven
版本:3.3.9
1)配置环境变量
/etc/profile
#MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$PATH:$JAVA_HOME/bin
2)测试 mvn -version
3)conf/setting.xml
添加内容:
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
6.ProtocolBuffer 2.5.0安装
./configure --prefix=/usr/local/protoc/
./make
./make install
安装完成后,配置环境变量,增加:export PROTOC_HOME=/usr/local/protoc/,和export PATH=..:$PROTOC_HOME/bin:..
验证:protoc --version
7.snappy安装
./configure
./make
./make install
8.CMake
sudo yum install cmake
9.hadoop编译
mvn clean package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib=/usr/local/lib
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec /user/kfk/datas/ /user/kfk/newCompress3