学习spark之hadoop编译部署

1.搭建环境

1.1安装并配置maven

maven3下载地址：http://mirror.bit.edu.cn/apache/maven/maven-3 本文使用的是3.3.9  http://apache.fayea.com/hadoop/common/hadoop-2.6.4/hadoop-2.6.4-src.tar.gz

下载安装包并解压到/usr/local/maven/

tar -xvf apache-maven-3.3.9-bin.tar.gz  -C /usr/local/maven/

配置环境变量，vim ~/.bash_profile

export MARVEN_HOME=/usr/local/maven/apache-maven-3.3.9
export PATH=$SCALA_HOME/bin:$JAVA_HOME/bin:$HOME/bin:$MARVEN_HOME/bin:$PATH

使之生效:

source   ~/.bash_profile

1.2安装必要包

yum install -y svn autoconf automake libtool cmake ncurses-devel openssl-devel gcc*

1.3安装并设置protobuf

protobuf需要安装2.5.0版本，否则后续编译会出现错误。下载页https://github.com/google/protobuf/releases?after=v3.0.0-beta-1

下载安装包并解压到/usr/local/protobuf

tar -xvf protobuf-2.5.0.tar -C /usr/local/protobuf

编译安装

./configure
make
make check
make install

2、编译hadoop

2.1 下载

hadoop下载页：http://hadoop.apache.org/releases.html，源码下载链接：http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4-src.tar.gz

2.2 编译

注意内存低于1G会出现outofmemory的异常

解压并进入目录执行命令：

mvn package -Pdist,native -DskipTests -Dtar -X

验证编译是否成功

file ./hadoop-dist/target/hadoop-2.6.4/lib/native/libhadoop.so.1.0.0 

hadoop-dist/target 目录下的hadoop-2.6.4.tar.gz就是hadoop的安装包

3、安装hadoop

3.1解压hadoop安装包 到/usr/local/sinasrv2/hadoop/hadoop-2.6.4

3.2在hadoop-2.6.4目录下创建子目录 tmp name data

3.3配置hadoop-env.sh

vim etc/hadoop/hadoop-env.sh 

添加JAVA_HOME和PATH

 25 export JAVA_HOME=/usr/local/sinasrv2/java/jdk1.7.0_80
 26 export JAVA_HOME=${JAVA_HOME}
 27 export PATH=$JAVA_HOME/bin:$PATH

source etc/hadoop/hadoop-env.sh 使之生效

3.4配置yarn-env.sh

vim etc/hadoop/yarn-env.sh 

 24 export JAVA_HOME=/usr/local/sinasrv2/java/jdk1.7.0_80
 25 

source  etc/hadoop/yarn-env.sh 使之生效

3.5配置core-site.xml

vim etc/hadoop/core-site.xml   注意hadoop.tmp.dir的配置要根据自己的目录结构

<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://hadoop1:9000</value>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop1:9000</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>131072</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/usr/local/sinasrv2/hadoop/hadoop-2.6.4/tmp</value>
    <description>Abase for other temporary directories.</description>
  </property>
  <property>
    <name>hadoop.proxyuser.hduser.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hduser.groups</name>
    <value>*</value>
  </property>
</configuration>

3.6 配置hdfs-site.xml

vim etc/hadoop/hdfs-site.xml   注意dfs.namenode.name.dir dfs.datanode.data.dir的配置要根据自己的目录结构

<configuration>
<property>
   <name>dfs.namenode.secondary.http-address</name>
   <value>hadoop1:9001</value>
  </property>
  <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/sinasrv2/hadoop/hadoop-2.6.4/name</value>
  </property>
  <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/sinasrv2/hadoop/hadoop-2.6.4/data</value>
  </property>
  <property>
   <name>dfs.replication</name>
   <value>2</value>
  </property>
  <property>
   <name>dfs.webhdfs.enabled</name>
   <value>true</value>
  </property>
</configuration>

3.7 配置mapred-site.xml

cp  etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml

vim etc/hadoop/mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop1:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop1:19888</value>
  </property>
</configuration>

3.8配置yarn-site.xml

vim etc/hadoop/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop1:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop1:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop1:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop1:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop1:8088</value>
  </property>
</configuration>

3.9配置slaves文件

vim etc/hadoop/slaves 

hadoop1

4启动部署

4.1格式化NameNode

./bin/hdfs namenode -format

4.2启动hdfs

./sbin/start-dfs.sh

4.3验证HDFS启动

增加调试信息

$ export HADOOP_ROOT_LOGGER=DEBUG,console

$ jps
3592 Master
3054 SecondaryNameNode
3665 Worker
2921 DataNode
2828 NameNode
14655 Jps

4.4启动yarn

./sbin/start-yarn.sh

4.5验证yarn启动

$ jps
15082 Jps
3592 Master
14793 NodeManager
3054 SecondaryNameNode
3665 Worker
2921 DataNode
14701 ResourceManager
2828 NameNode