hadoop-2.8编译以及集群安装

博客地址:http://www.fanlegefan.com
文章地址:http://www.fanlegefan.com/archives/hadoopbuild/

---

hadoop-2.8编译

本地编译环境

• centos-6.5 64位
• mvn 3.3.9
• jdk 1.8.0_91
• CMake 2.8.12.2
• protobuf-2.5.0

其他：

sudo yum -y install gcc*
sudo yum -y install cmake
sudo yum -y install glibc-headers
sudo yum -y install gcc-c++
sudo yum -y install zip-devel
sudo yum -y install openssl-devel
sudo yum -y install svn
sudo yum -y install git
sudo yum -y install ncurses-devel 
sudo yum -y install lzo-devel
sudo yum -y install autoconf
sudo yum -y install libtool
sudo yum -y install automake

安装protobuf

https://github.com/google/protobuf
wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
进入protobuf所在目录，然后按下面顺序操作
./configure
make 
make check
sudo make install

hadoop源码下载编译

wget http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.8.0/hadoop-2.8.0-src.tar.gz
tar -zxvf hadoop-2.8.0-src.tar.gz
cd hadoop-2.8.0-src
mvn package -Dmaven.javadoc.skip=true -Pdist,native -DskipTests -Dtar

编译好后的安装包路径：

hadoop-2.8.0-src/hadoop-dist/target/hadoop-2.8.0.tar.gz

---

集群部署

机器分配

host	ip
master	192.168.1.115
slave1	192.168.1.116
slave2	192.168.1.117

免密钥登入

在master机器执行

ssh-keygen -t rsa，一直按回车
cd ~/.ssh
cat id_rsa.pub >>authorized_keys 
chmod 600 authorized_keys
chmod 700 .ssh

将authorized_keys复制到slave1，slave2的.ssh目录下，权限要和master保持一致
在master，slave1，slave2三台机器上分别执行如下命令

mkdir -p /home/qun/data/hadoop-2.8/name 
mkdir -p /home/qun/data/hadoop-2.8/data 
mkdir -p /home/qun/data/hadoop-2.8/tmp 
mkdir -p /home/qun/data/hadoop-2.8/namesecondary 

解压

tar -zxvf hadoop-2.8.0.tar.gz 
cd hadoop-2.8.0/etc/hadoop

配置core-site.xml

<property> 
<name>fs.defaultFS</name> 
<value>hdfs://master:9000</value> 
</property> 
<property> 
<name>hadoop.tmp.dir</name> 
<value>/home/qun/data/hadoop-2.8/tmp</value> 
</property> 

<property> 
<name>fs.checkpoint.period</name> 
<value>3600</value> 
</property> 
<property> 
<name>fs.checkpoint.size</name> 
<value>67108864</value> 
</property> 
<property> 
<name>fs.checkpoint.dir</name> 
<value>/home/qun/data/hadoop-2.8/namesecondary</value> 
</property>

配置hdfs-site.xml

<property> 
<name>dfs.replication</name> 
<value>2</value> 
</property> 
<property> 
<name>dfs.namenode.name.dir</name> 
<value>/home/qun/data/hadoop-2.8/name</value> 
</property> 
<property> 
<name>dfs.datanode.data.dir</name> 
<value>/home/qun/data/hadoop-2.8/data</value> 
</property> 

配置yarn-site.xml

<property> 
<name>yarn.nodemanager.aux-services</name> 
<value>mapreduce_shuffle</value> 
</property> 
<property> 
<name>yarn.resourcemanager.address</name> 
<value>master:8032</value> 
</property> 
<property> 
<name>yarn.resourcemanager.resource-tracker.address</name> 
<value>master:8031</value> 
</property> 
<property> 
<name>yarn.resourcemanager.admin.address</name> 
<value>master:8033</value> 
</property> 
<property> 
<name>yarn.resourcemanager.scheduler.address</name> 
<value>master:8030</value> 
</property> 
<property> 
<name>yarn.web-proxy.address</name> 
<value>master:8888</value> 
</property> 
<property> 
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
<value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property> 

配置mapred-site.xml

<property> 
<name>mapreduce.framework.name</name> 
<value>yarn</value> 
</property> 

配置slave

vi slave

slave1
slave2

将配置好的文件复制到slave1和slave2上

scp -r hadoop-2.8.0 qun@slave1:~/soft/
scp -r hadoop-2.8.0 qun@slave2:~/soft/

设置HADOOP_HOME

在3台机器上分别执行vi ~/.bashrc

HADOOP_HOME=/home/qun/soft/hadoop-2.8.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin

刷新配置

source ~/.bashrc

格式化namenode

hadoop namenode -format

启动hadoop

start-dfs.sh
start-yarn.sh

执行jps查看进程

在所有节点上执行jps

master
7475 ResourceManager
7998 Master
7119 NameNode
7311 SecondaryNameNode

slave1，slave2
24385 NodeManager
24276 DataNode

控制台url

http://192.168.1.115:50070/dfshealth.html#tab-overview
http://192.168.1.115:8088/cluster