(超详细版) Linux下安装配置Hadoop2.7.7集群环境搭建—部署Hadoop

1. (超详细版) Linux下安装配置Hadoop2.7.7集群环境搭建—基础环境配置

2. (超详细版) Linux下安装配置Hadoop2.7.7集群环境搭建—部署Hadoop

一、下载并解压hadoop2.7.7

cd /usr/local
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
tar -zxvf hadoop-2.7.7.tar.gz

二、配置环境变量

vi /etc/profile
#添加如下配置
export HADOOP_HOME=/usr/local/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin
#使配置生效
source /etc/profile
#测试（输入hadoop有提示则表示成功）
hadoop
#将profile发送到其他节点
scp /etc/profile hadoop@xt2:/etc
scp /etc/profile hadoop@xt3:/etc
#记得在其他节点下source
source /etc/profile

三、Hadoop的配置

1. 创建文件目录

为了便于管理，给xt1的hdfs的NameNode、DataNode及临时文件，在用户目录下创建目录：

#在usr/local下创建hdfs目录
cd /usr/local 
mkdir hdfs
cd hdfs
mkdir name
mkdir data
mkdir tmp
#将hdfs文件夹传到其他两个节点，注意权限问题
chmod 777 usr/local #其他节点也要
chmod 777 usr/local/hdfs
scp -r /usr/local/hdfs hadoop@xt2:/usr/local
scp -r /usr/local/hdfs hadoop@xt3:/usr/local

2. Hadoop的配置

# 进入hadoop-2.7.7的配置目录：
cd  /user/hadoop-2.7.7/etc/hadoop

依次修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml以及slaves文件。

修改core-site.xml配置

vi core-site.xml

core-site.xml的配置如下

<configuration>
 <property>
   <name>hadoop.tmp.dir</name>
   <value>file:/usr/local/hdfs/tmp</value> 
  </property>
 <property>
   <name>io.file.buffer.size</name>
   <value>131072</value>
 </property>
 <property>
   <name>fs.default.name</name>
   <value>hdfs://xt1:9000</value>
 </property>
 <property>
   <name>hadoop.proxyuser.root.hosts</name>
   <value>*</value>
 </property>
 <property>
   <name>hadoop.proxyuser.root.groups</name>
   <value>*</value>
 </property>
</configuration>

修改hdfs-site.xml

vi hdfs-site.xml

hdfs-site.xml的配置如下

<configuration>
 <property>
   <name>dfs.replication</name>
   <value>2</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hdfs/name</value>
   <final>true</final>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hdfs/data</value>
   <final>true</final>
 </property>
 <property>
   <name>dfs.namenode.secondary.http-address</name>
   <value>xt1:9001</value>
 </property>
 <property>
   <name>dfs.webhdfs.enabled</name>
   <value>true</value>
 </property>
 <property>
   <name>dfs.permissions</name>
   <value>false</value>
 </property>
</configuration>

添加修改mapred-site.xml

#将mapred-site.xml.template复制为mapred-site.xml
cp mapred-site.xml.template  mapred-site.xml

mapred-site.xml的配置如下

<!--计算任务托管的资源框架名称-->
 <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>

修改yarn-site.xml

vi yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
 <property>
   <name>yarn.resourcemanager.address</name>
   <value>xt1:8010</value>
 </property>
 <property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>xt1:8011</value>
 </property>
 <property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>xt1:8012</value>
 </property>
 <property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>xt1:8013</value>
 </property>
 <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>xt1:8014</value>
 </property>
 <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>

 <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
  <property>
   <name>yarn.nodemanager.resource.memory-mb</name>
   <value>2048</value>
 </property>
 <property>
   <name>yarn.nodemanager.resource.cpu-vcores</name>
   <value>1</value>
 </property>
</configuration>

修改slaves和hadoop-env.sh文件

#修改slaves文件
vi /usr/local/hadoop-2.7.7/etc/hadoop/slaves
#将localhost改成下方
xt2
xt3
#修改hadoop-env.sh文件(注释掉之前的，添加完整路径)
vi /usr/local/hadoop-2.7.7/etc/hadoop/hadoop-env.sh 
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/local/jdk1.8.0_141

3. 将hadoop2.7.7的文件夹复制到xt2和xt3

scp -r /usr/local/hadoop-2.7.7 hadoop@xt2:/usr/local
scp -r /usr/local/hadoop-2.7.7 hadoop@xt3:/usr/local

四、运行Hadoop

#格式化
hadoop namenode -format
cd /usr/local/hadoop-2.7.7/sbin/
#启动hadoop
sh ./start-all.sh
#查看集群状态
/usr/local/hadoop-2.7.7/bin/hdfs dfsadmin -report

五、测试

#测试yarm，登录 地址是（yarn.resourcemanager.webapp.address）的value
http://192.168.123.100:8012/cluster/cluster

#测试HDFS
http://192.168.123.100:50070/dfshealth.html

配置完毕！