Hadoop分布式搭建之hadoop集群搭建实践

Hadoop2.7.2分布式搭建
1~5步三台机器都要做
1、 新建三个虚拟机
2、 分别给三台机器配置网络，修改主机名（重启永久生效）
3、 关闭防火墙
4、 配置映射文件

[root@hadoop-NN ~]# vi /etc/hosts		
192.168.28.77 hadoop-NN
192.168.28.78 hadoop-DN
192.168.28.79 hadoop-DN2
	IP地址		 主机名				

5、 配置免秘钥
a) 生成密钥对

ssh-keygen -t rsa

b) 发送秘钥给所有机器，包括自己（如果不配置映射，需要输入IP地址）

ssh-copy-id hadoop-NN (主机名)
ssh-copy-id hadoop-DN2
ssh-copy-id hadoop-DN

c) 验证是否配置成功（每台机器验证一次）

ssh hadoop-NN (主机名)
ssh hadoop-DN
ssh hadoop-DN2

6、 解压JDK，配置JAVA环境变量
a) 将jdk的包传到虚拟机中
b) 解压jdk的包

[root@hadoop-NN bigdata]# tar -zxvf jdk-8u40-linux-x64.gz

c) 删除压缩包

[root@hadoop-NN bigdata]# rm -f jdk-8u40-linux-x64.gz

d) 检查系统java环境

which java

e) 注销系统默认的java环境

[root@hadoop-NN bigdata]# mv /usr/bin/java /usr/bin/java.g
					java -version（检测是否注销掉）

f) 配置环境变量

[root@hadoop-NN bigdata]# vi /etc/profile
export JAVA_HOME=/opt/bigdata/jdk1.8.0_40
export PATH=$PATH:$JAVA_HOME/bin
	
	[root@hadoop-NN bigdata]# source /etc/profile  (刷新profile文件)

	[root@hadoop-NN bigdata]# java -version
验证是否
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

g)发送解压目录到其他两台机器

	[root@hadoop-NN ]# scp -r /opt/bigdata/ root@hadoop-DN:/opt/
	[root@hadoop-NN ]# scp -r /opt/bigdata/ root@hadoop-DN2:/opt/
	[root@hadoop-NN ]# scp -r /etc/profile root@hadoop-DN:/etc/
	[root@hadoop-NN ]# scp -r /etc/profile root@hadoop-DN2:/etc/

h)修改其他两台的环境

[root@hadoop-DN ~]# mv /usr/bin/java /usr/bin/java.1
[root@hadoop-DN ~]# source /etc/profile
[root@hadoop-DN2 ~]# mv /usr/bin/java /usr/bin/java.1
[root@hadoop-DN2 ~]# source /etc/profile

7、 解压hadoop、配置环境变量，并修改配置文件
7.1解压hadoop、配置环境变量
a)解压hadoop压缩包

	[root@hadoop-NN bigdata]# tar -zxvf hadoop-2.7.2.tar.gz
	[root@hadoop-NN bigdata]# rm -rf hadoop-2.7.2.tar.gz

b)配置hadoop环境变量

[root@hadoop-NN bigdata]# vi /etc/profile
export HADOOP_HOME=/opt/bigdata/hadoop-2.7.2
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 c)刷新profile文件
	[root@hadoop-NN bigdata]# source /etc/profile

 d)验证
	[root@hadoop-NN bigdata]# hadoop/hdfs

7.2修改配置文件

第一步[root@hadoop-NN ]# cd hadoop-2.7.2/etc/hadoop

[root@hadoop-NN hadoop]# vi hadoop-env.sh

export JAVA_HOME=/opt/bigdata/jdk1.8.0_40

第二步[root@hadoop-NN hadoop]# vi core-site.xml

<configuration>
<property>
	<name>fs.defaultFS</name>
	<value>hdfs://h101:8020</value>
</property>
<property>
	<name>io.file.buffer.size</name>
	<value>4096</value>
	</property>
<property>
	<name>hadoop.tmp.dir</name>
	<value>/opt/bigdata/tmp</value>
</property>
</configuration>

第三步[root@hadoop-NN hadoop]# vi hdfs-site.xml

<configuration>
<property>
	<name>dfs.replication</name>
	<value>2</value>
</property>
<property>
	<name>dfs.block.size</name>
	<value>134217728</value>
</property>
<property>
	<name>dfs.namenode.name.dir</name>
	<value>file:///opt/bigdata/tmp/dfs/name</value>
</property>
<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:///opt/bigdata/tmp/dfs/data</value>
</property>
<property>
	<name>fs.checkpoint.dir</name>
	<value>file:///opt/bigdata/tmp/checkpoint/dfs/cname</value>
</property>
<property>
	<name>fs.checkpoint.edits.dir</name>
	<value>file:///opt/bigdata/tmp/checkpoint/dfs/cname</value>
</property>
<property>
   <name>dfs.http.address</name>
   <value>hadoop-NN:50070</value>
</property>
<property>
   <name>dfs.secondary.http.address</name>
   <value>hadoop-NN:50090</value>
</property>
<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
</property>
<property>
	<name>dfs.permissions</name>
	<value>false</value>
</property>
</configuration>

第四步[root@hadoop-NN hadoop]# cp mapred-site.xml.template mapred-site.xml
[root@hadoop-NN hadoop]# vi mapred-site.xml

<configuration>
<!-- 指定mr运行在yarn上 -->
	<property>
    	<name>mapreduce.framework.name</name>
    	<value>yarn</value>
</property>
<property>
	<name>mapreduce.jobhistory.address</name>
	<value>hadoop-NN:10020</value>
</property>
<property>
	<name>mapreduce.jobhistory.webapp.address</name>
	<value>hadoop-NN:19888</value>
</property>
</configuration>

第五步root@hadoop-NN hadoop]# vi yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>hadoop-NN</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
</configuration>

第六步root@hadoop-NN hadoop]# vi slaves

hadoop-DN
hadoop-DN2

第七步 将配置远程发送到其他两台机器

root@hadoop-NN hadoop]# 	scp -r /opt/bigdata/hadoop-2.7.2/ root@hadoop-DN:/opt/bigdata/
root@hadoop-NN hadoop]# 	scp -r /opt/bigdata/hadoop-2.7.2/ root@hadoop-DN2:/opt/bigdata/

第八步格式化HDFS

hdfs namenode -format

第八步启动hadoop
.1和2选择其一即可

	1)start-all.sh(傻瓜式启动)
	
	2)	start-dfs.sh
		start-yarn.sh

第九步web访问hadoop

hadoop主机名:50070