Hadoop2.7.2分布式搭建
1~5步三台机器都要做
1、 新建三个虚拟机
2、 分别给三台机器配置网络,修改主机名(重启永久生效)
3、 关闭防火墙
4、 配置映射文件
[root@hadoop-NN ~]# vi /etc/hosts
192.168.28.77 hadoop-NN
192.168.28.78 hadoop-DN
192.168.28.79 hadoop-DN2
IP地址 主机名
5、 配置免秘钥
a) 生成密钥对
ssh-keygen -t rsa
b) 发送秘钥给所有机器,包括自己(如果不配置映射,需要输入IP地址)
ssh-copy-id hadoop-NN (主机名)
ssh-copy-id hadoop-DN2
ssh-copy-id hadoop-DN
c) 验证是否配置成功(每台机器验证一次)
ssh hadoop-NN (主机名)
ssh hadoop-DN
ssh hadoop-DN2
6、 解压JDK,配置JAVA环境变量
a) 将jdk的包传到虚拟机中
b) 解压jdk的包
[root@hadoop-NN bigdata]# tar -zxvf jdk-8u40-linux-x64.gz
c) 删除压缩包
[root@hadoop-NN bigdata]# rm -f jdk-8u40-linux-x64.gz
d) 检查系统java环境
which java
e) 注销系统默认的java环境
[root@hadoop-NN bigdata]# mv /usr/bin/java /usr/bin/java.g
java -version(检测是否注销掉)
f) 配置环境变量
[root@hadoop-NN bigdata]# vi /etc/profile
export JAVA_HOME=/opt/bigdata/jdk1.8.0_40
export PATH=$PATH:$JAVA_HOME/bin
[root@hadoop-NN bigdata]# source /etc/profile (刷新profile文件)
[root@hadoop-NN bigdata]# java -version
验证是否
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
g)发送解压目录到其他两台机器
[root@hadoop-NN ]# scp -r /opt/bigdata/ root@hadoop-DN:/opt/
[root@hadoop-NN ]# scp -r /opt/bigdata/ root@hadoop-DN2:/opt/
[root@hadoop-NN ]# scp -r /etc/profile root@hadoop-DN:/etc/
[root@hadoop-NN ]# scp -r /etc/profile root@hadoop-DN2:/etc/
h)修改其他两台的环境
[root@hadoop-DN ~]# mv /usr/bin/java /usr/bin/java.1
[root@hadoop-DN ~]# source /etc/profile
[root@hadoop-DN2 ~]# mv /usr/bin/java /usr/bin/java.1
[root@hadoop-DN2 ~]# source /etc/profile
7、 解压hadoop、配置环境变量,并修改配置文件
7.1解压hadoop、配置环境变量
a)解压hadoop压缩包
[root@hadoop-NN bigdata]# tar -zxvf hadoop-2.7.2.tar.gz
[root@hadoop-NN bigdata]# rm -rf hadoop-2.7.2.tar.gz
b)配置hadoop环境变量
[root@hadoop-NN bigdata]# vi /etc/profile
export HADOOP_HOME=/opt/bigdata/hadoop-2.7.2
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
c)刷新profile文件
[root@hadoop-NN bigdata]# source /etc/profile
d)验证
[root@hadoop-NN bigdata]# hadoop/hdfs
7.2修改配置文件
第一步[root@hadoop-NN ]# cd hadoop-2.7.2/etc/hadoop
[root@hadoop-NN hadoop]# vi hadoop-env.sh
export JAVA_HOME=/opt/bigdata/jdk1.8.0_40
第二步[root@hadoop-NN hadoop]# vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://h101:8020</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>4096</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/bigdata/tmp</value>
</property>
</configuration>
第三步[root@hadoop-NN hadoop]# vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.block.size</name>
<value>134217728</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///opt/bigdata/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///opt/bigdata/tmp/dfs/data</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>file:///opt/bigdata/tmp/checkpoint/dfs/cname</value>
</property>
<property>
<name>fs.checkpoint.edits.dir</name>
<value>file:///opt/bigdata/tmp/checkpoint/dfs/cname</value>
</property>
<property>
<name>dfs.http.address</name>
<value>hadoop-NN:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>hadoop-NN:50090</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
第四步[root@hadoop-NN hadoop]# cp mapred-site.xml.template mapred-site.xml
[root@hadoop-NN hadoop]# vi mapred-site.xml
<configuration>
<!-- 指定mr运行在yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-NN:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-NN:19888</value>
</property>
</configuration>
第五步root@hadoop-NN hadoop]# vi yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-NN</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
第六步root@hadoop-NN hadoop]# vi slaves
hadoop-DN
hadoop-DN2
第七步 将配置远程发送到其他两台机器
root@hadoop-NN hadoop]# scp -r /opt/bigdata/hadoop-2.7.2/ root@hadoop-DN:/opt/bigdata/
root@hadoop-NN hadoop]# scp -r /opt/bigdata/hadoop-2.7.2/ root@hadoop-DN2:/opt/bigdata/
第八步格式化HDFS
hdfs namenode -format
第八步启动hadoop
.1和2选择其一即可
1)start-all.sh(傻瓜式启动)
2) start-dfs.sh
start-yarn.sh
第九步web访问hadoop
hadoop主机名:50070