一.介绍
简介:基于Apache原生的Hadoop搭建Hadoop分布式集群。
二.环境准备
需要准备的环境:
1.虚拟机三台:ceotos7
ip映射:192.168.131.128 server1
192.168.131.131 server2
192.168.131.132 server3
2.jdk1.8(hadoop的服务运行在JVM上),下载linux版本的jdk
[jdk下载地址](https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html)
3.当前安装版本为hadoop2.8.5
下载链接:[https://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/]
下载文件:hadoop-2.8.5.tar.gz
三.环境配置
1. 关闭每个节点的防火墙(server1-server3)
多节点之间的数据的传输需要通过hadoop内部的网络实现,先关闭各节点防火墙
关闭: systemctl stop firewalld
关闭防火墙开启自启: system disable firewalld
查看防火墙状态: system status firewalld
配置ip映射:vim /etc/hosts
192.168.131.128 server1
192.168.131.131 server2
192.168.131.132 server3
三台服务器之间做免密登录:
ssh-ketgen -t rsa
ssh-copy-id server1
2.安装jdk1.8(server1-server3)
安装在/opt目录下
解压:tar -xzvf jdk-8u211-linux-x64.tar.gz
配置jdk的环境变量:
vim /etc/profile
文件末尾追加:
export JAVA_HOME=/opt/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile 刷新系统配置文件
执行命令:java
3.安装hadoop (server1-server3)
tar -xzvf hadoop-2.8.5.tar.gz
修改配置(hadoop的配置文件夹:/opt/hadoop/etc/hadoop)
配置hadoop的环境变量
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
**hadoop-env.sh**
#配置jdk路径
export JAVA_HOME=/opt/jdk1.8
**hdfs-site.xml**
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/data/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop/data/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>server1:50090</value>
</property>
</configuration>
**yarn-site.xml**
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>server1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<configuration>
**mapred-site.xml**
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
**slaves**
server2
server3
hadoop拷贝到另外两个节点server2 server3:
scp -r /opt/hadoop server2:/opt
scp -r /opt/hadoop server3:/opt
四.启动测试
1.格式化namenode: hadoop namenode -format
2.server1启动hadoop start-all.sh
3.查看启动日志
4.执行hadoop命令 往hdfs上传文件
hadoop fs -mkdir /test
hadoop fs -put test.txt /test
hadoop dfs -cat /test/test.txt
5.访问hdfs的web页面:http://192.168.131.128:50070