Linux环境下hadoop 2.6.0 分布式集群搭建（实战演练）

最新推荐文章于 2024-08-15 14:57:09 发布
最新推荐文章于 2024-08-15 14:57:09 发布
阅读量531 收藏
点赞数
分类专栏： 大数据 文章标签： hadoop 大数据 海量数据 分布式 集群

应用场景

之前介绍了单节点，部署伪分布式hadoop集群，可以作为自己使用，但是真正投入生产环境，伪分布式是不够的，仅仅作为个人研究测试使用，此时我们需要部署搭建hadoop完全分布式集群，此hadoop性能将更加强悍，满足生产需求，下面就搭建Apache Hadoop2.6.0环境演示。

操作步骤

1. 配置网络[每个节点]

网络配置博客

2. 关闭防火墙[每个节点]

<span style="color:#333333"><span style="color:#000000"><code> <span style="color:#880000"># systemctl stop firewalld.service</span>
 <span style="color:#880000"># systemctl disable firewalld.service</span>

 <span style="color:#880000"># vim /etc/selinux/config   #设置selinux = disabled</span></code></span></span>

3. 修改主机名和配置hosts[每个节点]

<span style="color:#333333"><span style="color:#000000"><code> <span style="color:#880000"># vim  /etc/hostname      #3个节点分别命为hadoop0，hadoop1，hadoop2</span>
 <span style="color:#880000"># vim  /etc/hosts    #添加3个节点ip以及对应的主机名</span></code></span></span>

4. 配置3个节点间的SSH互信

配置SSH互信博客

5. 配置时间同步

NTP配置时间同步博客

6. 安装jdk[每个节点]

jdk安装教程博客

7. 主节点安装mysql[主节点]

mysql安装教程博客

8. 安装配置hadoop2.6.0集群

hadoop2.6.0下载地址

8.1 安装hadoop 集群 

<span style="color:#333333"><span style="color:#000000"><code>操作步骤：
<span style="color:#006666">1</span>.将下载的hadoop2.<span style="color:#006666">6.0</span>压缩包，上传到主节点的opt目录下
<span style="color:#006666">2</span>.进行解压缩
<span style="color:#006666">3</span>.配置环境变量
<span style="color:#006666">4</span>.新建所需要的目录

<span style="color:#880000"># cd /opt</span>
<span style="color:#880000"># tar -xzvf hadoop-2.6.0-x64.tar.gz</span>
<span style="color:#880000"># mv hadoop-2.6.0 hadoop2.6.0     #解压hadoop安装包，并且修改目录为hadoop2.6.0</span>

<span style="color:#880000"># vim /etc/profile    修改配置文件，加入hadoop的环境变量</span>
    <span style="color:#000088">export</span> JAVA_HOME=/opt/jdk1.<span style="color:#006666">8</span>
    <span style="color:#000088">export</span> CLASSPATH=.:<span style="color:#009900">$JAVA_HOME</span>/lib/dt.jar:<span style="color:#009900">$JAVA_HOME</span>/lib/tools.jar
    <span style="color:#000088">export</span> PATH=<span style="color:#009900">$PATH</span>:<span style="color:#009900">$JAVA_HOME</span>/bin
    <span style="color:#000088">export</span> HADOOP_HOME=/opt/hadoop2.<span style="color:#006666">6.0</span>
    <span style="color:#000088">export</span> PATH=<span style="color:#009900">$HADOOP_HOME</span>/bin:<span style="color:#009900">$HADOOP_HOME</span>/sbin:<span style="color:#009900">$PATH</span>

<span style="color:#880000">#mkdir  /opt/hadoop2.6.0/tmp      #创建目录，后续搭建过程中需要使用</span>
<span style="color:#880000">#mkdir  /opt/hadoop2.6.0/var  </span>
<span style="color:#880000">#mkdir  /opt/hadoop2.6.0/dfs  </span>
<span style="color:#880000">#mkdir  /opt/hadoop2.6.0/dfs/name  </span>
<span style="color:#880000">#mkdir  /opt/hadoop2.6.0/dfs/data  </span></code></span></span>

8.2 修改hadoop-env.sh文件

# cd /opt/hadoop2.6.0/etc/hadoop/ 
# vim hadoop-env.sh 
将:export JAVA_HOME=${JAVA_HOME} 
修改为：export JAVA_HOME=/opt/jdk1.8 #修改为jdk目录

8.3 修改slaves文件

# cd /opt/hadoop2.6.0/etc/hadoop/ 
# vim slaves

hadoop0 
hadoop1 
hadoop2 
#此时是这种情况，hadoop0作为主节点，以及主备节点，管理节点，而同时hadoop0，hadoop1，hadoop2都作为数据节点！

8.4 修改core-site.xml文件

<span style="color:#333333"><span style="color:#000000"><code> # cd /opt/hadoop2.6.0/etc/hadoop/ 
 # vim core-site.xml

<span style="color:#006666"><<span style="color:#4f4f4f">configuration</span>></span>
 <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>hadoop.tmp.dir<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>/opt/hadoop2.6.0/tmp<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>Abase for other temporary directories.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>fs.default.name<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>hdfs://hadoop0:9000<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">configuration</span>></span></code></span></span>

8.5 修改hdfs-site.xml文件

<span style="color:#333333"><span style="color:#000000"><code> <span style="color:#880000"># cd /opt/hadoop2.6.0/etc/hadoop/ </span>
 <span style="color:#880000"># vim hdfs-site.xml</span>

<configuration>
<<span style="color:#000088">property</span>>
   <name>dfs.name.dir</name>
   <value>/opt/hadoop2<span style="color:#006666">.6</span><span style="color:#006666">.0</span>/dfs/name</value>
   <description>Path <span style="color:#000088">on</span> <span style="color:#000088">the</span> <span style="color:#000088">local</span> filesystem <span style="color:#000088">where</span> theNameNode stores <span style="color:#000088">the</span> namespace <span style="color:#000088">and</span> transactions logs persistently.</description>
</<span style="color:#000088">property</span>>
<<span style="color:#000088">property</span>>
   <name>dfs.data.dir</name>
   <value>/opt/hadoop2<span style="color:#006666">.6</span><span style="color:#006666">.0</span>/dfs/data</value>
   <description>Comma separated list <span style="color:#000088">of</span> paths <span style="color:#000088">on</span> <span style="color:#000088">the</span> localfilesystem <span style="color:#000088">of</span> a DataNode <span style="color:#000088">where</span> <span style="color:#000088">it</span> should store <span style="color:#000088">its</span> blocks.</description>
</<span style="color:#000088">property</span>>
<<span style="color:#000088">property</span>>
   <name>dfs.replication</name>
   <value><span style="color:#006666">3</span></value>
</<span style="color:#000088">property</span>>
<<span style="color:#000088">property</span>>
      <name>dfs.permissions</name>
      <value><span style="color:#009900">false</span></value>
      <description>need <span style="color:#000088">not</span> permissions</description>
</<span style="color:#000088">property</span>>
</configuration></code></span></span>

8.6 修改mapred-site.xml文件

<span style="color:#333333"><span style="color:#000000"><code> # cd /opt/hadoop2.6.0/etc/hadoop/ 
 # cp   mapred-site.xml.template mapred-site.xml  
 # vim hdfs-site.xml

<span style="color:#006666"><<span style="color:#4f4f4f">configuration</span>></span>
 <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
    <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>mapred.job.tracker<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
    <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>hadoop0:49001<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
      <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>mapred.local.dir<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
       <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>/opt/hadoop2.6.0/var<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
       <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>mapreduce.framework.name<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
       <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>yarn<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">configuration</span>></span></code></span></span>

8.6 修改yarn-site.xml文件

<span style="color:#333333"><span style="color:#000000"><code> # cd /opt/hadoop2.6.0/etc/hadoop/ 
 # vim yarn-site.xml

<span style="color:#006666"><<span style="color:#4f4f4f">configuration</span>></span>

<span style="color:#880000"><!-- Site specific YARN configuration properties --></span>
<span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.hostname<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>hadoop0<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>The address of the applications manager interface in the RM.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8032<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>The address of the scheduler interface.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.scheduler.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8030<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>The http address of the RM web application.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.webapp.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8088<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>The https adddress of the RM web application.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.webapp.https.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8090<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.resource-tracker.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8031<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">description</span>></span>The address of the RM admin interface.<span style="color:#006666"></<span style="color:#4f4f4f">description</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.resourcemanager.admin.address<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>${yarn.resourcemanager.hostname}:8033<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.nodemanager.aux-services<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>mapreduce_shuffle<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
   <span style="color:#006666"><<span style="color:#4f4f4f">property</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">name</span>></span>yarn.scheduler.maximum-allocation-mb<span style="color:#006666"></<span style="color:#4f4f4f">name</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">value</span>></span>12288<span style="color:#006666"></<span style="color:#4f4f4f">value</span>></span>
        <span style="color:#006666"><<span style="color:#4f4f4f">discription</span>></span>每个节点可用内存,单位MB,默认8182MB<span style="color:#006666"></<span style="color:#4f4f4f">discription</span>></span>
   <span style="color:#006666"></<span style="color:#4f4f4f">property</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">configuration</span>></span></code></span></span>

**注：在主节点上配置好hadoop包后，同步到另外两个节点，配置不用修改，三个节点的配置都一样！ 
拷贝过去要注意目录是否有权限：chmod 777 -R /opt/hadoop2.6.0 【如果没有权限，会导致data节点无法启动】**

9. hadoop集群的初始化以及启动【主节点hadoop0上操作】

在管理节点上进行初始化以及启动

# cd /opt/hadoop2.6.0/bin 
# ./hadoop namenode -format #初始化hadoop集群

格式化成功后，可以在看到在/opt/hadoop2.6.0/dfs/name/目录多了一个current目录，而且该目录内有4个文件。

# cd /opt/hadoop2.6.0/sbin 
# ./start-all.sh #启动hadoop集群

10. 集群验证

10.1 分别在3个节点上参看进程，用jps命令

这里写图片描述

这里写图片描述

这里写图片描述

10.2 web页面测试

这里写图片描述

这里写图片描述

版权声明：个人学习，转载自 https://blog.csdn.net/bingoxubin/article/details/78503370，如有侵权请告知

kinzxv
关注
专栏目录
hadoop2.6.0分布式集群搭建（详细），搭建es-hadoop
shakalaka1的博客
09-25 6624
hadoop分布式集群的搭建，es-hadoop的搭建
Centos7+hadoop2.6.0分布式集群搭建
01-21
在CentOS 7上搭建Hadoop 2.6.0分布式集群的详细过程涉及到了多个方面的配置和设置，为了确保整个集群能够高效、稳定地运行，下面将详细解释标题中提及的各个知识点。 ### 1. 配置静态网络（所有节点） 为了让集群中...
hadoop2.6版本集群环境搭建
stark_summer
01-05 419
一、环境说明 1、机器：一台物理机 和一台虚拟机 2、linux版本：[spark@S1PA11 ~]$ cat /etc/issueRed Hat Enterprise Linux Server release 5.4 (Tikanga) 3、JDK:[spark@S1PA11 ~]$ java -versionjava version "1.6.0_27"Java(TM) SE Run...
Linux下的Hadoop分布式安装详解
最新发布
weixin_43322583的博客
08-15 1123
Linux下分布式部署hadoop全流程详细记录附带图文。
Hadoop2.6.0 + zookeeper集群环境搭建
yhao2014的专栏
04-08 4797
在centos6.5上进行hadoop2.6.0+zookeeper集群初步安装
Linux下搭建Hadoop详细步骤
u012516914的专栏
07-15 2267
装好虚拟机+Linux，并且主机网络和虚拟机网络互通。以及Ｌｉｎｕｘ上装好ＪＤＫ １：在Linux下输入命令vi /etc/profile 添加HADOOP_HOME export JAVA_HOME=/home/hadoop/export/jdk export HADOOP_HOME=/home/hadoop/export/hadoop export PATH=.:$PATH:
hadoop2.6.0三个节点集群环境搭建（一）
forever19870418的博客
03-16 1019
一、操作系统配置 (1) 主机名更改 (2) Ip地址配置 (3) /etc/hosts文件配置 (4) 无密码登入 (5) Jdk安装 (6) /etc/hosts文件配置防火墙关闭 (7) Selinux关闭 (8) vm.swappiness参数配置 (9) 创建hadoop用户及目录 (1)主机名更改： [root@elephant catchup
【向Linux迁移记录】Deepin Linux下快速Hadoop完全分布式集群搭建-附件资源
03-05
【向Linux迁移记录】Deepin Linux下快速Hadoop完全分布式集群搭建-附件资源
Hadoop2.6.0伪分布式集群搭建
weixin_33895695的博客
11-10 127
应用场景 在研究hadoop的过程中，当然需要部署hadoop集群，如果想要在本地简单试用hadoop，并且没有那么多服务器供你使用，那么伪分布式hadoop环境绝对是你最好的选择。 操作步骤 1. 安装JDK 1.1 查看是否安装了openjdk # ...
hadoop2.6.0分布式集群搭建(手动和自动HA)
Hello 我的
01-31 1373
hadoop1和hadoop2的对比 hadoop1.0中hdfs和mapreduce同属于hadoop集群，而hadoop2.0则把mapreduce分离出来，和其他所有计算模型共 同放到yarn平台上运行，如果想在hadoop2.0的yarn平台上运行任何计算模型，需要实现yarn平台的借口。 hadoop2.0中的federation和HA federation主
Linux-Hadoop集群搭建详细简明教程
08-31
Linux 操作系统安装环境下Hadoop集群搭建详细简明教程
Linux下Hadoop2.6.0集群环境的搭建
weixin_30657999的博客
12-03 124
参见以下博文： Linux下Hadoop2.6.0集群环境的搭建 转载于:https://www.cnblogs.com/shujuxiong/p/10058529.html
Linux环境下Hadoop集群搭建
aaxx123546的博客
10-16 255
Linux环境下Hadoop集群搭建 前言： 最近来到了武汉大学，在这里开始了我的研究生生涯。昨天通过学长们的耐心培训，了解了Hadoop，Hdfs，Hive，Hbase，MangoDB等等相关的知识。虽然只是简单的介绍了相关的概念，但对于我来说像是打开了一个新的大门。当然，巧妇难为无米之炊，要想能够使用这些技术，首先要能够把相应的环境搭建起来。因此，在这里我将自己搭建Hadoop集...
Linux下Hadoop2.6的安装
mm_bit的博客
10-28 2131
linux：安装流程： 1、安装JDK，要求是jdk1.6及其以上的版本： 以jdk-8u40-linux-x64.gz为例，在你的java下载目录下 sudo tar -zxvf jdk-8u40-linux-x64.gz sudo mv hadoop-2.6.0 /usr/local/jdk1.8.0_40 然后 sudo gedit /etc/profi
Linux下搭建Hadoop分布式教程
戳一下有机会获得一份额外加个蛋的蛋炒饭
11-15 2054
1、首先在Linux中安装jdk与Hadoop，解压及安装成功后配置环境变量。 vi /etc/profile 打开文件后，将下面这段代码复制进去 #JAVA_HOME export JAVA_HOME=/opt/module/jdk1.8.0_231 export PATH=$PATH:$JAVA_HOME/bin #HADOOP_HOME export HADOOP_HOME=/opt...
linux下hadoop的搭建
m0_46651978的博客
08-16 1361
https://blog.csdn.net/weixin_44695793/article/details/108010521
一把手一把手教你在Linux上搭建hadoop
Lpf的博客
04-05 4540
昨天晚上吃的东西不怎么卫生今天肚子很难受，还好放清明了，今天刚刚从医院回来，玩一个hadoop的分布式搭建再休息休息 废话不多说 1、网络配置好，关闭iptables防火墙，关闭selinux 2、jdk1.7.0_67 hadoop-2.6.5.tar.gz 上传这两个包 3、安装jdk rpm -ivh jdk-7u80-linux-x64.rpm 4、编辑vi /etc/profile 添加...
Linux安装配置Hadoop2.6（保姆级教程）单机版
wanjialin的博客
10-25 4988
操作步骤： 准备工作：一台虚拟机（Centos 7），虚拟机已安装好jdk。 1、首先使用Xftp将hadoop-2.6.0-cdh5.14.2.tar.gz包放入 /opt/install文件夹下， install文件夹需要自己新建一下，参考命令mkdir install 2、输入命令tar -zxvf hadoop-2.6.0-cdh5.14.2.tar.gz -C ../soft -C用于指定目录，操作完成后，进入soft目录下查看 名字太长不方便以后使用，我们修改一下文件名 mv hadoop-
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值