Hadoop集群搭建

最新推荐文章于 2022-04-07 16:24:07 发布
最新推荐文章于 2022-04-07 16:24:07 发布
阅读量285 收藏 1
点赞数
文章标签： hadoop集群
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ZWLJavaWeb/article/details/84061449
版权

一、环境准备以及基础配置

1、准备环境，centos7虚拟机三台，192.168.2.150，151、152

2、创建hadoop用户

useradd -d /home/hadoop -m hadoop

3、修改hadoop密码

passwd hadoop

4、修改主机名(分别修改三台机器的主机名为master、slave1、slave2)

hostnamectl set-hostname master

5、配置hosts文件,在每个节点/etc/hosts文件中添加一下内容

192.168.2.150 master
192.168.2.151 slave1
192.168.2.152 slave2

5、配置ssh免密登录,是节点之间两两互通

ssh-keygen -t rsa 

ssh-copy-id uname@hostname

6、安装jdk并且配置环境变量

7、下载hadoop安装文件，解压在工作目录中

cd /home/hadoop/work
tar -zxvf hadoop-2.8.3.tar.gz
mv hadoop-2.8.3 hadoop

8、在工作目录下创建hdfs目录

cd /home/hadoop/work
mkdir hdfs
cd hdfs
mkdir data name tmp

9、添加hadoop环境变量到系统

export HADOOP_HOME=/home/hadoop/work/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

10、添加JAVA_HOME到hadoop

在/home/hadoop/work/hadoop/etc/hadoop/hadoop-env.sh最后添加一下内容

export  JAVA_HOME=/usr/local/jdk

二、配置集群

涉及core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、slaves五个配置文件，对应各个组件的配置。 位于 /home/hadoop/work/hadoop/etc/hadoop/ 目录下,文件说明如下：

文件说明
core-site.xmlCommon组件
hdfs-site.xmlHDFS组件
mapred-site.xmlMapReduce组件
yarn-site.xmlYARN组件
slavesslave节点信息

1、core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>file:/home/hadoop/work/hdfs/tmp</value>
     <description>A base for other temporary directories.</description>
  </property>
  <property>
     <name>io.file.buffer.size</name>
     <value>131072</value>
  </property>
  <property>
     <name>fs.default.name</name>
     <value>hdfs://master:9000</value>
  </property>
  <property>
     <name>hadoop.proxyuser.root.hosts</name>
     <value>*</value>
  </property>
  <property>
     <name>hadoop.proxyuser.root.groups</name>
     <value>*</value>
  </property>
</configuration>

2、hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/hadoop/work/hdfs/name</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/home/hadoop/work/hdfs/data</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>master:9001</value>
  </property>
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>
</configuration>

3、mapred-site.xml

MapReduce组件配置文件默认为、mapred-site.xml.template ，复制问津

cp mapred-site.xml.template  mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<!-- 通知框架MapReduce使用YARN -->
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

4、yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>master:18040</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:18030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:18088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:18025</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:18141</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

5、编辑slaves文件，添加从节点信息

去掉原本的localhost，换成以下内容。配置slaves的目录，是把所有节点连在一起，构成一个相连的集群，启动时，整个集群一起启动。

slave1
slave2

6、将配置好的hadoop目录拷贝到其他各节点上

cd /home/hadoop/work
scp -r hadoop slave1:/home/hadoop/work/
scp -r hadoop slave2:/home/hadoop/work/
scp -r hdfs slave1:/home/hadoop/work/
scp -r hdfs slave2:/home/hadoop/work/

7、格式化namenode

hadoop namenode -format

8、启动hadoop集群

start-all.sh

日志输出如下：

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/09/09 12:26:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: starting namenode, logging to /home/hadoop/work/hadoop/logs/hadoop-hadoop-namenode-master.out
slave2: starting datanode, logging to /home/hadoop/work/hadoop/logs/hadoop-hadoop-datanode-slave2.out
slave1: starting datanode, logging to /home/hadoop/work/hadoop/logs/hadoop-hadoop-datanode-slave1.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /home/hadoop/work/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
18/09/09 12:27:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/work/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave2: starting nodemanager, logging to /home/hadoop/work/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
slave1: starting nodemanager, logging to /home/hadoop/work/hadoop/logs/yarn-hadoop-nodemanager-slave1.out

9、各节点进程情况

master节点：

1664 NameNode
4657 Jps
2021 ResourceManager
1865 SecondaryNameNode
1515 QuorumPeerMain

slave节点：

2145 Jps
1156 DataNode 
1100 QuorumPeerMain
1244 NodeManager

10、访问web页面

访问master WEB UI界面，可以看另外2个节点都正常运行。
http://master:50070/

查看客户端节点： http://master:18088/cluster/nodes

hadoop集群搭建
kai763253075的博客
03-14 417
一、系统环境说明： 本次搭建系统一共使用3台服务器由一台主节点master和2台从节点slave组成： 后续会用master slave1 和slave2分别代表三台主机的hostname 二、 SSH无密码验证配置 Hadoop运行过程中，需要管理远端Hadoop守护进程，在Hadoop启动以后，NameNode是通过SSH（Secure Shell）来启动和停止各个DataNo
Hadoop大数据集群搭建（超详细）_hadoop集群搭建
最新发布
2401_84181070的博客
04-27 1207
打开C:\Windows\System32\drivers\etc下的hosts文件,添加以下内容（注：如果没有notepad++这类软件，可以通过记事本保存在其他位置，然后拖动到该文件夹下）同样通过mobaxterm的上的SFTP功能（或其他工具）上传到/export/software目录下，然后解压到/export/servers目录下。传完之后要在hadoop02和hadoop03上分别执行 source /etc/profile 命令，来刷新配置文件。在hadoop01上执行。
hadoop 集群搭建
weixin_30361641的博客
02-13 83
master　 192.168.132.131 slave3　　192.168.132.141 slave2　　192.168.132.144 slave3　　192.168.132.142 master和node节点都要添加到/etc/hosts 192.168.132.144 slave2 192.168.132.131 master 192.168.132.142...
3.基于hadoop集群搭建hbase
06-17
### 基于Hadoop集群搭建HBase集群详解 #### 一、引言 随着大数据技术的迅猛发展，海量数据的高效存储与处理成为企业关注的重点。Hadoop作为一款能够处理大量数据的基础框架，被广泛应用于各类场景之中。然而，在...
hadoop集群搭建(超级详细)
10-11
在大数据领域，Hadoop是一个广泛使用的开源框架，用于存储和处理海量数据。本文将详细讲解如何搭建一个...这个超详细的教程覆盖了从零开始到集群搭建完成的全过程，按照步骤执行，可以轻松掌握Hadoop集群搭建技术。
hadoop集群搭建_centos7配置静态IP
01-07
1.网络配置 设为NAT模式 虚拟机中选中节点,设置–>虚拟机设置—>网络适配器—>NAT模式 2.网络配置 打开虚拟机 ,选中需要修改的节点,步骤如下: 编辑—>虚拟网络编辑器—>VMnet8(NAT模式)—>移除...如果是集群配置,其
Hadoop 集群搭建及部分详解
Geek_SQ
06-26 169
环境搭建准备：（所有操作都是在root下） 　　主机两台，我没有采用虚拟机，建议使用最小化安装 1.jdk（每个节点都要配置） 　　去Java官网下后缀名为.tar.gz的包 　　 　　使用FTP工具，或者rz命令进行上传 　　我放在了/opt/Software/Java 下 　　使用 tar -zxvf解压 最小化安装若没提示找不到这个命令 百度安装这个工具 　　编辑/etc/p...
hadoop集群搭建简要记录
独行侠的守望の博客
04-02 130
2019/03/09 21:46 准备4台服务器或者虚拟机【centos7】，分别设置好静态ip【之所以设置静态ip主要就是为了省心！！！】 【 　　centos7下面配置静态IP 参考地址：https://www.linuxidc.com/Linux/2017-10/147449.htm 　　　　cd /etc/sysconfig/network-scripts 　　　　ls 　　　...
Hadoop集群搭建学习
weixin_35894821的博客
03-29 201
需要用到的安装包 JDK hadoop1.安装jdk、a. cp jdk名称 /opt/         【个人习惯 喜欢把安装包放在opt文件目录下，也可以直接在根目录下解压】b. rpm -ivh jdk名称        【解压】c. vi /etc/profile             【添加环境变量】   添加：JAVA_HOME=/usr/java/default     【jdk的...
linux cetos7 配置三个节点配置ssh免密登录 （最简便）
weixin_58220410的博客
04-07 2016
master 192.168.116.200 slave1 192.168.116.201 slave2 192.168.116.202 1.在 master节点生成 密钥对 ssh-keygen -t rsa 过程中需要按三次回车 2.发送到其他节点(每个节点都要生成密钥，然后发送其他节点包括自身) 每次发送都需要按提示 输入yes回车后再输入目标节点的密码 ssh-copy-id root@master ssh-copy-id root@slave1 ssh-copy-id ro
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值
>