Hadoop集群安装

已于 2023-01-11 21:15:08 修改
阅读量412 收藏 2
点赞数
文章标签： hadoop java big data
于 2022-06-02 20:44:02 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/maohuihua123/article/details/125106532
版权

Hadoop集群环境安装

准备工作

准备三台机器：

  • 192.168.100.133 hadoop1(master)
  • 192.168.100.134 hadoop2
  • 192.168.100.135 hadoop3

三个前提:

  • 首先保证三个虚拟机上的jdk环境都已安装，并配置了JAVA_HOME。

    安装JDK，安装方式如下或者其他：

    [root@localhost ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel

    通过yum安装JDK，是不会自动配置JAVA_HOME环境变量的，需要手动配置JAVA_HOME，方式如下：

    [root@dc6-80-283 ~]# which java
/usr/bin/java
[root@dc6-80-283 ~]# ll /usr/bin/java
lrwxrwxrwx. 1 root root 22 Jun  1 11:33 /usr/bin/java -> /etc/alternatives/java
[root@dc6-80-283 ~]# ll /etc/alternatives/java
lrwxrwxrwx. 1 root root 74 Jun  1 11:33 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64/jre/bin/java
[root@dc6-80-283 ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64
total 184
-rw-r--r--. 1 root root   1522 May 10 22:52 ASSEMBLY_EXCEPTION
drwxr-xr-x. 2 root root   4096 Jun  1 11:33 bin
drwxr-xr-x. 3 root root    132 Jun  1 11:33 include
drwxr-xr-x. 4 root root     95 Jun  1 11:33 jre
drwxr-xr-x. 3 root root    146 Jun  1 11:33 lib
-rw-r--r--. 1 root root  19274 May 10 22:52 LICENSE
drwxr-xr-x. 2 root root    208 Jun  1 11:33 tapset
-rw-r--r--. 1 root root 157063 May 10 22:52 THIRD_PARTY_README
[root@dc6-80-283 ~]#

    可以确认/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64就应该是的JAVA_HOME了,然后修改环境变量，也就是修改/etc/profile文件。

    vim /etc/profile

    在profile文件的末尾,按i切换到输入模式，插入如下配置

    # java
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin

    修改环境变量以后还需使之生效

    source /etc/profile

    通过echo $JAVA_HOME命令，检验结果，如下说明成功。

    [root@dc6-80-283 ~]# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64

  • 在/etc/hosts里面配置节点和别名（hadoop1，hadoop2，hadoop3）

    [root@dc6-80-283 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.133 hadoop1
192.168.100.134 hadoop2
192.168.100.135 hadoop3

  • 保证三台服务器之间可以相互SSH免密码登录。

    参考教程：https://blog.csdn.net/u010698107/article/details/119079821

下载Hadoop安装包

Hadoop官网：http://hadoop.apache.org
使用的Hadoop版本下载地址：http://archive.apache.org/dist/hadoop/core/hadoop-3.3.1/

解压

  • 新建目录/opt/hadoop

    mkdir /opt/hadoop

  • 将下载好的包解压到/opt/hadoop目录。

    tar -zxvf hadoop-3.3.1.tar.gz

  • 修改文件夹名称

    mv hadoop-3.3.1 hadoop

  • 最终三台机器hadoop都在如下路径

    /opt/hadoop/hadoop

修改配置文件

  • 进入目录 /opt/hadoop/hadoop/etc/hadoop

  • core-site.xml

    <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://hadoop1:9000</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/opt/hadoop/hadoopdata</value>
	</property>
</configuration>

  • hdfs-site.xml

    <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
</configuration>

  • yarn-site.xml

    <?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
 
<!-- Site specific YARN configuration properties -->
 
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop1:18040</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop1:18030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop1:18025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop1:18141</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop1:18088</value>
    </property>
 
</configuration>

  • mapred-site.xml

    <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
 
	<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
	</property>
    
</configuration>

  • 配置workers文件

    注意：在 hadoop3.0 中 slaves 改为了 workers

    vi /opt/hadoop/hadoop/etc/hadoop/workers

  • 加入从服务器名称，如果文件中有localhost这行信息，直接删除掉

    hadoop1
hadoop2
hadoop3

配置系统环境变量

此步骤需要在三个节点中都配置，其他无特殊说明的只需要在主节点master上配置

cd /opt/hadoop 
vim ~/.bash_profile

在其中加入如下内容

#HADOOP
export HADOOP_HOME=/opt/hadoop/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

修改环境变量以后还需使之生效

source ~/.bash_profile

在三个节点中都执行以上命令

创建数据存放目录

mkdir /opt/hadoop/hadoopdata

格式化文件系统

hadoop namenode -format

将配置文件复制到workers节点上

scp -r /opt/hadoop root@hadoop2:/opt
scp -r /opt/hadoop root@hadoop3:/opt

启动和关闭Hadoop集群

启动命令

cd /opt/hadoop/hadoop/sbin
start-all.sh

[root@dc6-80-283 sbin]# start-all.sh
Starting namenodes on [hadoop1]
Last login: Wed Jun  1 15:10:33 CST 2022 on pts/3
Starting datanodes
Last login: Wed Jun  1 15:10:39 CST 2022 on pts/3
Starting secondary namenodes [dc6-80-283.novalocal]
Last login: Wed Jun  1 15:10:41 CST 2022 on pts/3
2022-06-01 15:10:59,223 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Last login: Wed Jun  1 15:10:44 CST 2022 on pts/3
Starting nodemanagers
Last login: Wed Jun  1 15:11:00 CST 2022 on pts/3

关闭命令

cd /opt/hadoop/hadoop/sbin
stop-all.sh

[root@dc6-80-283 sbin]# stop-all.sh
Stopping namenodes on [hadoop1]
Last login: Wed Jun  1 15:11:02 CST 2022 on pts/3
Stopping datanodes
Last login: Wed Jun  1 15:12:26 CST 2022 on pts/3
Stopping secondary namenodes [dc6-80-283.novalocal]
Last login: Wed Jun  1 15:12:26 CST 2022 on pts/3
2022-06-01 15:12:32,120 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping nodemanagers
Last login: Wed Jun  1 15:12:28 CST 2022 on pts/3
hadoop3: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
hadoop2: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
Stopping resourcemanager
Last login: Wed Jun  1 15:12:32 CST 2022 on pts/3

检查是否启动成功

jps

# 主节点
[root@dc6-80-283 sbin]# jps
10672 NodeManager
10145 SecondaryNameNode
10966 Jps
9912 DataNode

# 从节点
[root@dc6-80-275 hadoop]# jps
26458 NodeManager
26286 DataNode
26639 Jps

踩坑

ERROR: Attempting to operate on hdfs namenode as root的方法

  • 描述：hadoop-3.3.1启动hadoop集群时还有可能可能会报如下错误

    [root@localhost sbin]# start-all.sh
Starting namenodes on [hadoop]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hadoop]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-07-16 05:45:04,628 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.

  • 解决方案一

    输入如下命令，在环境变量中添加下面的配置

    vim /etc/profile

    然后向里面加入如下的内容

    export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

    输入如下命令使改动生效

    source /etc/profile

  • 解决方案二

    将start-dfs.sh，stop-dfs.sh(在hadoop安装目录的sbin里)两个文件顶部添加以下参数

    HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

    将start-yarn.sh，stop-yarn.sh(在hadoop安装目录的sbin里)两个文件顶部添加以下参数

    YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

  • 参考文章

    https://blog.csdn.net/a6661314/article/details/124454376

ERROR: JAVA_HOME is not set and could not be found

  • 问题描述：启动时显示如下错误

    [root@dc6-80-283 sbin]# start-all.sh
Starting namenodes on [hadoop1]
Last login: Wed Jun  1 14:47:53 CST 2022 on pts/3
Starting datanodes
Last login: Wed Jun  1 14:58:55 CST 2022 on pts/3
hadoop2: Warning: Permanently added 'hadoop2' (ECDSA) to the list of known hosts.
hadoop2: ERROR: JAVA_HOME is not set and could not be found.
hadoop3: ERROR: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [dc6-80-283.novalocal]
Last login: Wed Jun  1 14:58:57 CST 2022 on pts/3
2022-06-01 14:59:19,231 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Last login: Wed Jun  1 14:58:58 CST 2022 on pts/3
Starting nodemanagers
Last login: Wed Jun  1 14:59:20 CST 2022 on pts/3
hadoop3: ERROR: JAVA_HOME is not set and could not be found.
hadoop2: ERROR: JAVA_HOME is not set and could not be found.

  • 解决方案

    查看JAVA_HOME目录

    [root@dc6-80-283 ~]# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64

    在hadoop的配置目录etc/hadoop中(我的是/opt/hadoop/hadoop/etc/hadoop/)修改hadoop-env.sh配置

    vim /opt/hadoop/hadoop/etc/hadoop/hadoop-env.sh

    把JAVA_HOME换成正确的目录（/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64）

    # Technically, the only required environment variable is JAVA_HOME.
# All others are optional.  However, the defaults are probably not
# preferred.  Many sites configure these options outside of Hadoop,
# such as in /etc/profile.d

# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.aarch64

    参考

    参考教程：https://blog.csdn.net/xiaoluo520112/article/details/118576034

沧浪之水NaN
关注
关于Hadoop全分布构建no HDFS_NAMENODE_USER 错误解决
weixin_51341142的博客
11-22 2674
在Hadoop全分布构建中，常常会出现以下错误 [root@Hadoopc1 hadoop]# start-all.sh Starting namenodes on [192.168.13.245] ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR..
hadoop搭建伪分布环境
枣泥馅的博客
04-14 582
伪分布环境 特点：（1）在单机上模拟一个分布式环境 HDFS：NameNode、DataNode、SecondaryNameNode Yarn：ResourceManager、NodeManager （2）具备Hadoop的所有功能 （3）用于开发和测试 注意...
there is no HDFS_NAMENODE_USER defined
lsl520hah的博客
03-04 5059
一、问题信息：ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation 二、解决方法 在Hadoop安装目录下找到sbin文件夹，修改里面的四个文件 1、对于start-dfs.sh和stop-dfs.sh文件，添加下列参数： #!/usr/bin/env bash HDFS_DATANODE_USER=root HADOOP_SECURE_DN_USER=hdfs HDFS_NAMENODE_USER=ro
大数据分析技术：hadoop伪分布模式安装
u011062044的博客
05-16 849
hadoop伪分布模式安装 1配置核心组件文件 Hadoop 可以在单节点上以伪分布式的方式运行，Hadoop 进程以分离的 Java 进程来运行，节点既作为 NameNode 也作为 DataNode，同时，读取的是 HDFS 中的文件。 通常情况下，Hadoop集群中需要配置的文件主要包括四个，分别是core-site.xml、hdfs-site.xml、mapred-site.xml和yarn-site.xml，这四个文件分别是对不同组件的配置参数。 序号 配置.
启动hdfs报错：Attempting to operate on hdfs namenode as root but there is no HDFS NAMENODE USER defined.
m0_67656269的博客
01-05 2327
hadoop启动的时候报错
hadoop集群安装
12-07
hadoop的分布式安装，三台服务器实现为例，分别是master
Hadoop集群安装和搭建（全面超详细的过程） 文章目录 Hadoop集群安装和搭建（全面超详细的过程） 前言 一、虚拟机的安装
最新发布
09-13
hadoop集群搭建Hadoop集群安装和搭建（全面超详细的过程） 文章目录 Hadoop集群安装和搭建（全面超详细的过程） 前言 一、虚拟机的安装 二、Linux系统安装 1.环境准备 2.虚拟机安装 三、Centos系统安装 四、静态网络...
hadoop集群安装手册
04-07
【Hadoop集群安装手册】 Hadoop是一个开源的分布式计算框架，由Apache基金会开发，主要用于处理和存储海量数据。本手册将详细介绍如何在5台虚拟机上手动安装和配置Hadoop集群，供初学者和专业人士参考。 ### 安装...
(防坑笔记)hadoop3.0 (一) 环境部署与伪分布式(hdfs)
m0_54852350的博客
04-14 1140
防坑留名： 为了避免以后自己遇到什么坑爹的东西，先留脚印给自己。这个hadoop呢，主要是可以让用户可以在不了解分布式底层细节的情况下，开发分布式程序，充分利用集群的威力进行高速运算和存储。这点比较厉害了。 它主要是用来做数据分析，支持低端服务器集群(这点美滋滋- - ），先抓取大量数据，利用数据运算分析，获取日志，显示报表~； 本例子用的 环境： jdk 1.8.0_152 centos7 hadoop3.0 通解：（分布式文件管理系统） 一个电脑或操作系统管辖的范围存不下的文件，分布到更多的操作系统上，
[hadoop 的坑] there is no HDFS_NAMENODE_USER defined
qq_44172732的博客
02-23 8476
问题： ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as roo...
Linux安装配置hadoop
qq_43540354的博客
05-02 3360
安装配置hadoop 1.到 《官网》 下载Hadoop 2.使用cd /usr命令，进入到usr目录下 并使用mkdir hadoop命令，创建一个hadoop目录，使用ls查看是否创建成功（一般没毒的话，应该都会创建成功） 4.将下载好的文件上传到linux环境中，准确说是上一步建好的hadoop目录下，方法有三： (1)可以使用XShell的sudo rz的命令上传，但是有的可能会出现“传输失败”，如果失败了可以换换使用sudo rz -be进行上传 (2)使用XShell配套的xftp进行上
在centOS7 下搭建 HADOOP3.0.3 完全分布式集群环境
tsrj_520的博客
08-15 3109
一、基础环境 在Linux上安装Hadoop之前，需要先安装两个程序： 1.1 安装说明 1. JDK 1.8或更高版本（本文所提到的安装的是jdk1.8）; 2. SSH(安全外壳协议)，推荐安装OpenSSH。 下面简述一下安装这两个程序的原因： 1. Hadoop是用Java开发的，Hadoop的编译及MapReduce的运行都需要使用JDK。 2. Hadoop需要通过SSH...
Hadoop 3.x主要变化(相对于Hadoop 2.x)
猿类崛起
02-01 2518
         今天有人问Hadoop 3.x的主要变动在哪里，这里在官网(http://hadoop.apache.org/docs/r3.0.0/index.html)查了下，总结简单翻译如下： 1.要求JDK&gt;=1.7 2.HDFS支持纠删码        与副本相比纠删码是一种更节省空间的数据持久化存储方法。标准编码(比如Reed-Solomon(10,4))会有 ...
Hadoop3.3.0完全分布式部署
zhuce_2012的专栏
08-05 2826
一、配置hadoop 1.1 总纲 采用1+1+3的集群配置（一台主服务器，一台备主服务器，3台从服务器） 这些配置都只是先在master1作，然后用命SCP拷贝复制到其它服务器 下载最新hadoop镜像：hadoop-3.3.0 解压到相应目录 配置环境变量在/etc/hosts, /etc/profile，hadoop-en.sh, yarn-en.sh, mapreduce-env.sh 修改文件目录权限 设置免密登录 配置相应文件：workers, core-site.xmlhdfs-site.x
基于linux系统安装hadoop+zk+flume+kafka+mysql+hive+redis+es+Rmq+hbase+spark+storm+azkaban
weixin_43432597的博客
11-19 764
基于linux系统安装hadoop+zk+flume+kafka+mysql+hive+redis+es+Rmq+hbase+spark+storm+azkaban 安装zookeeper （1）解压zookeeper安装包（/opt） #tar -zxvf zookeeper-3.4.7.tar.gz （2）将zookeeper解压包移到/usr/house/zookeeper下 mv ...
异常：hadoop启动时namenode节点找不到以至于启动不起来
世界之大
03-08 6737
问题描述：启动hadoop时有三个节点：namenode、datanode和secondarynamenode，结果只有两个启动了，而namenode节点报找不到的错误。输入了jps发现：[root@lmy-pc hadoop]# jps3819 SecondaryNameNode4058 Jps3667 DataNode进入日志logs文件夹中，发现只有datanode和secondearyna...
Hadoop集群安装配置实战指南
"这篇文档是关于Hadoop集群安装配置的精华集锦，主要涵盖了Hadoop的基本概念、集群部署介绍以及环境说明。" 在Hadoop安装配置的过程中，了解Hadoop的基础知识至关重要。Hadoop是一个由Apache软件基金会开发的开源...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值