apache hadoop 3.2.0安装

最新推荐文章于 2024-04-25 13:57:42 发布
最新推荐文章于 2024-04-25 13:57:42 发布
阅读量391 收藏
点赞数
分类专栏： hadoop 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_40817778/article/details/119543218
版权

一、环境准备

1、关闭防火墙

systemctl stop firewalld

systemctl disable firewalld.service

2、修改主机名和hosts文件

hostnamectl set-hostname master

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.159.132 master

192.168.159.133 slave1

192.168.159.134 slave2

3、更改selinux 为disabled

vi /etc/selinux/config

SELINUX=disabled

重启生效。（临时生效：setenforce 0）

4、安装并开启服务器http服务

yum install -y httpd

systemctl restart httpd.service

systemctl enable httpd.service

5、安装epel-release

yum install -y epel-release

6、创建需要的目录上传需要的包

mkdir /opt/module mkdir /opt/software

chown hadoop:hadooop -R /opt/*

7、卸载自带的openJDK并安装oracle-j2sdk1.8-1.8.0

rpm -qa | grep -i java | xargs -n1 rpm -e –nodeps

tar -zxvf jdk-8u180-linux-x64.tar.gz -C /user/

二、安装hadoop

1、解压

tar -zxvf hadoop hadoop-3.2.2.tar.gz -C /opt/module/

2、环境变量配置

vim /etc/profile

#JAVA HOME

export JAVA_HOME=/usr/java/jdk1.8.0_181

export PATH=$PATH:$JAVA_HOME/bin

#Hadoop

export HADOOP_HOME=/opt/module/hadoop-3.2.2

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

source /etc/profile

3、测试hadoop、java安装是否成功

4、克隆服务器

5、配置克隆虚拟机的hosts文件和主机名

6、配置主机间hadoop用户ssh免密

[hadoop@master ~]$ ssh-keygen

[hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@slave1

[hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@slave2

[hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@master

7、集群规划

　

master

slave1

slave2

HDFS

NameNode

SecondaryNameNode

　

　

DataNode

DataNode

DataNode

YARM

　

JobHistory Server

Resourcemanager

　

NodeManager

NodeManager

NodeManager

8、配置文件修改（只在一台服务器上修改然后下发）

目录：/opt/module/hadoop-3.2.2/etc/Hadoop

vim hadoop-env.sh

添加：export JAVA_HOME=/usr/java/jdk1.8.0_181

vim core-site.xml

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <!--Configurations for NameNode(SecondaryNameNode)、DataNode、NodeManager:-->

  <!-- 指定 NameNode 的地址 -->

  <property>

    <name>fs.defaultFS</name>

    <value>hdfs://master:9000</value>

    <description>NameNode URI</description>

  </property>

  <!-- 指定 hadoop 数据的存储目录 -->

  <property>

  <name>hadoop.tmp.dir</name>

  <value>/opt/module/hadoop-3.2.2/data</value>

  </property>

  <!-- 配置 HDFS 网页登录使用的静态用户为 hadoop -->

  <property>

  <name>hadoop.http.staticuser.user</name>

  <value>hadoop</value>

  </property>

  <property>

    <name>io.file.buffer.size</name>

    <value>131072</value>

    <description>Size of read/write buffer used in SequenceFiles,The default value is 131072</description>

  </property>

</configuration>

vim hdfs-site.xml

<configuration>

  <!--Configurations for NameNode:-->

  <!--nn 目录 -->

  <property>

    <name>dfs.namenode.name.dir</name>

    <value>file:///opt/module/hadoop-3.2.2/namenode</value>

  <!-- nn web 端访问地址-->

  <property>

  <name>dfs.namenode.http-address</name>

  <value>master:9870</value>

  </property>

  </property>

  <!-- 2nn web 端访问地址-->

  <property>

    <name>dfs.namenode.secondary.http-address</name>

    <value>slave1:50090</value>

  </property>

  <!-- 副本个数-->

  <property>

    <name>dfs.replication</name>

    <value>3</value>

  </property>

  <!-- 块大小-->

  <property>

    <name>dfs.blocksize</name>

    <value>268435456</value>

  </property>

  <property>

    <name>dfs.namenode.handler.count</name>

    <value>100</value>

  </property>

</configuration>

vim yarn-site.xml

<configuration>

  <!--Configurations for ResourceManager and NodeManager:-->

  <property>

    <name>yarn.acl.enable</name>

    <value>false</value>

    <description>Enable ACLs? Defaults to false. The value of the optional is "true" or "false"</description>

  </property>

        <property>

    <name>yarn.admin.acl</name>

    <value>*</value>

    <description>ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of jus

t space means no one has access</description>

  </property>

  <property>

    <name>yarn.log-aggregation-enable</name>

    <value>true</value>

    <description>Configuration to enable or disable log aggregation</description>

  </property>

  <!--Congrations for ResourceManager:-->

  <property>

    <name>yarn.resourcemanager.address</name>

    <value>slave2:8032</value>

    <description>ResourceManager host:port for clients to submit jobs.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.</description>

  </property>

  <property>

    <name>yarn.resourcemanager.scheduler.address</name>

    <value>slave2:8030</value>

    <description>ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostna

me</description>

  </property>

  <property>

    <name>yarn.resourcemanager.resource-tracker.address</name>

    <value>slave2:8031</value>

    <description>ResourceManager host:port for NodeManagers.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname</description>

  </property>

  <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>slave2:8033</value>

        <description>ResourceManager host:port for administrative commands.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.</description>

  </property>

  <property>

    <name>yarn.resourcemanager.webapp.address</name>

    <value>slave2:8088</value>

    <description>ResourceManager web-ui host:port. NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname</description>

  </property>

  <property>

    <name>yarn.resourcemanager.hostname</name>

    <value>slave2</value>

    <description>ResourceManager host</description>

  </property>

  <property>

    <name>yarn.resourcemanager.scheduler.class</name>

 <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

    <description>ResourceManager Scheduler class CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler.The default value is "org.apache.hadoop.yarn.server.res

ourcemanager.scheduler.capacity.CapacityScheduler".

    </description>

  </property>

  <property>

    <name>yarn.scheduler.minimum-allocation-mb</name>

    <value>512</value>

    <description>Minimum limit of memory to allocate to each container request at the Resource Manager.NOTES:In MBs</description>

  </property>

  <property>

    <name>yarn.scheduler.maximum-allocation-mb</name>

    <value>1024</value>

    <description>Maximum  limit of memory to allocate to each container request at the Resource Manager.NOTES:In MBs</description>

  </property>

  <!--Congrations for History Server:-->

  <property>

    <name>yarn.log-aggregation.retain-seconds</name>

    <value>-1</value>

    <description>How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.</description>

  </property>

  <property>

    <name>yarn.log-aggregation.retain-check-interval-seconds</name>

    <value>-1</value>

    <description>Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful,

 set this too small and you will spam the name node.</description>

  </property>

  <!--Configurations for Configurations for NodeManager:-->

  <property>

    <name>yarn.nodemanager.resource.memory-mb</name>

    <value>1024</value>

    <description>Resource i.e. available physical memory, in MB, for given NodeManager.

        The default value is 8192.

        NOTES:Defines total available resources on the NodeManager to be made available to running containers

    </description>

  </property>

  <property>

    <name>yarn.nodemanager.vmem-pmem-ratio</name>

    <value>2.1</value>

    <description>Maximum ratio by which virtual memory usage of tasks may exceed physical memory.

        The default value is 2.1

        NOTES:The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its p

hysical memory usage by this ratio.

    </description>

  </property>

  <property>

    <name>yarn.nodemanager.local-dir</name>

    <value>${hadoop.tmp.dir}/nm-local-dir</value>

    <description>Comma-separated list of paths on the local filesystem where intermediate data is written.

        The default value is "${hadoop.tmp.dir}/nm-local-dir"

        NOTES:Multiple paths help spread disk i/o.

    </description>

  </property>

  <property>

    <name>yarn.nodemanager.log-dirs</name>

    <value>${yarn.log.dir}/userlogs</value>

    <description>Comma-separated list of paths on the local filesystem where logs are written

        The default value is "${yarn.log.dir}/userlogs"

        NOTES:Multiple paths help spread disk i/o.

        </description>

  </property>

   <property>

    <name>yarn.nodemanager.log.retain-seconds</name>

    <value>10800</value>

    <description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.

        The default value is "10800"

    </description>

  </property>

  <property>

    <name>yarn.application.classpath</name>

    <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>

  </property>

   <property>

    <name>yarn.nodemanager.remote-app-log-dir</name>

    <value>/logs</value>

    <description>HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled.

    The default value is "/logs" or "/tmp/logs"

    </description>

  </property>

   <property>

    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>

    <value>logs</value>

    <description>Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled

.</description>

  </property>

   <property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

    <description>Shuffle service that needs to be set for Map Reduce applications.</description>

  </property>

</configuration>

vim mapred-site.xml

<configuration>

  <!--Configurations for MapReduce Applications:-->

  <property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

    <description>Execution framework set to Hadoop YARN.</description>

  </property>

  <property>

    <name>mapreduce.map.memory.mb</name>

    <value>512</value>

    <description>Larger resource limit for maps.</description>

  </property>

  <property>

    <name>mapreduce.map.java.opts</name>

    <value>Xmx1024M</value>

    <description>Larger heap-size for child jvms of maps.</description>

  </property>

  <property>

    <name>mapreduce.reduce.memory.mb</name>

    <value>512</value>

    <description>Larger resource limit for reduces.</description>

  </property>

  <property>

    <name>mapreduce.reduce.java.opts</name>

    <value>Xmx1024M</value>

    <description></description>

  </property>

  <property>

    <name>mapreduce.task.io.sort.mb</name>

    <value>512</value>

    <description></description>

  </property>

  <property>

    <name>mapreduce.task.io.sort.factor</name>

    <value>10</value>

    <description>More streams merged at once while sorting files.</description>

  </property>

  <property>

    <name>mapreduce.reduce.shuffle.parallelcopies</name>

    <value>5</value>

    <description>Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.</description>

  </property>

  <!--Configurations for MapReduce JobHistory Server:-->

        <property>

    <name>mapreduce.jobhistory.address</name>

    <value>slave1:10020</value>

    <description>MapReduce JobHistory Server host:port Default port is 10020</description>

  </property>

  <property>

    <name>mapreduce.jobhistory.webapp.address</name>

    <value>slave1:19888</value>

    <description>MapReduce JobHistory Server Web UI host:port Default port is 19888</description>

  </property>

  <property>

    <name>mapreduce.jobhistory.intermediate-done-dir</name>

    <value>/opt/module/hadoop-3.2.2/mr-history/tmp</value>

    <description>Directory where history files are written by MapReduce jobs. Defalut is "/mr-history/tmp"</description>

  </property>

  <property>

    <name>mapreduce.jobhistory.done-dir</name>

    <value>/opt/module/hadoop-3.2.2/mr-history/done</value>

    <description>Directory where history files are managed by the MR JobHistory Server.Default value is "/mr-history/done"</description>

  </property>

</configuration>

vim workers

master

slave1

slave2

9、初始化集群并启动

cd /opt/module/hadoop-3.2.2

hdfs namenode -format

[hadoop@master hadoop-3.2.2]$ sbin/start-dfs.sh

Starting namenodes on [master]

Starting datanodes

slave2: WARNING: /opt/module/hadoop-3.2.2/logs does not exist. Creating.

slave1: WARNING: /opt/module/hadoop-3.2.2/logs does not exist. Creating.

Starting secondary namenodes [slave1]

[hadoop@salve2 hadoop-3.2.2]$ sbin/start-yarn.sh

Starting resourcemanager

Starting nodemanagers

[hadoop@slave1 sbin]$ mr-jobhistory-daemon.sh start historyserver

WARNING: Use of this script to start the MR JobHistory daemon is deprecated.

WARNING: Attempting to execute replacement "mapred --daemon start" instead.

注意注意：启动服务的时候，需要在相应的主节点去启动。

启动情况：服务器上jps

网页查看：NameNode:9870

ReasourceManager:8088/cluster

JobHistory:19888

NodeManager:8042

无极_之道
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hadoop完全分布式配置
sailing124856的博客
04-11 1501
Hadoop完全分布式配置环境准备Hadoop3.2.1VMwork+Centos7VMwork安装Linux系统（centos7）网络配置(centos7)二级目录三级目录 centos7+VMwork+Hadoop3.2.1+JDK 准备三台客户机（静态ip、关闭防火墙、主机名称） 环境准备 Hadoop3.2.1 进入Apache hadoop官网，选择hadoop 3.2.1 Binary...
hadoop-3.2.0.tar.gz
07-26
Hadoop是Apache软件基金会开发的一个开源分布式计算框架，它的核心设计是解决大数据处理的问题。Hadoop 3.2.0是Hadoop发展过程中的一个重要版本，它在前一个版本的基础上进行了许多优化和改进，旨在提高系统的性能、...
解决Hadoop无法启动JobHistoryServer
热门推荐
大佛拈花的博客
03-01 1万+
gosaint@gosaint:/usr/local/hadoop$ sbin/mr-jobhistory-daemon.sh start ==== 出现如下问题: WARNING: Use of this script to start the MR JobHistory daemon is deprecated. WARNING: Attempting to execute replace...
centos 部署Hadoop-3.0-高性能集群（一）安装
冰恋云的专栏
10-18 2611
Hadoop概述：Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。 Hadoop的框架最核心的设计就是：HDFS和MapReduce。HDFS分布式文件系统为海量的数据提供了存储，则MapReduce为海量的数据提供了计算。 网方网站：http://hadoop.apache.org...
HADOOP安装详细步骤
最新发布
l996833997的博客
04-25 2039
上述目录中/input 目录是输入数据存放的目录，/output 目录是输出数据存放的目录。重启 SSH 服务，并切换到hadoop用户下验证能否嵌套登录本机，若可以不输入密码登录，则本机通过密钥登录认证成功。在master,slave1,slave2上将 id_rsa.pub 追加到授权 key 文件中。将master,slave1,slave2都转到用户hadoop下。配置 slaves 文件，添加slave1，slave2的ip地址。查看返回值中有没有master，slave1，slave2。
Hadoop3.2.0详细安装步骤
Maqiuqiu520的博客
12-05 1322
一、环境准备 ①准备三台虚拟机，查看ip ②先修改主机名(每个节点统一命名规范) 修改/etc/hosts文件：（如下） 172.18.96.1 master 172.18.96.2 slave1 172.18.96.3 slave3 二、永久关闭防火墙 systemctl stop firewalld systemctl disable firewalld 三、配置ssh免密登录 三台机器都运...
Hadoop集群搭建安装教程（详细完整）
zhang_232的专栏
11-21 2885
Hadoop是Apache旗下的一个用java语言实现开源软件框架，是一个开发和运行处理大规模数据的软件平台。
hadoop-3.2.0winutils.rar
11-28
Hadoop是Apache软件基金会开发的一个开源分布式计算框架，它允许在大规模集群中高效处理和存储大量数据。在Windows环境下配置和使用Hadoop可能会遇到一些挑战，因为Hadoop最初是为Linux设计的。针对Windows用户，...
spark-3.2.0-bin-hadoop3-without-hive
04-24
Hadoop 3 是 Apache Hadoop 的一个重要里程碑，它在扩展性、容错性和资源管理上有所提升。Hadoop 3 支持多活 NameNode，增强了集群的可用性；YARN（Yet Another Resource Negotiator）进行了升级，提高了资源调度...
hadoop3.x带snappy(可用于windows本地开发)
10-26
Hadoop是一个开源的大数据处理框架，由Apache软件基金会开发，它使得在分布式环境中存储和处理大规模数据变得可能。而Snappy是一种高效的压缩和解压缩库，旨在提高数据处理的速度。 【描述】"编译过的hadoop3.0版本...
spark-3.2.0-bin-hadoop3.2.tgz
01-04
这个压缩包"spark-3.2.0-bin-hadoop3.2.tgz"包含了Spark 3.2.0版本的二进制文件，以及针对Hadoop 3.2的兼容构建。 Spark的核心组件包括：Spark Core、Spark SQL、Spark Streaming、MLlib（机器学习库）和GraphX（图...
Hadoop(01) Hadoop3.3.6安装教程，单机/伪分布式配置
weixin_38735917的博客
09-26 6378
在安装 Hadoop 3.3.6 前，需要满足以下前置条件：Java Development Kit (JDK)：Hadoop 是用 Java 编写的，因此需要安装并配置适当版本的 JDK。Hadoop 3.3.6 建议使用 JDK 8 或更高版本。确保正确安装 JDK，并设置 JAVA_HOME 环境变量。SSH：Hadoop 集群中的节点需要通过 SSH 进行通信和管理。确保在所有节点上安装和配置了 SSH，并且可以通过 SSH 无密码登录到集群的各个节点。
hadoop的安装和配置
05-07 626
hadoop安装 在Apache Hadoop主页的下载页面https://hadoop.apache.org/releases.html选择版本进行下载： 下载下来的是压缩包： 将压缩包使用Xftp上传hadoop-1的/usr目录下： 执行命令： 1 # tar -zxvf hadoop-2.7.7.tar.gz 解压完成后会在/usr目录下生成hadoop-2.7.7目录...
Hadoop安装及使用
huivictor的博客
10-09 2041
（1）Hadoop项目为可靠、可扩展的分布式计算开发开源软件。（2）Hadoop 软件库是一个框架，它允许使用简单的编程模型跨计算机集群分布式处理大型数据集。它旨在从单个服务器扩展到数千台机器，每台机器都提供本地计算和存储。库本身不是依靠硬件来提供高可用性，而是设计用于检测和处理应用层的故障，因此在计算机集群之上提供高可用性服务，每台计算机都可能容易出现故障。（4）Hadoop项目模块：Hadoop 分布式文件系统 (HDFS)：一种分布式文件系统，提供对程序数据高吞吐量访问（存储）。
hadoop3.2.2集群搭建
m0_67393828的博客
03-25 2292
说明：本文以HDFS High Availability Using the Quorum Journal Manager模式搭建hadoop3.2.0集群。 环境 CentOS7.5.1804、jdk1.8.0_181、zookeeper3.6.2、hadoop3.2.2 虚拟机安装CentOS7五台，如下： hostname IP roles node-1 192.168.56.129 QuorumPeerMain、NameNode、JournalNode、DFSZKFailoverControll
目前apache hadoop发布的版本主要有
03-16
Apache Hadoop发布的版本主要有以下几个： 1. Hadoop 1.x系列：包括1.0.0、1.1.0、1.2.0等版本，是最早的Hadoop版本，主要用于处理大规模数据。 2. Hadoop 2.x系列：包括2.0.0、2.1.0、2.2.0等版本，是Hadoop的...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值