hadoop集群安装指南-hadoop3.2.4版本

champion_风雨中

已于 2022-09-19 17:53:46 修改

阅读量1.8k

点赞数 1

分类专栏：数据中台文章标签： hadoop hdfs

于 2022-09-19 17:45:59 首次发布

本文链接：https://blog.csdn.net/champion2009/article/details/126938328

版权

数据中台专栏收录该内容

1 篇文章 0 订阅

订阅专栏

hadoop3.2.4集群安装指南

下载地址：https://archive.apache.org/dist/hadoop/common/

前置准备

     三台服务器
     10.243.80.42 
     10.243.80.43 
     10.243.80.44

1:安装jdk1.8

2:修改主机名方便集群间的访问

 hostname hadoop42
 hostname hadoop43
 hostname hadoop44
 同步修改/etc/hosts文件
 10.243.80.42 hadoop42
 10.243.80.43 hadoop43
 10.243.80.44 hadoop44

3:配置SSH免密码远程登录

     一、创建公钥/私钥
        ssh-keygen -t rsa 然后yes加一路回车
    二、将公钥拷贝到要免密登录的目标机器上
       ssh-copy-id hadoop42
       ssh-copy-id hadoop43
       ssh-copy-id hadoop44

4:创建hadoop用户

1：增加hadoop用户和密码
adduser hadoop
passwd hadoop
2：把hadoop加入hadoop用户组
usermod -a -G hadoop hadoop
3：然后用cat /etc/group |grep hadoop 查看结果
4：增加hadoop的权限
vi  /etc/sudores
hadoop ALL=(ALL)       ALL
5：将hadoop文件夹的所有者指定为hadoop用户
chown -R hadoop:hadoop /data01/bigdata/hadoop-3.2.4

hadoop安装

1.下载解压hadoop包

下载地址：http://archive.apache.org/dist/hadoop/core/
tar -no-same-owner -zxvf hadoop-3.2.4.tar.gz
创建软连接---方便版本升级
ln -s hadoop-3.2.4 hadoop
chown -R hadoop:hadoop /data01/bigdata/hadoop

2.修改环境变量

vim /etc/profile 添加
export HADOOP_HOME=/data01/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

3:Hadoop集群配置=HDFS集群配置 + MapReduce集群配置 + Yarn集群配置

3.1：HDFS集群配置

1. 将JDK路径明确配置给HDFS（修改hadoop-env.sh）
    cd etc/hadoop
    vim hadoop-env.sh
    export JAVA_HOME=/usr/local/java 
2. 指定NameNode节点以及数据存储目录（修改core-site.xml）
   <property>
     <name>hadoop.tmp.dir</name>
     <value>/data01/bigdata/hadoop/tmp</value>
     <description>Abase for other temporary directories.</description>
   </property>

  <property>
    <name>fs.default.name</name>
    <value>hdfs://hadoop42:9000</value>
  </property>
3. 指定SecondaryNameNode节点（修改hdfs-site.xml）
<property>
   <name>dfs.name.dir</name>
   <value>/data01/bigdata/hadoop/dfs/name</value>
   <description>Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently.</description>
</property>
 
<property>
   <name>dfs.data.dir</name>
   <value>/data01/bigdata/hadoop/dfs/data</value>
   <description>Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks.</description>
</property>

 <!-- 指定Hadoop辅助名称节点主机配置 -->
<property>
   <name>dfs.namenode.secondary.http-address</name>
   <value>hadoop43:50090</value>
</property>
 
  <property>
    <name>dfs.namenode.servicerpc-address</name>
    <value>hadoop42:8022</value>
  </property>
  <property>
    <name>dfs.https.address</name>
    <value>hadoop42:50470</value>
  </property>
  <property>
    <name>dfs.https.port</name>
    <value>50470</value>
  </property>
<!-- 配置namenode的web界面-->
<property>
   <name>dfs.namenode.http-address</name>
   <value>hadoop42:50070</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>3</value>
</property>
4. 指定DataNode从节点（修改workers文件，每个节点配置信息占一行）
 hadoop42
 hadoop43
 hadoop44

3.2：mapreduce集群配置

  1. 将JDK路径明确配置给MapReduce（修改mapred-env.sh）
     export JAVA_HOME=/usr/local/java
  2. 指定MapReduce计算框架运行Yarn资源调度框架（修改mapred-site.xml）
    <property>
	   <name>mapred.job.tracker</name>
	   <value>hadoop42:49001</value>
	</property>
	<property>
	   <name>mapred.local.dir</name>
	   <value>/data01/bigdata/hadoop/var</value>
	</property>
	<property>
	   <name>mapreduce.framework.name</name>
	   <value>yarn</value>
	</property>

3.3:Yarn集群配置

 1. 将JDK路径明确配置给Yarn（修改yarn-env.sh）
  export JAVA_HOME=/usr/local/java
 2. 指定ResourceManager老大节点所在计算机节点（修改yarn-site.xml）
   <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop42</value>
   </property>
 
   <property>
        <description>The address of the applications manager interface in the RM.</description>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
   </property>
 
   <property>
        <description>The address of the scheduler interface.</description>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
   </property>
 
   <property>
        <description>The http address of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
   </property>
 
   <property>
        <description>The https adddress of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>${yarn.resourcemanager.hostname}:8090</value>
   </property>
 
   <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
   </property>
 
   <property>
        <description>The address of the RM admin interface.</description>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
   </property>
 
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
   </property>
 
   <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>10240</value>
        <discription>每个节点可用内存,单位MB,默认8182MB</discription>
   </property>
 
   <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
   </property>
 
   <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>10240</value>
   </property>
   <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
   </property>
 3. 修改start-dfs.sh，stop-dfs.sh文件，在文件头部添加以下配置
   cd $HADOOP_HOME/sbin
   vim start-dfs.sh 和 vim stop-dfs.sh
    
	HDFS_NAMENODE_USER=hadoop
	HDFS_DATANODE_USER=hadoop
	HDFS_SECONDARYNAMENODE_USER=hadoop
	YARN_RESOURCEMANAGER_USER=hadoop
	YARN_NODEMANAGER_USER=hadoop
4.修改start-yarn.sh，stop-yarn.sh文件，在文件头部添加以下配置
	vim start-yarn.sh 和 vim stop-yarn.sh
	 
	RN_RESOURCEMANAGER_USER=hadoop
	HADOOP_SECURE_DN_USER=yarn
	YARN_NODEMANAGER_USER=hadoop
	YARN_RESOURCEMANAGER_USER=hadoop

4.分发配置

scp -r hadoop-3.2.4 hadoop43:/data01/bigdata   
scp -r hadoop-3.2.4 hadoop44:/data01/bigdata 

scp /etc/profile hadoop43:/etc/profile 
scp /etc/profile hadoop44:/etc/profile

5.启动服务

hadoop格式化

cd $HADOOP_HOME
bin/hadoop namenode -format

启动hadoop服务

cd $HADOOP_HOME
sbin/start-dfs.sh

启动yarn服务

cd $HADOOP_HOME
bin/start-yarn.sh

hadoop和yarn访问验证

hadoop访问地址：hadoop42:50070或10.243.80.42:50070
yarn访问地址：hadoop42:8088或10.243.80.42:8088