【大数据基石】Hadoop环境搭建

✨这里是第七人格的博客。小七，欢迎您的到来~✨

🍅系列专栏:【大数据基石】🍅

✈️本篇内容: Hadoop环境搭建✈️

🍱本篇收录完整代码地址：无🍱

前言

本篇文章是基于Centos7搭建的。

配置hosts

vim /etc/hosts

在3台机器上分别修改hosts文件，加入以下内容

192.168.75.3 hadoop01
192.168.75.4 hadoop02
192.168.75.5 hadoop03

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld.servicemkdir -p /mydata/redis/conf

配置SSH免密

ssh-keygen -t rsa

cd /root/.ssh/

cat id_rsa.pub > authorized_keys

cat authorized_keys

下载Hadoop

wget --no-check-certificate https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.2.2/hadoop-3.2.2.tar.gz

解压Hadoop到指定目录

我这里解压到

/usr/local/hadoop-space/

添加环境变量

vi ~/.bash_profile

添加以下内容，注意修改成自己的路径

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export HADOOP_PID_DIR=/usr/local/hadoop-space/
export HADOOP_HOME=/usr/local/hadoop-space/hadoop-3.2.2/

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/

PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$HOME/bin

export PATH

source ~/.bash_profile

修改Hadoop配置文件

进入 $HADOOP_HOME/etc/hadoop目录下。我这里是/usr/local/hadoop-space/hadoop-3.2.2/etc/hadoop/

cd /usr/local/hadoop-space/hadoop-3.2.2/etc/hadoop/

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定 NameNode 的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop01:8020</value>
    </property>
    <!-- 指定 hadoop 数据的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-space/hadoop-3.2.2/data</value>
    </property>
    <!-- 配置 HDFS 网页登录使用的静态用户为当前操作系统用户 -->
    <property>
      <name>hadoop.http.staticuser.user</name>
      <value>root</value>
    </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- NameNode web 端访问地址-->
  <property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop01:9870</value>
  </property>
  <!-- 2NameNode web 端访问地址-->  
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop03:9868</value>
  </property>
  <!-- 备份数量 -->
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <!-- 打开webhdfs -->
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
  <!-- 指定 MR 走 shuffle -->  
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <!-- 指定 ResourceManager 的地址-->  
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop02</value>
  </property>
  <!-- 环境变量的继承 跑示例时要用到-->
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
  <!-- 开启日志聚集功能 -->  
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <!-- 设置日志聚集服务器地址 -->  
  <property>
    <name>yarn.log.server.url</name>
    <value>http://hadoop01:19888/jobhistory/logs</value>
  </property>
  <!-- 设置日志保留时间为 7 天 -->  
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- 指定 MapReduce 程序运行在 Yarn 上 -->  
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <!-- 历史服务器 web 端地址 -->  
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop01:19888</value>
  </property>
</configuration>

workers

hadoop01
hadoop02
hadoop03

hadoop-env.sh

修改Java环境变量

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/

其他2台服务器也这样配置

初始化NameNode

在Hadoop01初始化NameNode

hadoop namenode -format

启动hdfs

在Hadoop01

cd /usr/local/hadoop-space/hadoop-3.2.2/sbin/

start-dfs.sh

这时候访问 http://hadoop01:9870 就可以访问到HDFS的Web页面了。

启动Yarn

切换到Hadoop02的机器上

cd /usr/local/hadoop-space/hadoop-3.2.2/sbin/

start-yarn.sh

这时候访问 http://hadoop02:8088/ 可以进入以下页面

启动历史记录服务器

切换到Hadoop01服务器

mapred --daemon start historyserver

这时候访问 http://hadoop01:19888/ 可以进入以下页面

3881)]