centos安装 hadoop 3.3.1

centos安装 hadoop 3.3.1

安装前准备

1. 预装jdk环境

参考 [jdk安装教程](https://blog.csdn.net/lingfeian/article/details/104589422)

2. 下载安装包

	[hadoop安装包下载地址](https://hadoop.apache.org/releases.html)

3. 将安装包解压到安装目录

	cd /mnt/servers/haddop/
 tar zxvf hadoop-3.3.1-aarch64.tar.gz 

执行完前面3步，可说我们的准备工作已经做完了，接下来就是真正的表演了。开始hadoop环境的配置相关工作。

hadoop安装

1. hadoop环境配置

在/etc/profile 配置handoop的环境，在文件的末尾添加HADOOP_HOME的配置

编辑完成之后保存,并通过 resource 命令使配置生效。

resource /etc/profile

检查配置是否生效,执行以下命令，结果如下图所示，证明安装成功。

hadoop version

如下图所示，hadoop 3.3.1版本我们已经完成安装。

hadoop 配置

hadoop配置主要文件有4个， 配置文件的目录
/mnt/servers/haddop/hadoop-3.3.1**/etc/hadoop**
不同的安装目录略有差异，自行调整
hdfs-site.xml ： hdfs文件存放路径，配置备份数量

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

hadoop-env.sh ：hadoop运行环境信息，如jdk等 ，不配置，使用默认

 # set to the root of your Java installation
  export JAVA_HOME=/usr/java/latest

core-site.xml ：一个全局的配置，配置默认端口

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

mapred-site.xml ： 配置Reduce的运行方式

hadoop 启动

配置完成后 执行 ./start-all.sh 命令即可启动hadoop

具体的配置参考官文地址：
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

如果启动的时候提示错误：

Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [CentOS-NS02]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2021-10-13 17:15:56,553 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.

那么需要在 hadoop-env.sh 文件中添加默认用户信息

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

配置完成后重新启动即可
可以通多访问 查看hadoop信息
http://localhost:8088/cluster/apps