centos安装 hadoop 3.3.1
安装前准备
1. 预装jdk环境
参考 [jdk安装教程](https://blog.csdn.net/lingfeian/article/details/104589422)
2. 下载安装包
[hadoop安装包下载地址](https://hadoop.apache.org/releases.html)
3. 将安装包解压到安装目录
cd /mnt/servers/haddop/
tar zxvf hadoop-3.3.1-aarch64.tar.gz
执行完前面3步,可说我们的准备工作已经做完了,接下来就是真正的表演了。开始hadoop环境的配置相关工作。
hadoop安装
1. hadoop环境配置
在/etc/profile 配置handoop的环境,在文件的末尾添加HADOOP_HOME的配置
编辑完成之后保存,并通过 resource 命令使配置生效。
resource /etc/profile
检查配置是否生效,执行以下命令,结果如下图所示,证明安装成功。
hadoop version
如下图所示,hadoop 3.3.1版本我们已经完成安装。
hadoop 配置
hadoop配置主要文件有4个, 配置文件的目录
/mnt/servers/haddop/hadoop-3.3.1**/etc/hadoop**
不同的安装目录略有差异,自行调整
hdfs-site.xml : hdfs文件存放路径,配置备份数量
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
hadoop-env.sh :hadoop运行环境信息,如jdk等 ,不配置,使用默认
# set to the root of your Java installation
export JAVA_HOME=/usr/java/latest
core-site.xml :一个全局的配置,配置默认端口
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
mapred-site.xml : 配置Reduce的运行方式
hadoop 启动
配置完成后 执行 ./start-all.sh 命令即可启动hadoop
具体的配置参考官文地址:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
如果启动的时候提示错误:
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [CentOS-NS02]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2021-10-13 17:15:56,553 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
那么需要在 hadoop-env.sh 文件中添加默认用户信息
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
配置完成后重新启动即可
可以通多访问 查看hadoop信息
http://localhost:8088/cluster/apps