CentOS7搭建大数据环境之路（二）------安装Hadoop

前言

就是记录在CentOS上安装Hadoop的步骤。

步骤

步骤一、安装

下载地址

https://downloads.apache.org/hadoop/common/hadoop-3.2.1/

传到服务器

mkdir /app
rz
选择上传文件即可

解压

tar -zxvf hadoop-3.2.1.tar.gz

安装目录

/app/hadoop-3.2.1

步骤二、配置

JDK

vi /app/hadoop-3.2.1/etc/hadoop/hadoop-env.sh 
找到export JAVA_HOME
修改为：
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64

HADOOP_HOME

vi /etc/profile
export HADOOP_HOME=/app/hadoop-3.2.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
source /etc/profile

伪分布式

修改core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

修改hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/hadoop/hadoopdata/dfs/name</value>
    </property>
    <property>
        <name>dfs.namenode.data.dir</name>
        <value>/data/hadoop/hadoopdata/dfs/data</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:50070</value>
    </property>
</configuration>

创建目录/data/hadoop/hadoopdata/dfs

修改yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

</configuration>

修改mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

步骤三、启动

格式化文件系统

hadoop namenode -format

设置主机允许无密码SSH链接

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

启动hdfs

cd /app/hadoop-3.2.1/sbin
./start-dfs.sh

可能会报如下错误：

[root@localhost sbin]# ./start-dfs.sh 
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

修改start-dfs.sh和stop-dfs.sh，顶部追加如下：

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

修改start-yarn.sh和stop-yarn.sh，顶部追加如下：

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

验证hdfs

[root@localhost sbin]# jps
10064 SecondaryNameNode
10275 Jps
9592 NameNode
9771 DataNode

启动yarn

cd /app/hadoop-3.2.1/sbin
./start-yarn.sh 

验证yarn

[root@localhost sbin]# jps
10064 SecondaryNameNode
13665 ResourceManager
9592 NameNode
14249 Jps
9771 DataNode
13823 NodeManager

页面方式验证

HDFS直接访问ip:50070
YARN直接访问ip:8088