前言
就是记录在CentOS上安装Hadoop的步骤。
步骤
步骤一、安装
下载地址
https://downloads.apache.org/hadoop/common/hadoop-3.2.1/
传到服务器
mkdir /app
rz
选择上传文件即可
解压
tar -zxvf hadoop-3.2.1.tar.gz
安装目录
/app/hadoop-3.2.1
步骤二、配置
JDK
vi /app/hadoop-3.2.1/etc/hadoop/hadoop-env.sh
找到export JAVA_HOME
修改为:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64
HADOOP_HOME
vi /etc/profile
export HADOOP_HOME=/app/hadoop-3.2.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
source /etc/profile
伪分布式
修改core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
修改hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/hadoop/hadoopdata/dfs/name</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>/data/hadoop/hadoopdata/dfs/data</value>
</property>
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
</configuration>
创建目录/data/hadoop/hadoopdata/dfs
修改yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
修改mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
步骤三、启动
格式化文件系统
hadoop namenode -format
设置主机允许无密码SSH链接
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
启动hdfs
cd /app/hadoop-3.2.1/sbin
./start-dfs.sh
可能会报如下错误:
[root@localhost sbin]# ./start-dfs.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
修改start-dfs.sh和stop-dfs.sh,顶部追加如下:
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
修改start-yarn.sh和stop-yarn.sh,顶部追加如下:
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
验证hdfs
[root@localhost sbin]# jps
10064 SecondaryNameNode
10275 Jps
9592 NameNode
9771 DataNode
启动yarn
cd /app/hadoop-3.2.1/sbin
./start-yarn.sh
验证yarn
[root@localhost sbin]# jps
10064 SecondaryNameNode
13665 ResourceManager
9592 NameNode
14249 Jps
9771 DataNode
13823 NodeManager
页面方式验证
HDFS直接访问ip:50070
YARN直接访问ip:8088