Hadoop安装与部署

Hadoop安装

Hadoop下载

Hadoop安装

1. 官网下载Hadoop压缩包
   Hadoop官网：hadoop.apache.org
2. 解压Hadoop压缩包到 ~/app目录下

tar -zxvf hadoop-3.3.4.tar.gz -C ~/app 

3. 创建软链接

[hadoop@bigdata31 app]$ ln -s hadoop-3.3.4/ hadoop

Hadoop环境变量配置

在用户家目录下配置

vim ~/.bashrc

#HADOOP_HOME
export HADOOP_HOME=/home/hadoop/app/hadoop
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
source  ~/.bashrc 

vim ~/app/hadoop3.1.3/etc/hadoop/hadoop-env.sh

追加：export JAVA_HOME=/home/hadoop/app/java

配置Hadoop

在 ~/app/hadoop3.1.3/etc/hadoop/ 目录下进行配置

hdfs配置

vim core-site.xml

<property>
        <name>fs.defaultFS</name>
		<value>hdfs://主机名:9000</value>
</property>

vim hdfs-site.xml

<property>
	<name>dfs.replication</name>
	<value>1</value>
</property>

yarn配置

vim mapred-site.xml

	<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>

vim yarn-site.xml

	<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>

ssh免密登录配置

逐行执行

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

测试

[hadoop@bigdata31 hadoop]$ ssh localhost
Last login: Mon Jun  5 03:16:56 2023

免密登录配置成功

Hadoop启动

hdfs格式化：

hdfs namenode -format

启动hdfs：

start-dfs.sh

启动yarn：

start-yarn.sh

web ui 查看

http://IP:9870/