Hadoop伪分布式搭建

1 配置SSH免密登录

        1.生成密钥

# ssh-keygen -t rsa

        注意：需要经过4次回车

        查看密钥及公钥

# cd /root/.ssh

        拷贝公钥

# cp id_rsa.pub authorized_keys

        2 测试本地免密登录

2 下载Hadoop安装包

        使用wget命令从华为云上下载Hadoop安装文件

# wget -P /opt https://mirrors.huaweicloud.com/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

3 安装Hadoop

        1.在opt目录下创建software目录

# mkdir /opt/software

        2.把hadoop2.7.7文件解压到/opt/software目录下

# tar -zxvf /opt/hadoop-2.7.7.tar.gz -C /opt/software

        3.把hadoop-2.7.7目录修改为hadoop

# mv /opt/software/hadoop-2.7.7 /opt/software/hadoop

        4.配置Hadoop环境变量

# vim /etc/profile

        环境变量立即生效

# source /etc/profile

        5.对Hadoop进行配置

        修改hadoop-env.sh文件，添加对jdk的支持

# vim /opt/software/hadoop/etc/hadoop/hadoop-env.sh

        修改core-site.xml

# vim /opt/software/hadoop/etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/software/hadoop/tmp</value>
                <description>配置临时文件存放地址</description>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:8020</value>
                <description>配置HDFS服务地址及端口</description>
        </property>
</configuration>

        修改hdfs-site.xml

# vim /opt/software/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>

        配置mapred-site.xml

        复制模板文件

# cp /opt/software/hadoop/etc/hadoop/mapred-site.xml.template /opt/software/hadoop/etc/hadoop/ mapred-site.xml

        配置文件内容

# vim /opt/software/hadoop/etc/hadoop/mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>

        配置yarn-site.xml

# vim /opt/software/hadoop/etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>3072</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

        6.格式化hdfs

# hdfs namenode -format

        注意：如果不是首次格式化，需要删除本地Hadoop的tmp目录下的所有内容，然后再格式化

        7.启动Hadoop

# start-dfs.sh

# start-yarn.sh

        可以通过jps查看启动情况

        如果出现上述内容，说明启动成功

        8.访问Hadoop

        在Windows系统下通过浏览器访问http://master:8088

        访问http://master:50070