Hadoop分布式集群搭建(云服务器版)

java下载地址

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
本例采用：jdk1.8.0_261.tar.gz

教程

https://www.cnblogs.com/lumama520/p/11058927.html

配置JAVA文件

export JAVA_HOME=/usr/local/java/jdk1.8.0_261
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

建立软连接

ln -s /usr/local/java/jdk1.8.0_261/bin/java /usr/bin/java

配置/etc/hosts

这里注意如果是自己的ip要用自己私网的ip，其余的都用公网ip

eg：
120.79.x.x master #公网ip
172.19.x.x slave1 #私网ip

hadoop免密

#教程

https://www.linuxidc.com/Linux/2019-02/156882.htm

每个节点上：

之后把其他服务器上的id_rsa.pub复制粘贴到authorized_keys中

ssh-keygen -t rsa
cd .ssh
cp id_rsa.pub authorized_keys

之后把其他服务器上的id_rsa.pub复制粘贴到authorized_keys中

#linux新建用户

教程

https://blog.csdn.net/sinat_34886122/article/details/88353871
密码设置为hadoop

adduser hadoop
passwd hadoop

赋权

vi /etc/sudoers

找到## Allows people in group wheel to run all commands这句话
加入
hadoop ALL=(ALL) ALL
用wq!强制保存

hadoop下载

http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common->下载hadoop-2.10.1.tar.gz；

本例采用：hadoop-2.10.1.tar.gz

解压

tar -zxvf hadoop-2.10.1.tar.gz

配置hadoop文件

export HADOOP_HOME=/home/hadoop/hadoop-2.10.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

core-site.xml

<property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hadoop-2.10.1/data/tmp</value>
</property>
<property>
    <name>io.file.buffer.size</name>
    <value>131702</value>
</property>

hdfs.xml

<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/hadoop/hadoop-2.10.1/dfs/name</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hadoop-2.10.1/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>master:50090</value>
</property>
<property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
</property>

mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <final>true</final>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>master:50030</value>
</property>
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
</property>
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
</property>
<property>
    <name>mapred.job.tracker</name>
    <value>http://master:9001</value>
</property>

yarn-site.xml

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
</property>

slaves

删除其中的localhost，添加slave1

配置hadoop.env.sh、yarn-env.sh的JAVA_HOME

取消JAVA_HOME的注释，设置为 export JAVA_HOME=/usr/local/java/jdk1.8.0_261

初始化

hdfs namenode -format

启动

start-all.sh

停止

stop-all.sh