Hadoop分布式部署

假设有三台虚拟机其ip分别为（在别的网络环境以当时ip地址为准）：
master:192.168.1.140
slave1:192.168.1.141
slave2:192.168.1.142

配置主机名（分别在各自主机中运行上述命令，改为各自的主机名）

$ vi /etc/hostnname

修改每台机器的hosts文件，将主机名及ip地址映射填写进去

$ vi /etc/hosts

## 在文件底部添加:
192.168.1.140 master
192.168.1.141 slave1
192.168.1.142 slave2

配置后可用ping命令检测是否能够连通

$ ping master
$ ping slave1
$ ping slave2

配置免密登录

$ ssh-keygen -t rsa
$ cd /root/.ssh
$ cat id_rsa.pub >> ./authorized_keys

运行过ssh-keygen后会产生俩个文件

• id_rsa私钥
• id_rsa.pub公钥

将master的公钥传送给slave节点

配置jdk

$ vi /etc/profile
### 在底部追加:
export JAVA_HOME=/usr/local/share/jdk
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/home/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

解压Hadoop压缩包

暂定hadoop目录为/home

$ cd /home/hadoop/etc/hadoop
$ vi hadoop-env.sh
### 补齐JAVA_HOME HADOOP_HOME

### 创建一个临时文件路径
mkdir /home/tmp

配置core-site.xml

<property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/tmp</value>
</property>

配置hdfs-site.xml

<property>
    <name>dfs.name.dir</name>
    <value>/home/hadoop/hdfs/name</value>
</property>
<property>
    <name>dfs.data.dir</name>
    <value>/home/hadoop/hdfs/data</value>
</property>

配置mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
</property>
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
</property>

配置yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>master:8032</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:8031</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
    <value>master:8088</value>
    </property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

将设置完成的hadoop发送至slave1和slave2对应的路径上

添加USER定义

$ vi start-dfs.sh
#!/usr/bin/env bash
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

$ vi stop-dfs.sh
#!/usr/bin/env bash
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

$ vi start-yarn.sh stop-yarn.sh
#!/usr/bin/env bash
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

$ vi stop-yarn.sh
#!/usr/bin/env bash
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

查看服务

$ jps