安装Hadoop集群

1.修改主机名：

Vi /etc/sysconfig/network

Reboot

2.IP与主机映射

Vi /etc/hosts

3.配置文件修改：

Core-site.xml:

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://192.168.25.11:9000</value>

<description>默认的HDFS路径</description>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/opt/modules/hadoop-2.7.1/data/tmp</value>

<description>路径默认是NameNode、DataNode等存放数据的公目录</description>

</property>

<property>

<name>hadoop.http.staticuser.user</name>

<value>root</value>

<description>网页界面访问数据使用的用户名</description>

</property>

</configuration>

 

Hdfs-site.xml:

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

<description>备份几份数据</description>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:///opt/modules/hadoop-2.7.1/data/dfs/nn/name</value>

<description>NN使用的元数据保存</description>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:///opt/modules/hadoop-2.7.1/data/dfs/dn</value>

<description>DN使用的数据保存</description>

</property>

<property>

<name>dfs.hosts.exclude</name>

<value>/opt/modules/hadoop-2.7.1/etc/hadoop/dfs-hosts.exclude</value>

<description>增加、卸载节点的文件的路径</description>

</property>

<property>

<name>dfs.namenode.edits.dir</name>

<value>file:///opt/modules/hadoop-2.7.1/data/dfs/nn/edits</value>

<description>edits文件的存储路径</description>

</property>

<property>

<name>dfs.namenode.checkpoint.edits.dir</name>

<value>/opt/modules/hadoop-2.7.1/data/dfs/snn/edits</value>

<description>SecondaryNameNode在文件系统中存储用于合并的临时edits文件的目录</description>

</property>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>master:50090</value>

<description>secondarynamenode提供网络服务的端口和地址</description>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

<description>文件操作时的权限检查标识</description>

</property>

<property>

<name>dfs.namenode.http-address</name>

<value>master:50070</value>

<description>namenode提供网络服务的端口和地址</description>

</property>

</configuration>

Mapred-site.xml:

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

<description>使用yarn运行mapreduce程序</description>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>master:10020</value>

<description>MapReduce的JobHistoryserver的IP和端口</description>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master:19888</value>

<description>MapReduce的JobHistoryserver在web端查看的地址

</description>

</property>

<property>

<name>mapreduce.job.ubertask.enable</name>

<value>true</value>

<description>是否开启小作业优化</description>

</property>

</configuration>

Yarn-site.xml:

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.log-aggregation-enable</name>

<value>false</value>

<description>是否启用日志聚合功能，日志聚合开启后保存到HDFS上</description>

</property>

<property>

<name>yarn.log-aggregation.retain-seconds</name>

<value>604800</value>

<description>聚合后的日志在HDFS上保存多长时间，单位为s </description>

</property>

<property>

<name>yarn.resourcemanager.hostname</name>

<value>slave1</value>

<description>把resourcemanager放到那台机器</description>

</property>

</configuration>

SSH

免密码登录

1.cd .ssh/

如图：

2.ssh-keygen -t rsa之后4个enter键

出现如图效果：

即生成了秘钥

3.复制

Ssh-copy-id root@master

Ssh-copy-id root@slave1

Ssh-copy-id root@slave2

注意：三台机器每台都需要做上述步骤

启动集群前，需要我们格式化集群：

Bin/hdfs namenode –format

启动DFS：sbin/start-dfs.sh

启动yarn：sbin/start-yarn.sh

关闭DFS：sbin/stop-dfs.sh

关闭yarn：sbin/stop-yarn.sh