1.准备环境
以三台为例:
1)准备3台客户机(关闭防火墙、静态ip、主机名称)
2)配置ssh
3)安装JDK
4)安装Hadoop
5)配置环境变量
6)配置集群
7)群起并测试集群
2.配置ssh
参考:https://blog.csdn.net/Aeve_imp/article/details/105393061
3.安装JDK
省略
4.安装Hadoop
- 上传并解压
tar -zxvf hadoop-2.7.2.tar.gz -C /opt/software/
- 配置环境变量
vi ~/.bash_profile
export JAVA_HOME=/opt/software/jdk1.8.0_181
export HADOOP_HOME=/opt/software/hadoop-2.7.7
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:
source ~/.bash_profile,刷新环境变量,查看版本
java -version
hadoop version
- 修改hadoop-env.xml
在javaHome那里添加javaHome的位置,设置hadoop pid文件夹,默认存储在tmp目录下,会被系统清理,导致后面不能群关集群,vi hadoop-env.sh
export JAVA_HOME=/opt/software/jdk1.8.0_181
export HADOOP_PID_DIR=/opt/software/hadoop-2.7.2/pids
- 修改core-site.xml,添加如下配置
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop102:9000</value>
</property>
<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/software/hadoop-2.7.2/data/tmp</value>
</property>
<!-- 垃圾桶 -->
<property>
<name>fs.trash.interval</name>
<value>1440</value>
</property>
- 修改hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!-- 指定Hadoop辅助名称节点主机配置 -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop104:50090</value>
</property>
<!-- 安装用户是谁,xxx改成谁 -->
<property>
<name>hadoop.proxyuser.bigdata.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.bigdata.groups</name>
<value>*</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>true</value>
</property>
- 修改mapred-site.xml
<!-- 指定MR运行在Yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
- 修改yarn-site.xml
<!-- Reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop103</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://xxx:19888/jobhistory/logs</value>
</property>
- 修改slave,删除localhost,添加如下:
hadoop0002
hadoop0003
hadoop0004
- 分发文件
scp -r hadoop-2.7.2 用户名@主机名:$PWD
- namenode 格式化
hadoop namenode -format
5.集群时间同步
date -s "xxxx-xx-xx xx:xx:xx"
6.群起集群
start-dfs.sh
start-yarn.sh
# NameNode和ResourceManger如果不是同一台机器,
#不能在NameNode上启动 YARN,应该在ResouceManager所在的机器上启动YARN
7.验证
进入页面查看状态,hdfs端口50070,yarn端口8088
http://xxxx:50070
http://xxxx:8088
使用hadoop example测试wordcount:
#编辑文件,上传hdfs,提交任务
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount wcinput wcoutput