集群安装:
1、下载编译
下载hadoop2.6.0,解压
2、修改配置文件
打开hadoop-env.sh(一般在hadoop-2.6.0/etc/hadoop/文件夹下)
修改:
export JAVA_HOME=/A/soft/jdk1.7.0_79
修改core-site.xml,将下面的内容copy到core-site.xml中
<configuration>
<!--指定namenode地址-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://59.67.152.31:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/A/soft/hadoop-2.6.0/data/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>10080</value>
</property>
</configuration>
修改 hdfs-site.xml,将下面的内容copy到dfs-site.xml中
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>59.67.152.31:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>59.67.152.33:50090</value>
</property>
</configuration>
将mapred-site.xml.template修改为mapred-site.xml,修改mapred-site.xml,将将下面的内容copy到mapred-site.xml中
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>59.67.152.31:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>59.67.152.31:19888</value>
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
</configuration>
修改yarn-site.xml,将将下面的内容copy到mapred-site.xml中
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>59.67.152.32</value>
</property>
<property>
<name>yarn.web-proxy.address</name>
<value>59.67.152.32:8888</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<!--yarn的内存-->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
</property>
<!--YARN CPU-->
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>
</configuration>
修改slave,指定计算节点
59.67.152.31
59.67.152.32
59.67.152.33
3、格式化
进入当前安装目录下cd /A/soft/hadoop-2.6.0/
格式化:bin/hadoop namenode -format
格式化完成标志:successfully formated
4、向其他机器发送安装文件
scp -r hadoop-2.6.0/ 59.67.152.32:/A/soft/
scp -r hadoop-2.6.0/ 59.67.152.33:/A/soft/
5、Hadoop启动
在59.67.152.31机器启动服务HDFS
进入Hadoop目录àsbin/start-dfs.sh
在59.67.152.32机器启动yarn
进入Hadoop目录àsbin/start-yarn.sh
在59.67.152.31机器启动jobhistory
进入Hadoop目录àsbin/mr-jobhistory-daemon.sh start historyserver
在59.67.152.32机器启动防护进程
进入Hadoop目录àsbin/yarn-daemon.sh start proxyserver
jps可以查看是否启动成功
首次启动按照以上步骤,之后启动直接可以进入Hadoop目录àsbin/start-all.sh即可
6、集群测试
上传文件file1:bin/hdfs dfs -put /etc/profile /file1 /profile
查看59.67.152.32:8080/cluster/nodes是否存在上传的文件
运行wordcount:bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /profile /out
查看结果bin/hdfs dfs -cat /out/part-r-00000