MR的搭建

MR的搭建

:hadoop 2.x yarn

	部署&配置:
		cd $HADOOP_HOME/etc/hadoop
		vi mapred-site.xml
			    <property>
					<name>mapreduce.framework.name</name>
					<value>yarn</value>
				</property>
		
		vi yarn-site.xml
			    <property>
					<name>yarn.nodemanager.aux-services</name>
					<value>mapreduce_shuffle</value>
				</property>
				
				 <property>
				   <name>yarn.resourcemanager.ha.enabled</name>
				   <value>true</value>
				 </property>
				 <property>
				   <name>yarn.resourcemanager.zk-address</name>
				   <value>node02:2181,node03:2181,node04:2181</value>
				 </property>
				 
				 <property>
				   <name>yarn.resourcemanager.cluster-id</name>
				   <value>cluster1</value>
				 </property>
				 
				 <property>
				   <name>yarn.resourcemanager.ha.rm-ids</name>
				   <value>rm1,rm2</value>
				 </property>
				 <property>
				   <name>yarn.resourcemanager.hostname.rm1</name>
				   <value>node03</value>
				 </property>
				 <property>
				   <name>yarn.resourcemanager.hostname.rm2</name>
				   <value>node04</value>
				 </property>




	启动:
		start-yarn.sh
			只能启动NN
		node03~node04:
			yarn-daemon.sh start resourcemanager
		http://node03:8088


	使用:
		cd  /opt/sxt/hadoop-2.6.5/share/hadoop/mapreduce
		hadoop  jar hadoop-mapreduce-examples-2.6.5.jar   wordcount  /user/root/test.txt   /data/wc/output

hadoop 开发:
windows:
c:/usr/
hadoop安装包解压
hadoop src 解压
hadoop-lib <- 部署包中share/hadoop/{common,hdfs,tools,mapreduce,yarn} 打开见到jar和lib里的jar
环境变量:
HADOOP_HOME
PATH
HADOOP_USER_NAME:root
Eclipse:最好就用我的解压
配置:user libs
创建java工程
导入 hadoop-jar,junit
创建conf导入集群中: core,hdfs,mapred,yarn -site.xml
as source
插件:
配置中指出安装路径
试图中添加locations:
node01:8020