Hadoop完全分布式搭建

最新推荐文章于 2024-05-06 20:41:36 发布

_TIM_

最新推荐文章于 2024-05-06 20:41:36 发布

阅读量92

点赞数

文章标签： hadoop

本文链接：https://blog.csdn.net/fang19970714/article/details/116089781

版权

集群运行服务规划

NameNode在内存中保存着整个文件系统的名称空间和文件数据块的地址映射,Hadoop中的NameNode好比是人的心脏，非常重要!绝对不可以停止工作
hadoop 中HDFS的高可靠指的是可以同时启动2个NameNode。其中一个处于工作状态，另一个处于随时待命状态。这样，当一个NameNode所在的服务器宕机时，可以在数据不丢失的情况下，手工或者自动切换到另一个NameNode提供服务。两个NameNode为了数据同步，会通过一组称作JournalNodes的独立进程进行相互通信(两个NameNode之间同步元数据信息)。当active状态的NameNode的命名空间有任何修改时，会告知大部分的JournalNodes进程。standby状态的NameNode有能力读取JNs中的变更信息，并且一直监控edit log的变化，把变化应用于自己的命名空间。standby可以确保在集群出错时，命名空间状态已经完全同步了。使用完全分布式，实现namenode高可用, ResourceManager的高可用

	192.169.40.100	192.169.40.100	192.169.40.100
zookeeper	zk	zk	zk
HDFS	JournalNode	JournalNode	JournalNode
	NameNode(active)	NameNode(standBy)
	ZKFC	ZKFC
	DataNode	DataNode	DataNode
Yarn		ResourceManager(standBy)	ResourceManager(active)
	NodeManager	NodeManager	Nodemanager
MapReduce			JobHistoryServer

ZKFC: ZKFailoverController守护进程监看NameNode运行的健康信息的报告

关键配置文件
core-site.xml

<configuration>
 <property>
	<name>ha.zookeeper.quorum</name>
	<value>node01:2181,node02:2181,node03:2181</value>
 </property>
 
  <property>
	<name>fs.defaultFS</name>
	<value>hdfs://ns</value>
 </property>
 
  <property>
	<name>hadoop.tmp.dir</name>
	<value>/export/servers/hadoop-2.7.5/data/temp</value>
 </property>
 
  <property>
	<name>fs.trash.interval</name>
	<value>10080</value>
 </property>
</configuration>

hdfs-site.xml

<configuration>
	<property>
		<name>dfs.nameservices</name>
		<value>ns</value>
	</property>
	
	<property>
		<name>dfs.ha.namenodes.ns</name>
		<value>nn1,nn2</value>
	</property>
	
	<property>
		<name>dfs.namenode.rpc-address.ns.nn1</name>
		<value>node01:8020</value>
	</property>
	
	<property>
		<name>dfs.namenode.rpc-address.ns.nn2</name>
		<value>node02:8020</value>
	</property>
	
	<property>
		<name>dfs.namenode.servicerpc-address.ns.nn1</name>
		<value>node01:8022</value>
	</property>
	
	<property>
		<name>dfs.namenode.servicerpc-address.ns.nn2</name>
		<value>node02:8022</value>
	</property>
	
	<property>
		<name>dfs.namenode.http-address.ns.nn1</name>
		<value>node01:50070</value>
	</property>
	
	<property>
		<name>dfs.namenode.http-address.ns.nn2</name>
		<value>node02:50070</value>
	</property>
	
	<property>
		<name>dfs.namenode.shared.edits.dir</name>
		<value>qjournal://node01:8485;node02:8485;node03:8485/ns</value>
	</property>
	
	<property>
		<name>dfs.client.failover.proxy.provider.ns</name>
		<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
	</property>
	
	<property>
		<name>dfs.ha.fencing.methods</name>
		<value>sshfence</value>
	</property>
	
	<property>
		<name>dfs.ha.fencing.ssh.private-key-files</name>
		<value>/root/.ssh/id_rsa</value>
	</property>
	
	<property>
		<name>dfs.journalnode.edits.dir</name>
		<value>/export/servers/hadoop-2.7.5/data/dfs/jn</value>
	</property>
	
	<property>
		<name>dfs.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
	
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:///export/servers/hadoop-2.7.5/data/dfs/nn/name</value>
	</property>
	
	<property>
		<name>dfs.namenode.edits.dir</name>
		<value>file:///export/servers/hadoop-2.7.5/data/dfs/nn/edits</value>
	</property>
	
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:///export/servers/hadoop-2.7.5/data/dfs/dn</value>
	</property>
	
	<property>
		<name>dfs.permissions</name>
		<value>false</value>
	</property>
	
	<property>
		<name>dfs.blocksize</name>
		<value>134217728</value>
	</property>
</configuration>

mapred_site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>node03:10020</value>
	</property>
	
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>node03:19888</value>
	</property>
	
	<property>
		<name>mapreduce.jobtracker.system.dir</name>
		<value>/export/servers/hadoop-2.7.5/data/system/jobtracker</value>
	</property>
	
	<property>
		<name>mapreduce.map.memory.mb</name>
		<value>1024</value>
	</property>
	
	<property>
		<name>mapreduce.task.io.sort.mb</name>
		<value>100</value>
	</property>
	
	<property>
		<name>mapreduce.task.io.sort.factor</name>
		<value>10</value>
	</property>
	
	<property>
		<name>mapreduce.reduce.shuffle.parallelcopies</name>
		<value>25</value>
	</property>
	
	<property>
		<name>yarn.app.mapreduce.am.command-opts</name>
		<value>-Xmx1024m</value>
	</property>
	
	<property>
		<name>yarn.app.mapreduce.am.resource.mb</name>
		<value>1536</value>
	</property>
	
	<property>
		<name>mapreduce.cluster.local.dir</name>
		<value>/export/servers/hadoop-2.7.5/data/system/local</value>
	</property>
</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
	</property>

	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>mycluster</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.hostname.rm1</name>
		<value>node03</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.hostname.rm2</name>
		<value>node02</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.adress.rm1</name>
		<value>node03:8032</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm1</name>
		<value>node03:8030</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
		<value>node03:8031</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.admin.address.rm1</name>
		<value>node03:8033</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.webapp.address.rm1</name>
		<value>node03:8088</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.adress.rm2</name>
		<value>node02:8032</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm2</name>
		<value>node02:8030</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
		<value>node02:8031</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.admin.address.rm2</name>
		<value>node02:8033</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.webapp.address.rm2</name>
		<value>node02:8088</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.recovery.enabled</name>
		<value>true</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.ha.id</name>
		<value>rm2</value>
		<description>If we want to launch more than one Rm in the single node, we need this config</description>
	</property>
	
	<property>
		<name>yarn.resourcemanager.store.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>node02:2181,node03:2181,node01:2181</value>
		<description>For multiple zk services, separate them with comma</description>
	</property>
	
	<property>
		<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
		<value>true</value>
		<description>Enable automatic failover; By default, it is enabled only when HA is enabled.</description>
	</property>
	
	<property>
		<name>yarn.client.failover-proxy-provider</name>
		<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.resource.cpu-vcores</name>
		<value>4</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.resource.memory-mb</name>
		<value>512</value>
	</property>
	
	<property>
		<name>yarn.scheduler.minimum-allocation-mb</name>
		<value>512</value>
	</property>
	
	<property>
		<name>yarn.scheduler.maximum-allocation-mb</name>
		<value>512</value>
	</property>
	
	<property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>2592000</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.log.retain-seconds</name>
		<value>604800</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.log-aggregation.compression-type</name>
		<value>gz</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.local-dirs</name>
		<value>/export/servers/hadoop-2.7.5/yarn/local</value>
	</property>
	
	<property>
		<name>yarn.resourcemanager.max-completed-applications</name>
		<value>1000</value>
	</property>
	
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
		
	<property>
		<name>yarn.resourcemanager.connect.retry-interval.ms</name>
		<value>2000</value>
	</property>
	</property>
</configuration>

_TIM_

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Hadoop完全分布式搭建

集群运行服务规划NameNode在内存中保存着整个文件系统的名称空间和文件数据块的地址映射,Hadoop中的NameNode好比是人的心脏，非常重要!绝对不可以停止工作hadoop 中HDFS的高可靠指的是可以同时启动2个NameNode。其中一个处于工作状态，另一个处于随时待命状态。这样，当一个NameNode所在的服务器宕机时，可以在数据不丢失的情况下，手工或者自动切换到另一个NameNode提供服务。两个NameNode为了数据同步，会通过一组称作JournalNodes的独立进程进行相互通信(
复制链接

扫一扫