-
HA 登陆master1 >cd {hadoop-install-path} >wget 对应的hadoop 包 >tar zxvf hadoop-x.tar.gz >cd hadoop-x/etc/hadoop
- vi code-site.xml 替换 下面{hadoop-home} 和{hostname}中的参数 ``` <property>
<name>fs.defaultFS</name>
<value>hdfs://ns</value>
</property> <property> <name>hadoop.tmp.dir</name> <value>/{hadoop-home}/tmp</value> </property> <property> <name>io.file.buffer.size</name> <value>131702</value> </property><!-- zk ha 配置 将其中的-->
<property> <name>ha.zookeeper.quorum</name> <value>{hostname}:2181,{hostname}:2181,{hostname}:2181</value> </property>
``` - vi code-site.xml 替换 下面{hadoop-home} 和{hostname}中的参数 ``` <property>
-
vi hdfs-site.xml
- 注意nn1 和nn2 地址 对应的是 master1 和 master2
- {journal-address} 主要是要保持 奇数
- {hadoop-home} 填写上合适的地址，{other-path}最好是不要和其它程序使用一个硬盘。如果没有，可以和{hadoop-home} 一致
- dfs.datanode.data.dir 如果有多个硬盘，可以填写多个地址
<property>
<name>dfs.namenode.name.dir</name>
<value>{hadoop-home}/dfs/name</value> #
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>{other-path}/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>#这里有几台从机就配置几 例如：主机*1+从机*2 这里就配置2
</property>
<property>
<name>dfs.nameservices</name>
<value>ns</value>
</property>
<!-- ns下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.ns</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns.nn1</name>
<value>mast1:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns.nn1</name>
<value>mast1:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns.nn2</name>
<value>mast2:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns.nn2</name>
<value>mast2:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://{journal-address}:8485;{journal-address}:8485;{journal-address}:8485/ns</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>{hadoop-home}/hdfs/journal</value>
</property>
<!-- 开启NameNode故障时自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!-- 使用隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>{user-dir}/.ssh/id_rsa</value>
</property>
<!-- 在NN和DN上开启WebHDFS (REST API)功能,不是必须 -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
- vi mapred-site.xml 如果目录中只有 mapred-site.xml.template 使用命令 cp mapred-site.xml.template mapred-site.xml 注意{master}
<configuration>
<!--2.0以后的配置 ，mapreduce 升级为 yarn -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>{master}:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>{master}:19888</value>
</property>
</configuration>
- vi yarn-site.xml
- single node
- ha
<configuration> <!--rm失联后重新链接的时间--> <property> <name>yarn.resourcemanager.connect.retry-interval.ms</name> <value>2000</value> </property> <!--开启resourcemanagerHA,默认为false--> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!--配置resourcemanager--> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>master:2181,node1:2181</value> </property> <!--开启故障自动切换--> <property> <name>yarn.resourcemanager.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>master</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node1</value> </property> <!-- 在hadoop001上配置rm1,在hadoop002上配置rm2, 注意：一般都喜欢把配置好的文件远程复制到其它机器上，但这个在YARN的另一个机器上一定要修改 --> <property> <name>yarn.resourcemanager.ha.id</name> <value>rm1</value> <description>If we want to launch more than one RM in single node,we need this configuration</description> </property> <!--开启自动恢复功能--> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!--配置与zookeeper的连接地址--> <property> <name>yarn.resourcemanager.zk-state-store.address</name> <value>master:2181,node1:2181</value> </property> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>master:2181,node1:2181</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>appcluster-yarn</value> </property> <!--schelduler失联等待连接时间--> <property> <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name> <value>5000</value> </property> <!--配置rm1--> <property> <name>yarn.resourcemanager.address.rm1</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address.rm1</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>master:8088</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address.rm1</name> <value>master:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address.rm1</name> <value>master:8033</value> </property> <property> <name>yarn.resourcemanager.ha.admin.address.rm1</name> <value>master:23142</value> </property> <!--配置rm2--> <property> <name>yarn.resourcemanager.address.rm2</name> <value>node1:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address.rm2</name> <value>node1:8030</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>node1:8088</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address.rm2</name> <value>node1:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address.rm2</name> <value>node1:8033</value> </property> <property> <name>yarn.resourcemanager.ha.admin.address.rm2</name> <value>node1:23142</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/data/hadoop/yarn/local</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/data/hadoop/yarn/log</value> </property> <property> <name>mapreduce.shuffle.port</name> <value>23080</value> </property> <!--故障处理类--> <property> <name>yarn.client.failover-proxy-provider</name> <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value> </property> <property> <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name> <value>/yarn-leader-election</value> <description>Optionalsetting.Thedefaultvalueis/yarn-leader-election</description> </property> <property> <name>yarn.scheduler.fair.preemption</name> <value>true</value> <description>是否支持抢占,默认值为false</description> </property> <property> <name>yarn.scheduler.fair.sizebasedweight</name> <value>false</value> <description>是否启用按应用程序资源需求分配资源,默认值为false即采用公平轮询的方法分配资源</description> </property> <property> <name>yarn.scheduler.increment-allocation-mb</name> <value>1024</value> <description>仅fair有效,内存规整化单位,墨认值1024.(示例一个container请求1.5G,则调度器规整化为2G)</description> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>14336</value> <discription>每个节点可用内存,单位MB，默认是8g，spark需要大量内存，这里调整为18g</discription> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>12</value> <discription>1真core=2 vcores</discription> </property> </configuration>
* vi hadoop-env.sh
修改 export JAVA_HOME=${JAVA_HOME} 将 java_home 改为实际地址
* vi slvaes
追加 从节点hostname，有几台加几台
* scp hadoop
将hadoop 发送到其他的节点,包括 master2
* 启动
* 第一次启动 hdfs 启动
在dfs.namenode.shared.edits.dir配置的机器下执行下面的命令，启动journalnode
>sbin/hadoop-daemon.sh start journalnode
master 执行
bin/hdfs namenode -format
bin/hdfs zkfc -formatZK
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start zkfc
node1 执行
bin/hdfs namenode -bootstrapStandby
sbin/hadoop-daemon.sh start namenode
master 执行
sbin/hadoop-daemon.sh start zkfc
sbin/hadoop-daemons.sh start datanode
* 第N次启动 hdfs 启动
sbin/start-dfs.sh
* 第1-N次yarn 启动
在master 启动
sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemons.sh start nodemanager
在master2 启动
sbin/yarn-daemon.sh start resourcemanager
扫一扫
416
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
