hadoop2.6集群安装（有问题请留言，持续维护，分享使人快乐，学习也使人快乐，619你在哪）

qq_25980437

于 2019-06-26 17:15:24 发布

阅读量81

点赞数

本文链接：https://blog.csdn.net/qq_25980437/article/details/93737369

版权

提交安装好zookeeper （看本人其他文章或百度）
三个节点集群
官网下载上传到 /data/app 解压 tar zxf hadoop-2.6.5.tar.gz （每个节点一样，建议上传压缩包再解压，速度快）
软链 ln -sf /data/app/hadoop-2.6.5 /usr/local/hadoop
配置环境变量 vim /etc/profile
生效
配置hadoop cd /usr/local/hadoop/etc/hadoop/
配置jdk给hadoop 这三个有注释的放开效果
配置 vim core-site.xml <?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>ha.zookeeper.quorum</name>
<value>node1:2181,node2:2181,node3:2181</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://az</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>
vim hdfs-site.xml<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.nameservices</name>
<value>az</value>
</property>
<property>
<name>dfs.ha.namenodes.az</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.az.nn1</name>
<value>node1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.az.nn2</name>
<value>node2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.az.nn1</name>
<value>node1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.az.nn2</name>
<value>node2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node1:8485;node2:8485;node3:8485/az</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.az</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop/tmp/jn</value></property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/usr/local/hadoop/etc/hadoop/excludes</value>
</property>
</configuration>
vim yarn-site.xml<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>azyarn</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>node1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>node2</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>node1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>node2:8088</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node1:2181,node2:2181,node3:2181</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
<value>/yarn-ha</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>3072</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4.1</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>2</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
</configuration>
vim mapred-site.xml <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
vim slaves node1
node2
node3
scp到其他节点配置完成
先启动zookeeper
三个节点都启动 journalnode hadoop-daemon.sh start journalnode
节点node1 格式化 namenode hdfs namenode -format 启动 hadoop-daemon.sh start namenode
节点node2 同步 hdfs namenode -bootstrapStandby
节点node1 格式化 zkfc hdfs zkfc -formatZK
启动 stop-all.sh start-all.sh
ui 查看 http://192.168.136.132:50070/
yarn 高可用所以另一个节点单独启动 start-yarn.sh （为了省事）
测试