apache安装_apache-hadoop-2.7.7集群安装-HA高可用版本

1. 环境准备及版本介绍

1.Linux版本介绍

CentOS release 6.8 (Final)Linux version 2.6.32-642.el6.x86_64 (mockbuild@worker1.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Tue May 10 17:27:01 UTC 2016镜像版本：CentOS-6.8-x86_64-minimal.iso

2.JDK版本

java version "1.8.0_131"Java(TM) SE Runtime Environment (build 1.8.0_131-b11)Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)环境变量配置：vim /etc/profile 添加以下文件内容 然后source /etc/profileexport JAVA_HOME=/home/software/jdk/jdk1.8.0_131export PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/rt.jar

3.Hadoop版本

hadoop-2.7.7

2.节点安排

节点安排

备注/etc/hosts：192.168.1.211 z1192.168.1.212 z2192.168.1.213 z3192.168.1.214 z4

3.主机之间免密操作

1.生成公钥$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 2.将公钥追加至~/.ssh/authorized_keys$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys 如与其他主机进行免密则需要将公钥传输至目标主机并追加至该主机的~/.ssh/authorized_keys，首先也得在目标主机执行 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 使用scp命令进行传输

4.安装zookeeper

1.zoo.cfg文件配置

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/opt/zookeeper# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1server.1=z1:2888:3888server.2=z2:2888:3888server.3=z3:2888:3888

2.dataDir配置的路径下创建myid文件

dataDir=/opt/zookeeper编辑myid在对应的机器上输入自己内容，与机器对应server.1=z1:2888:3888server.2=z2:2888:3888server.3=z3:2888:3888例如：在z1机器上的/opt/zookeeper创建一个myid文件 内容为：1

3.启动zookeeper

进入zookeeper的安装的bin目录，在所有安装的机器执行./zkServer.sh start执行./zkServer.sh status查看是否启动成功

5.开始安装Hadoop

1.hadoop-env.sh文件配置

export JAVA_HOME=/home/software/jdk/jdk1.8.0_131

2.hdfs-site.xml文件配置

dfs.nameservicesbigdatadfs.ha.namenodes.bigdatann1,nn2dfs.namenode.rpc-address.bigdata.nn1z1:8020dfs.namenode.rpc-address.bigdata.nn2z2:8020dfs.namenode.http-address.bigdata.nn1z1:50070dfs.namenode.http-address.bigdata.nn2z2:50070dfs.namenode.shared.edits.dirqjournal://z2:8485;z3:8485;z4:8485/bigdatadfs.client.failover.proxy.provider.bigdataorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProviderdfs.ha.fencing.methodssshfencedfs.ha.fencing.ssh.private-key-files/root/.ssh/id_dsadfs.journalnode.edits.dir/opt/journaldatadfs.ha.automatic-failover.enabledtrue

3.core-site.xml配置文件

fs.defaultFShdfs://bigdataha.zookeeper.quorumz1:2181,z2:2181,z3:2181hadoop.tmp.dir/home/workspace/hadoop-2.7.7

4.slaves配置文件

z2z3z4

6.启动集群

1.第一步

在安装journalnode的主机上(z2,z3,z4)上启动journalnode执行：./hadoop-daemon.sh start journalnode

2.第二步

选择一台namenode主机进行操作执行 ./hdfs namenode –format来格式化任意一个namenode执行 ./hadoop-daemon.sh start namenode来启动这个格式化的namenode

3.第三步

另一台主机进行如下操作执行：./hdfs namenode –bootstrapStandby

4.第四步

执行 ./stop-dfs.sh停止所有服务

5.第五步

执行 ./hdfs zkfc –formatZK

6.第六步

执行 ./start-dfs.sh 启动集群

7.第七步 访问集群

浏览器 http://192.168.1.211:50070 访问集群

7.安装MapReduce

1.mapred-site.xml配置

mapreduce.framework.nameyarnmapreduce.jobhistory.addressz1:10020MapReduce JobHistory Server IPC host:portmapreduce.jobhistory.webapp.addressz1:19888MapReduce JobHistory Server Web UI host:portmapreduce.jobhistory.intermediate-done-dir/home/mr-history/tmpDirectory where history files are written by MapReduce jobsmapreduce.jobhistory.done-dir/home/mr-history/doneDirectory where history files are managed by the MR JobHistory Server

2. yarn-site.xml配置

yarn.nodemanager.aux-servicesmapreduce_shuffleyarn.resourcemanager.ha.enabledtrueyarn.resourcemanager.cluster-idbigdatayarn.resourcemanager.ha.rm-idsrm1,rm2yarn.resourcemanager.hostname.rm1z1yarn.resourcemanager.hostname.rm2z2yarn.resourcemanager.zk-addressz1:2181,z2:2181,z3:2181yarn.log-aggregation-enabletrueyarn.log.server.urlhttp://z1:19888/jobhistory/logs

3.启动MapReduce

进入sbin目录 执行 ./start-yarn.sh 启动

4.启动JobHistoryServer

进入sbin目录 执行 ./ mr-jobhistory-daemon.sh start historyserver 启动浏览器输入 http://z1:19888 访问

---

安装完毕！！！感谢阅读，别忘点赞哦，很辛苦的 亲测成功！！！