HDFS中高可用性HA的讲解-稍微有点复杂

HDFS HA介绍

相比于Hadoop1.0，Hadoop 2.0中的HDFS增加了两个重大特性，HA和Federaion。HA即为High Availability，用于解决NameNode单点故障问题，该特性通过热备的方式为主NameNode提供一个备用者，一旦主NameNode出现故障，可以迅速切换至备NameNode，从而实现不间断对外提供服务。Federation即为“联邦”，该特性允许一个HDFS集群中存在多个NameNode同时对外提供服务，这些NameNode分管一部分目录（水平切分），彼此之间相互隔离，但共享底层的DataNode存储资源。

在一个典型的HDFS HA场景中，通常由两个NameNode组成，一个处于active状态，另一个处于standby状态。Active NameNode对外提供服务，比如处理来自客户端的RPC请求，而Standby NameNode则不对外提供服务，仅同步active namenode的状态，以便能够在它失败时快速进行切换。

为了能够实时同步Active和Standby两个NameNode的元数据信息（实际上editlog），需提供一个共享存储系统，可以是NFS、QJM（Quorum Journal Manager）或者Bookeeper，Active Namenode将数据写入共享存储系统，而Standby监听该系统，一旦发现有新数据写入，则读取这些数据，并加载到自己内存中，以保证自己内存状态与Active NameNode保持基本一致，如此这般，在紧急情况下standby便可快速切为active namenode。

Hadoop0.23.2版本之前, NameNode是HDFS集群的单点故障点，每一个集群只有一个NameNode,如果这个机器或进程不可用，整个集群就无法使用，直到重启NameNode或者新启动一个NameNode节点。影响HDFS集群不可用主要包括以下两种情况：

1）第一种情况是如机器宕机这样的意外情况，将导致集群不可用，只有在重启NameNode之后才可使用。

2）第二种情况是计划内的软件或硬件升级(NameNode节点)，将导致集群在短时间范围内不可用。

 

HDFS的HA就是为了解决上述问题，通过提供选择运行在同一集群中的一个热备用的“主/备”两个冗余NameNodes。这允许在机器宕机或系统维护的时候，快速的转移到另一个NameNode.

一个典型的HA集群，两个单独的机器配置为NameNodes，在任何时候，一个NameNode处于活动状态，另一个处于待机状态，活动的NameNode负责处理集群中所有客户端的操作，待机时仅仅作为一个slave，保持足够的状态，如果有必要提供一个快速的故障转移.

为了保持备用节点与活动节点状态的同步，目前的实现需要两个节点同时访问一个共享存储设备（例如从NASNFS挂载）到一个目录。将有可能在未来的版本中放宽此限制。当活动节点对命名空间进行任何修改，它将把修改记录写到共享目录下的一个日志文件，备用节点会监听这个目录，当发现更改时，它会把修改内容同步到自己的命名空间。备用节点在故障转移时，它将保证已经读取了所有共享目录内的更改记录，保证在发生故障前的状态与活动节点保持完全一致。

为了提供快速的故障转移，必须保证备用节点有最新的集群中块的位置信息，为了达到这一点，Datanode节点需要配置两个nameNode的位置，同时发送块的位置信息和心跳信息到两个nameNode。

任何时候只有一个namenode处于活动状态，对于HA集群的操作是至关重要的，否则两个节点之间的状态就会产生冲突，数据丢失或其它不正确的结果，为了达到这个目的或者所谓的“裂脑场景”出现，管理员必须为共享存储配置至少一个（fencing）方法。在宕机期间，如果不能确定之间的活动节点已经放弃活动状态，fencing进程负责中断以前的活动节点编辑存储的共享访问。这可以防止任何进一步的修改命名空间，允许新的活动节点安全地进行故障转移。

注:目前，只有手动故障转移支持。这就意味着HA nameNode不能自动检测活动nameNode的失败，而是通过手动启动故障转移。自动故障检测和故障转移将在未来的版本中实现。

HDFS Using QJM

　　HA使用的是分布式的日志管理方式

一：概述

1.背景

　　如果namenode出现问题，整个HDFS集群将不能使用。

　　是不是可以有两个namenode呢

　　　　一个为对外服务->active

　　　　一个处于待机->standby

　　　　他们的之间共享的元数据叫 nameservice

 

2.HDFS HA的几大中重点

　　1）保证两个namenode里面的内存中存储的文件的元数据同步

　　　　->namenode启动时，会读镜像文件

　　2）变化的记录信息同步

　　3）日志文件的安全性

　　　　->分布式的存储日志文件（cloudera公司提出来的）

　　　　　　->2n+1个，使用副本数保证安全性

　　　　->使用zookeeper监控

　　　　　　->监控两个namenode，当一个出现了问题，可以达到自动故障转移。

　　　　　　->如果出现了问题，不会影响整个集群

　　　　　　->zookeeper对时间同步要求比较高。

　　4）客户端如何知道访问哪一个namenode

　　　　->使用proxy代理

　　　　->隔离机制

　　　　->使用的是sshfence

　　　　->两个namenode之间无密码登录

　　5）namenode是哪一个是active

　　　　->zookeeper通过选举选出zookeeper。

　　　　->然后zookeeper开始监控，如果出现文件，自动故障转移。

 

3.架构

　　注意两个namenode之间的话。

　　

 

4.使用QJM

　　JournalNode：日志节点，进程管理。

　　

 

 

二：准备

3.规划集群

　　namenode　　　　namenode

　　journalnode　　   journalnode　　　　journalnode -->日志的分布，这是日志节点，考虑的是日志的安全性。

　　datanode　　　　 datanode　　　　　 datanode

 

4.关闭所有的进程

　　

5.保存分布式的源数据，复制一份用来HDFS HA的检测。

　　先是第一台，先将分布式的etc/hadoop，保存为dist-hadoop，保存源数据。

　　同时，新建tmp。

　　至于第二台以及第三台，在分发之间再进行配置。

　　

三：配置文件

6.将core-site.xml中的文件系统删除，并添加新的文件系统

　　以前的是使用是配置一台，现在配置多态namenode，使用的方式是nameservices的名称的方式。

　　

　　添加配置

　　

 

7.配置hdfs-site.xml

　　

8.继续配置hdfs-site.xml

　　dfs.nameservices的配置

　　

　　dfs.ha.namenodes.[nameservice ID]的配置

　　包括rpc，http的namenodde地址。

　　

　　dfs.namenode.shared.edits.dir的配置

　　这是journalnode的地址

　　

　　dfs.journalnode.edits.dir 的配置

　　这是journalnode的日志存储的目录

　　先新建目录：

　　

　　

 

　　dfs.client.failover.proxy.provider的配置（客户端）

　　

　　dfs.ha.fencing.methods的配置

　　使用的方式为ssh拦截

　　

　　

　　

9.配置完成，在分发之前先进行的是目录的规划

　　

　　

10.分发

　　

 

四：启动

1.启动三台zookeeper

　　现在有些奇怪，这三台zookeeper为啥要启动，好像没用到。

　　看视频上启动的，在这里暂时不启动了，因为没有用到，在自动故障转移的时候才去使用。

 

2.启动三台的日志节点

　　

　　

3.格式化第一台虚拟机

　　因为是共享数据，所以格式化一台虚拟机即可。

　　因为管理的是同一个元数据，所以另外一个不需要格式化。

　　

 

4.启动namenode（第一台）

 

5.紧接着，同步元数据（在第二台上写命令）

　　最好是bin/hdfs namenode -help查看

　　

 

6.启动namenode（第两台虚拟机）

　　

7.启动三台的datanode

　　

8.观看两台的启动状态

　　关于这个问题，刚才在官网上看了一下，的确是启动后会是两个standby。

　　官网如下：

　　

 

　　

　　

 

9.强制切换状态

　　1）、查找帮助命令，属于bin/hdfs haadmin

　　

　　2）、具体命令

　　

 

10.结果

　　1）、

　　

　　或者：

　　这个查看第一台的状态，也可以在第二台虚拟机上查看节点一，例如：

　　

 

　　2）、

　　

 

五：再次测试

1.在HDFS上新建目录并上传文件

　　

2.杀死第一台的namenode，进行测试

　　

3.将avtive的状态切换到第二台

　　这个命令应该在第二台虚拟机上进行操作。

　　通过实验发现，如果不进行强制转换成active，这个命令是不会成功的，所以需要添加上--forceactive。

　　

 

4.看第一台是否可以观看hdfs中的文件

　　如果可以，说明，HA发挥作用了。

　　因为这时proxy提供的接口变成nn2.

　　

 

六：自动故障转移

　　前提：关闭所有的进程。

　　依赖：zookeeper的监控，组件为：ZKFC。

　　启动以后都是standby，选举一个active。

　　规划：

　　namenode　　　　namenode

　　ZKFC     　　     　ZKFC

　　journalnode　　   journalnode　　　　journalnode

　　datanode　　　　 datanode　　　　　 datanode

　　

1.看看官网的说法

　　这个地方关于zookeeper的使用不是太理解，需要到官网上研究一下。

　　还配置了一个zookeeper集群，用于ZKFC（DFSZKFailoverCont roller）故障转移。

　　文章最后做一下说明。对ZKFC的原理。

　　

 

2.配置core-site.xml

　　添加zookeeper的服务，包括主机名和端口号。

　　

 

3.配置hdfs-site.xml

　　添加自动故障转移的使能。

　　

 

4.分发

　　

 

5.确定关闭所有的进程

 　　这一步是开始的基础。

　　官网上的说法：

　　Before you begin configuring automatic failover, you should shut down your cluster. It is not currently possible to transition from a manual failover setup to an automatic failover setup while the cluster is running.

 

6.开启三台的zookeeper服务

　　先开启监控，然后方便下面进行初始化ZK，产生一个ZNode目录。

 　　

 

7.初始化HA在zookeeper中的状态bin/hdfs zkfc -formatZK

　　在zookepper上创建znode节点，用于存储数据，方便自动实现故障转移。

　　关于这个ZKFC的初始化需要在哪台虚拟机上进行，通过思考与实践，在可以进行HA的那台机器上即可，而且做一次就行了。

　　在下面有说明情况。

　　

 

8.观察成功与否

　　进入zookeeper目录

　　命令：bin/zkCli.sh

　　

 

9.启动sbin/start-dfs.sh

　　前两台虚拟机会出现DFZKFailoverController。

　　如果没有开启DFZKFailoverController，可以手动开启，命令是sbin/hadoop-daemon.sh start zkfc。

　　查看官网的说法：

　　

 

　　

 

　

七：简单检测

1.展示前两台的虚拟机状态

　　

　　

2.杀死第二台的虚拟机

　　

 

3.结果

　　这时，第一台虚拟机变成active。

 　　

 

八：HA流程

1.说明

　　ZKGZ通过RPC协议获取namenode的状态

　　ZKFC连接zookeeper，完成选举。

　　stabdby节点ZKFC监控本机的namenode。

　　ZKFC在监控周期内检测到失效的namenode。

　　ZKFC记录失败的active namenode信息到zookeeper。

　　zookeeper发送一个通知给stabdby的节点ZKFC

　　standby节点ZKFC进入切换的流程，隔离刚才的主机。

　　ZKFC进行选举，确保新的active节点。

 

九：ZKFC原理说明

1.基本原理

　　zk的基本特性：
　　(1) 可靠存储小量数据且提供强一致性
　　(2) ephemeral node, 在创建它的客户端关闭后，可以自动删除
　　(3) 对于node状态的变化，可以提供异步的通知(watcher)

　　zk在zkfc中可以提供的功能：
　　(1) Failure detector: 及时发现出故障的NN，并通知zkfc
　　(2) Active node locator: 帮助客户端定位哪个是Active的NN
　　(3) Mutual exclusion of active state: 保证某一时刻只有一个Active的NN

 

2.模块

　　(1) ZKFailoverController(DFSZKFailoverController): 驱动整个ZKFC的运转，通过向HealthMonitor和ActiveStandbyElector注册回调函数的方式，subscribe HealthMonitor和ActiveStandbyElector的事件，并做相应的处理
　　(2) HealthMonitor: 定期check NN的健康状况，在NN健康状况发生变化时，通过回调函数把变化通知给ZKFailoverController
　　(3) ActiveStandbyElector: 管理NN在zookeeper上的状态，zookeeper上对应node的结点发生变化时，通过回调函数把变化通知给ZKFailoverController
　　(4) FailoverController: 提供做graceful failover的相关功能(dfs admin可以通过命令行工具手工发起failover) 

 

3.架构

　　

 

4.线程模型

　　ZKFC的线程模型总体上来讲比较简单的，它主要包括三类线程，一是主线程；一是HealthMonitor线程; 一是zookeeper客户端的线程。它们的主要工作方式是：
　　(1) 主线程在启动所有的服务后就开始循环等待
　　(2) HealthMonitor是一个单独的线程，它定期向NN发包，检查NN的健康状况
　　(3) 当NN的状态发生变化时，HealthMonitor线程会回调ZKFailoverController注册进来的回调函数，通知ZKFailoverController NN的状态发生了变化
　　(4) ZKFailoverController收到通知后，会调用ActiveStandbyElector的API，来管理在zookeeper上的结点的状态
　　(5) ActiveStandbyElector会调用zookeeper客户端API监控zookeeper上结点的状态，发生变化时，回调ZKFailoverController的回调函数，通知ZKFailoverController，做出相应的变化

 

5.类关系图

　　ZKFC的主类是org.apache.hadoop.hdfs.tools.DFSZKFailoverController。

　　

　　

6.formatZK

　　创建特定目录，作为后续写节点状态的父路径。如果该目录已经存在，清理原有目录为空目录。

 

7.继续说明

•  HealthMonitor

　　在一个独立线程中，通过RPC方式，周期性的调用HAServiceProtocol接口的monitorHealth方法，获取NN的状态。并把状态报告给ActiveStandbyElector 。

•  ActiveStandbyElector

　　ActiveStandbyElector 负责判断哪个NN可以成为Active。它通过ZK，看哪个能够成功的创建一个特定的ephemeral lock file (znode)，哪个就是Active，其它的成为Standby。在一个节点被通知变成Active后，它必须确保自己能够提供一致性的服务（数据一致性），否则它需要主动退出选举。

　　如果一个Active因HealthMonitor监控到状态异常，这里会作出判断，先通过Fenceing功能关闭它（确保关闭或者不能提供服务），然后在ZK上删除它对应ZNode。

　　发送上述事件后，在另外一台机器上的ZKFC中的ActiveStandbyElector 会收到事件，并重新进行选举（尝试创建特定ZNode），它将获得成功并更改NN中状态，从而实现Active节点的变更。

 

十：formatZK日志说明

1.日志

18/01/28 19:48:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/01/28 19:48:30 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at linux-hadoop01.ibeifeng.com/192.168.140.121:8020
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:host.name=linux-hadoop01.ibeifeng.com
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/modules/jdk1.7.0_67/jre
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/fullyDistributions/ha/hadoop-2.5.0/etc/hadoop:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/contrib/capacity-scheduler/*.jar
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/fullyDistributions/ha/hadoop-2.5.0/lib/native
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.el6.x86_64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.name=beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/fullyDistributions/ha/hadoop-2.5.0
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=linux-hadoop01.ibeifeng.com:2181,linux-hadoop02.ibeifeng.com:2181,linux-hadoop03.ibeifeng.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@32dd803
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Opening socket connection to server linux-hadoop02.ibeifeng.com/192.168.140.122:2181. Will not attempt to authenticate using SASL (unknown error)
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Socket connection established to linux-hadoop02.ibeifeng.com/192.168.140.122:2181, initiating session
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Session establishment complete on server linux-hadoop02.ibeifeng.com/192.168.140.122:2181, sessionid = 0x2613c8b97f50000, negotiated timeout = 5000
18/01/28 19:48:31 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Session: 0x2613c8b97f50000 closed
18/01/28 19:48:31 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2613c8b97f50000
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: EventThread shut down

 

2.作用

　　在倒数第四行有一行说法。

　　创建了一个目录：

 /hadoop-ha/ns1 in ZK

 

3.操作

　　上面有日志说connect 02虚拟机。

　　进入第二台zookeeper看看是否存在这个目录。