HBASE-1.1.12集群启动之后Hmaster，HregionServer进程会很快死掉问题

一、查看日志信息如下

  2017-11-25 10:37:25,646 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop-2.7.4/lib/native
2017-11-25 10:37:25,646 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-11-25 10:37:25,646 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-11-25 10:37:25,647 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-11-25 10:37:25,647 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-11-25 10:37:25,647 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-514.el7.x86_64
2017-11-25 10:37:25,647 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-11-25 10:37:25,647 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-11-25 10:37:25,648 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2017-11-25 10:37:25,650 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=master:600000x0, quorum=master:2181,slave1:2181,slave2:2181, baseZNode=/hbase
2017-11-25 10:37:25,859 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave2/172.16.169.12:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:25,927 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Socket connection established to slave2/172.16.169.12:2181, initiating session
2017-11-25 10:37:26,096 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-11-25 10:37:26,240 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=master:2181,slave1:2181,slave2:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2017-11-25 10:37:26,871 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Opening socket connection to server master/172.16.169.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:26,873 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Socket connection established to master/172.16.169.10:2181, initiating session
2017-11-25 10:37:26,905 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/172.16.169.10:2181, sessionid = 0x15ff10795050002, negotiated timeout = 40000
2017-11-25 10:37:27,708 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting
2017-11-25 10:37:27,731 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: starting
2017-11-25 10:37:29,361 INFO  [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-11-25 10:37:29,412 INFO  [main] http.HttpRequestLog: Http request log for http.requests.master is not defined
2017-11-25 10:37:29,585 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2017-11-25 10:37:29,636 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master
2017-11-25 10:37:29,638 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2017-11-25 10:37:29,639 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2017-11-25 10:37:29,860 INFO  [main] http.HttpServer: Jetty bound to port 60010
2017-11-25 10:37:29,860 INFO  [main] mortbay.log: jetty-6.1.26
2017-11-25 10:37:34,181 INFO  [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:60010
2017-11-25 10:37:34,210 INFO  [main] master.HMaster: hbase.rootdir=hdfs://master:9000/hbase, hbase.cluster.distributed=true
2017-11-25 10:37:34,375 INFO  [main] master.HMaster: Adding backup master ZNode /hbase/backup-masters/master,60000,1511577430262
2017-11-25 10:37:34,997 INFO  [master:60000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/master,60000,1511577430262 from backup master directory
2017-11-25 10:37:35,154 INFO  [master:60000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=master,60000,1511577430262
2017-11-25 10:37:36,813 INFO  [master/master/172.16.169.10:60000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6ffbec93 connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:36,835 INFO  [master/master/172.16.169.10:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=hconnection-0x6ffbec930x0, quorum=master:2181,slave1:2181,slave2:2181, baseZNode=/hbase
2017-11-25 10:37:36,932 INFO  [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Opening socket connection to server master/172.16.169.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:36,936 INFO  [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Socket connection established to master/172.16.169.10:2181, initiating session
2017-11-25 10:37:37,017 INFO  [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/172.16.169.10:2181, sessionid = 0x15ff10795050004, negotiated timeout = 40000
2017-11-25 10:37:37,431 INFO  [master/master/172.16.169.10:60000] regionserver.HRegionServer: ClusterId : 54fafe3b-fc82-40f5-9791-47ef66133d4f
2017-11-25 10:37:43,571 INFO  [master:60000.activeMasterManager] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2017-11-25 10:37:43,649 INFO  [master:60000.activeMasterManager] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes
2017-11-25 10:37:43,932 INFO  [master:60000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x157f9d2f connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:43,933 INFO  [master:60000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=hconnection-0x157f9d2f0x0, quorum=master:2181,slave1:2181,slave2:2181, baseZNode=/hbase
2017-11-25 10:37:43,949 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave1/172.16.169.11:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:43,956 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Socket connection established to slave1/172.16.169.11:2181, initiating session
2017-11-25 10:37:43,964 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave1/172.16.169.11:2181, sessionid = 0x25ff10795560003, negotiated timeout = 40000
2017-11-25 10:37:44,063 INFO  [master:60000.activeMasterManager] balancer.StochasticLoadBalancer: loading config
2017-11-25 10:37:44,365 INFO  [master:60000.activeMasterManager] master.HMaster: Server active/primary master=master,60000,1511577430262, sessionid=0x15ff10795050002, setting cluster-up flag (Was=true)
2017-11-25 10:37:44,609 INFO  [master:60000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort
2017-11-25 10:37:44,732 INFO  [master:60000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort
2017-11-25 10:37:45,007 INFO  [master:60000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled
2017-11-25 10:37:45,189 INFO  [master:60000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5
2017-11-25 10:37:45,191 INFO  [master:60000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2017-11-25 10:37:45,207 INFO  [master:60000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log
2017-11-25 10:37:45,308 INFO  [master:60000.activeMasterManager] util.FSHDFSUtils: recoverLease=false, attempt=0 on file=hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log after 100ms
2017-11-25 10:37:49,317 INFO  [master:60000.activeMasterManager] util.FSHDFSUtils: recoverLease=true, attempt=1 on file=hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log after 4109ms
2017-11-25 10:37:49,414 WARN  [master:60000.activeMasterManager] wal.WALProcedureStore: Unable to read tracker for hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log - Invalid Trailer version. got 202 expected 1
2017-11-25 10:37:49,503 INFO  [master:60000.activeMasterManager] wal.WALProcedureStore: Lease acquired for flushLogId: 5
2017-11-25 10:37:49,646 WARN  [master:60000.activeMasterManager] wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2017-11-25 10:37:49,785 INFO  [master:60000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:49,785 INFO  [master:60000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=replicationLogCleaner0x0, quorum=master:2181,slave1:2181,slave2:2181, baseZNode=/hbase
2017-11-25 10:37:49,829 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave1/172.16.169.11:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:49,841 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Socket connection established to slave1/172.16.169.11:2181, initiating session
2017-11-25 10:37:49,864 INFO  [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave1/172.16.169.11:2181, sessionid = 0x25ff10795560004, negotiated timeout = 40000
2017-11-25 10:37:49,899 INFO  [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-11-25 10:37:51,441 INFO  [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1542 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-11-25 10:37:52,378 INFO  [PriorityRpcServer.handler=5,queue=1,port=60000] master.ServerManager: Registering server=slave1,16020,1511577422976
2017-11-25 10:37:52,399 INFO  [PriorityRpcServer.handler=3,queue=1,port=60000] master.ServerManager: Registering server=slave2,16020,1511577423313
2017-11-25 10:37:52,399 INFO  [PriorityRpcServer.handler=1,queue=1,port=60000] master.ServerManager: Registering server=master,16020,1511577430553
2017-11-25 10:37:52,430 INFO  [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 2531 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-11-25 10:37:52,567 INFO  [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log
2017-11-25 10:37:52,568 INFO  [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000005.log, hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000006.log]
2017-11-25 10:37:52,599 INFO  [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000005.log
2017-11-25 10:37:52,600 INFO  [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000006.log]
2017-11-25 10:37:53,933 INFO  [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 4034 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-11-25 10:37:54,415 INFO  [master:60000.activeMasterManager] master.ServerManager: Finished waiting for region servers count to settle; checked in 3, slept for 4516 ms, expecting minimum of 1, maximum of 2147483647, master is running
2017-11-25 10:37:54,470 INFO  [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/master,16020,1511532189674 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,528 INFO  [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/master,16020,1511577430553 belongs to an existing region server
2017-11-25 10:37:54,581 INFO  [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,618 INFO  [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,805 INFO  [master:60000.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:37:54,855 INFO  [master:60000.activeMasterManager] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:37:55,654 INFO  [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:00,482 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta acquired by master,16020,1511577430553
2017-11-25 10:38:00,658 INFO  [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta=last_update = 1511577480483 last_version = 1 cur_worker_name = master,16020,1511577430553 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:06,250 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta entered state: DONE master,16020,1511577430553
2017-11-25 10:38:06,464 INFO  [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting/slave2%2C16020%2C1511532219830..meta.1511532416907.meta to hdfs://master:9000/hbase/oldWALs/slave2%2C16020%2C1511532219830..meta.1511532416907.meta
2017-11-25 10:38:06,471 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta
2017-11-25 10:38:06,545 WARN  [master:60000.activeMasterManager] master.SplitLogManager: returning success without actually splitting and deleting all the log files in path hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting
2017-11-25 10:38:06,545 INFO  [master:60000.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 11690ms
2017-11-25 10:38:06,929 INFO  [master:60000.activeMasterManager] zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at address=slave2,16020,1511532219830, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on slave2,16020,1511577423313
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2915)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:979)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1259)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2145)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:748)


2017-11-25 10:38:06,941 INFO  [master:60000.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:38:06,951 INFO  [master:60000.activeMasterManager] master.SplitLogManager: hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting is empty dir, no logs to split
2017-11-25 10:38:06,951 INFO  [master:60000.activeMasterManager] master.SplitLogManager: started splitting 0 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:38:06,977 WARN  [master:60000.activeMasterManager] master.SplitLogManager: returning success without actually splitting and deleting all the log files in path hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting
2017-11-25 10:38:06,977 INFO  [master:60000.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 26ms
2017-11-25 10:38:06,978 INFO  [master:60000.activeMasterManager] zookeeper.MetaTableLocator: Deleting hbase:meta region location in ZooKeeper
2017-11-25 10:38:07,292 INFO  [master:60000.activeMasterManager] master.AssignmentManager: Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2017-11-25 10:38:07,385 INFO  [master:60000.activeMasterManager] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to slave2,16020,1511577423313
2017-11-25 10:38:07,385 INFO  [master:60000.activeMasterManager] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1511577487292, server=null} to {1588230740 state=PENDING_OPEN, ts=1511577487385, server=slave2,16020,1511577423313}
2017-11-25 10:38:07,595 INFO  [master:60000.activeMasterManager] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2017-11-25 10:38:07,837 INFO  [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1511577487385, server=slave2,16020,1511577423313} to {1588230740 state=OPENING, ts=1511577487837, server=slave2,16020,1511577423313}
2017-11-25 10:38:12,722 INFO  [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1511577487837, server=slave2,16020,1511577423313} to {1588230740 state=OPEN, ts=1511577492722, server=slave2,16020,1511577423313}
2017-11-25 10:38:12,729 INFO  [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination: Handling OPENED of 1588230740 from master,60000,1511577430262; deleting unassigned node
2017-11-25 10:38:12,882 INFO  [master:60000.activeMasterManager] master.HMaster: hbase:meta with replicaId 0 assigned=1, rit=false, location=slave2,16020,1511577423313
2017-11-25 10:38:13,691 INFO  [master:60000.activeMasterManager] hbase.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2017-11-25 10:38:13,878 INFO  [master:60000.activeMasterManager] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OPEN, ts=1511577493878, server=slave1,16020,1511532206980} to {e6bff28294ecdbdc5fadca964ceafb26 state=OFFLINE, ts=1511577493878, server=slave1,16020,1511532206980}
2017-11-25 10:38:13,899 INFO  [master:60000.activeMasterManager] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OPEN, ts=1511577493899, server=master,16020,1511532189674} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OFFLINE, ts=1511577493899, server=master,16020,1511532189674}
2017-11-25 10:38:13,994 INFO  [master:60000.activeMasterManager] master.AssignmentManager: Found regions out on cluster or in RIT; presuming failover
2017-11-25 10:38:14,105 INFO  [master:60000.activeMasterManager] master.AssignmentManager: Joined the cluster in 414ms, failover=true
2017-11-25 10:38:14,334 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Splitting logs for slave1,16020,1511532206980 before assignment; region count=1
2017-11-25 10:38:14,367 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Splitting logs for slave2,16020,1511532219830 before assignment; region count=0
2017-11-25 10:38:14,424 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Splitting logs for master,16020,1511532189674 before assignment; region count=1
2017-11-25 10:38:14,483 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: dead splitlog workers [slave1,16020,1511532206980]
2017-11-25 10:38:14,506 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting] for [slave1,16020,1511532206980]
2017-11-25 10:38:14,559 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:38:14,617 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:38:14,655 INFO  [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 2 unassigned = 2 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995=last_update = 1511577494615 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:14,693 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995 acquired by slave2,16020,1511577423313
2017-11-25 10:38:14,818 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: dead splitlog workers [master,16020,1511532189674]
2017-11-25 10:38:14,907 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting] for [master,16020,1511532189674]
2017-11-25 10:38:15,171 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089 acquired by master,16020,1511577430553
2017-11-25 10:38:15,337 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539 acquired by slave1,16020,1511577422976
2017-11-25 10:38:20,646 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995 entered state: DONE slave2,16020,1511577423313
2017-11-25 10:38:20,654 INFO  [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 3 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539=last_update = 1511577496132 last_version = 2 cur_worker_name = slave1,16020,1511577422976 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089=last_update = 1511577495619 last_version = 2 cur_worker_name = master,16020,1511577430553 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995=last_update = 1511577500561 last_version = 2 cur_worker_name = slave2,16020,1511577423313 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:20,905 INFO  [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting/slave1%2C16020%2C1511532206980.default.1511532378995 to hdfs://master:9000/hbase/oldWALs/slave1%2C16020%2C1511532206980.default.1511532378995
2017-11-25 10:38:20,909 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995
2017-11-25 10:38:20,920 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting] in 6414ms
2017-11-25 10:38:20,927 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Reassigning 1 region(s) that slave1,16020,1511532206980 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:20,928 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 1 regions from the dead server slave1,16020,1511532206980
2017-11-25 10:38:21,241 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089 entered state: DONE master,16020,1511577430553
2017-11-25 10:38:21,262 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart assignment. 
2017-11-25 10:38:21,263 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] master.AssignmentManager: Bulk assigning 1 region(s) across 3 server(s), retainAssignment=true
2017-11-25 10:38:21,332 INFO  [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting/slave2%2C16020%2C1511532219830.default.1511532372089 to hdfs://master:9000/hbase/oldWALs/slave2%2C16020%2C1511532219830.default.1511532372089
2017-11-25 10:38:21,342 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089
2017-11-25 10:38:21,361 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 6744ms
2017-11-25 10:38:21,362 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Reassigning 0 region(s) that slave2,16020,1511532219830 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:21,362 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 0 regions from the dead server slave2,16020,1511532219830
2017-11-25 10:38:21,363 INFO  [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Finished processing of shutdown of slave2,16020,1511532219830
2017-11-25 10:38:21,424 INFO  [master,60000,1511577430262-GeneralBulkAssigner-1] master.AssignmentManager: Assigning 1 region(s) to slave1,16020,1511577422976
2017-11-25 10:38:21,631 INFO  [master,60000,1511577430262-GeneralBulkAssigner-1] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OFFLINE, ts=1511577501524, server=slave1,16020,1511532206980} to {e6bff28294ecdbdc5fadca964ceafb26 state=PENDING_OPEN, ts=1511577501631, server=slave1,16020,1511577422976}
2017-11-25 10:38:22,777 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] master.AssignmentManager: Bulk assigning done
2017-11-25 10:38:22,777 INFO  [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Finished processing of shutdown of slave1,16020,1511532206980
2017-11-25 10:38:23,054 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539 entered state: DONE slave1,16020,1511577422976
2017-11-25 10:38:23,068 INFO  [AM.ZK.Worker-pool2-t5] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=PENDING_OPEN, ts=1511577501631, server=slave1,16020,1511577422976} to {e6bff28294ecdbdc5fadca964ceafb26 state=OPENING, ts=1511577503068, server=slave1,16020,1511577422976}
2017-11-25 10:38:23,137 INFO  [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting/master%2C16020%2C1511532189674.default.1511532381539 to hdfs://master:9000/hbase/oldWALs/master%2C16020%2C1511532189674.default.1511532381539
2017-11-25 10:38:23,142 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539
2017-11-25 10:38:23,177 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting] in 8270ms
2017-11-25 10:38:23,180 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Reassigning 1 region(s) that master,16020,1511532189674 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:23,181 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 1 regions from the dead server master,16020,1511532189674
2017-11-25 10:38:23,245 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart assignment. 
2017-11-25 10:38:23,258 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] master.AssignmentManager: Bulk assigning 1 region(s) across 3 server(s), retainAssignment=true
2017-11-25 10:38:23,284 INFO  [master,60000,1511577430262-GeneralBulkAssigner-0] master.AssignmentManager: Assigning 1 region(s) to master,16020,1511577430553
2017-11-25 10:38:23,395 INFO  [master,60000,1511577430262-GeneralBulkAssigner-0] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OFFLINE, ts=1511577503339, server=master,16020,1511532189674} to {1d3b7b34795a72f8fb15b68d64c56e53 state=PENDING_OPEN, ts=1511577503395, server=master,16020,1511577430553}
2017-11-25 10:38:24,489 INFO  [AM.ZK.Worker-pool2-t7] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=PENDING_OPEN, ts=1511577503395, server=master,16020,1511577430553} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OPENING, ts=1511577504489, server=master,16020,1511577430553}
2017-11-25 10:38:29,242 INFO  [AM.ZK.Worker-pool2-t8] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OPENING, ts=1511577503068, server=slave1,16020,1511577422976} to {e6bff28294ecdbdc5fadca964ceafb26 state=OPEN, ts=1511577509242, server=slave1,16020,1511577422976}
2017-11-25 10:38:29,372 INFO  [AM.ZK.Worker-pool2-t10] master.RegionStates: Offlined e6bff28294ecdbdc5fadca964ceafb26 from slave1,16020,1511532206980
2017-11-25 10:38:34,136 INFO  [master:60000.activeMasterManager] master.HMaster: Master has completed initialization
2017-11-25 10:38:34,219 INFO  [master:60000.activeMasterManager] quotas.MasterQuotaManager: Quota support disabled
2017-11-25 10:38:34,264 INFO  [master:60000.activeMasterManager] zookeeper.ZooKeeperWatcher: not a secure deployment, proceeding
2017-11-25 10:38:34,891 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OPENING, ts=1511577504489, server=master,16020,1511577430553} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OPEN, ts=1511577514891, server=master,16020,1511577430553}
2017-11-25 10:38:34,915 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates: Offlined 1d3b7b34795a72f8fb15b68d64c56e53 from master,16020,1511532189674
2017-11-25 10:38:34,917 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] master.AssignmentManager: Bulk assigning done
2017-11-25 10:38:34,917 INFO  [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Finished processing of shutdown of master,16020,1511532189674

二、在网上查看了很多的介绍，感觉与我的情况不太相符

  于是我查看了集群中的进程信息发现zookeeper集群的的进程没有了，重启zookeeper集群，在启动HBASE就行

使用netstat  -nltp 命令查看端口信息

60010端口已经正常了，问题解决。