HBase配置HMaster死亡

在配置HBase的时候，我们经常遇到一个问题就是regionserver正常运行，但是HMaster进程会在开始启动之后的十几秒内退出。使得HBase配置失效，经常发生的错误如下所示：

2015-06-06 17:23:33,232 INFO  [master1:16020.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave1,16020,1433581428769]
2015-06-06 17:23:33,234 INFO  [master1:16020.activeMasterManager] master.SplitLogManager: hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting is empty dir, no logs to split
2015-06-06 17:23:33,258 INFO  [master1:16020.activeMasterManager] master.SplitLogManager: started splitting 0 logs in [hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting] for [slave1,16020,1433581428769]
2015-06-06 17:23:33,265 INFO  [master1:16020.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting] in 7ms
2015-06-06 17:23:33,265 INFO  [master1:16020.activeMasterManager] zookeeper.MetaTableLocator: Deleting hbase:meta region location in ZooKeeper
2015-06-06 17:23:33,306 INFO  [master1:16020.activeMasterManager] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to slave2,16020,1433582611910
2015-06-06 17:23:33,306 INFO  [master1:16020.activeMasterManager] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1433582613274, server=null} to {1588230740 state=PENDING_OPEN, ts=1433582613306, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,423 INFO  [master1:16020.activeMasterManager] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-06-06 17:23:33,477 INFO  [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1433582613306, server=slave2,16020,1433582611910} to {1588230740 state=OPENING, ts=1433582613477, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,891 INFO  [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1433582613477, server=slave2,16020,1433582611910} to {1588230740 state=OPEN, ts=1433582613891, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,893 INFO  [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination: Handling OPENED of 1588230740 from master1,16020,1433581867821; deleting unassigned node
2015-06-06 17:23:33,898 INFO  [AM.ZK.Worker-pool2-t3] master.RegionStates: Onlined 1588230740 on slave2,16020,1433582611910
2015-06-06 17:23:33,904 INFO  [master1:16020.activeMasterManager] master.HMaster: hbase:meta assigned=1, rit=false, location=slave2,16020,1433582611910
2015-06-06 17:23:34,027 INFO  [master1:16020.activeMasterManager] hbase.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-06-06 17:23:34,027 INFO  [master1:16020.activeMasterManager] hbase.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-06-06 17:23:34,037 INFO  [master1:16020.activeMasterManager] master.AssignmentManager: Clean cluster startup. Assigning user regions
2015-06-06 17:23:34,039 INFO  [master1:16020.activeMasterManager] master.AssignmentManager: Joined the cluster in 11ms, failover=false
2015-06-06 17:23:34,079 INFO  [master1:16020.activeMasterManager] master.TableNamespaceManager: Namespace table not found. Creating...
2015-06-06 17:23:34,115 FATAL [master1:16020.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:151)
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:124)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:871)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:722)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:165)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1428)
at java.lang.Thread.run(Thread.java:745)
2015-06-06 17:23:34,116 FATAL [master1:16020.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
2015-06-06 17:23:34,116 FATAL [master1:16020.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:151)
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:124)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:871)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:722)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:165)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1428)
at java.lang.Thread.run(Thread.java:745)
2015-06-06 17:23:34,117 INFO  [master1:16020.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2015-06-06 17:23:34,117 INFO  [master/master1/192.168.1.107:16020] regionserver.HRegionServer: Stopping infoServer
2015-06-06 17:23:34,151 INFO  [master/master1/192.168.1.107:16020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2015-06-06 17:23:34,253 INFO  [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821
2015-06-06 17:23:34,263 INFO  [master/master1/192.168.1.107:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24dc68e699e0007
2015-06-06 17:23:34,271 INFO  [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x24dc68e699e0007 closed
2015-06-06 17:23:34,272 INFO  [master/master1/192.168.1.107:16020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,277 INFO  [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821; all regions closed.
2015-06-06 17:23:34,280 INFO  [master1:16020.archivedHFileCleaner] cleaner.HFileCleaner: master1:16020.archivedHFileCleaner exiting
2015-06-06 17:23:34,280 INFO  [master1,16020,1433581867821-BalancerChore] balancer.BalancerChore: master1,16020,1433581867821-BalancerChore exiting
2015-06-06 17:23:34,280 INFO  [master1:16020.oldLogCleaner] cleaner.LogCleaner: master1:16020.oldLogCleaner exiting
2015-06-06 17:23:34,280 INFO  [master1:16020.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x24dc68e699e0008, quorum=slave1:2181,master1:2181,slave2:2181, baseZNode=/hbase
2015-06-06 17:23:34,281 INFO  [CatalogJanitor-master1:16020] master.CatalogJanitor: CatalogJanitor-master1:16020 exiting
2015-06-06 17:23:34,281 INFO  [master1,16020,1433581867821-ClusterStatusChore] balancer.ClusterStatusChore: master1,16020,1433581867821-ClusterStatusChore exiting
2015-06-06 17:23:34,283 INFO  [master1:16020.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x24dc68e699e0008 closed
2015-06-06 17:23:34,283 INFO  [master1:16020.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,286 INFO  [master/master1/192.168.1.107:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x34dc7d3c9ed0002
2015-06-06 17:23:34,289 INFO  [master1:16020.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,290 INFO  [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x34dc7d3c9ed0002 closed
2015-06-06 17:23:34,391 INFO  [master1,16020,1433581867821.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: master1,16020,1433581867821.splitLogManagerTimeoutMonitor exiting
2015-06-06 17:23:34,391 INFO  [master/master1/192.168.1.107:16020] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2015-06-06 17:23:34,391 INFO  [master/master1/192.168.1.107:16020] ipc.RpcServer: Stopping server on 16020
2015-06-06 17:23:34,392 INFO  [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2015-06-06 17:23:34,394 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-06-06 17:23:34,394 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-06-06 17:23:34,404 INFO  [master/master1/192.168.1.107:16020] zookeeper.RecoverableZooKeeper: Node /hbase/rs/master1,16020,1433581867821 already deleted, retry=false
2015-06-06 17:23:34,409 INFO  [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x34dc7d3c9ed0001 closed
2015-06-06 17:23:34,409 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,409 INFO  [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821; zookeeper connection closed.
2015-06-06 17:23:34,410 INFO  [master/master1/192.168.1.107:16020] regionserver.HRegionServer: master/master1/192.168.1.107:16020 exiting

发生这样错误的主要原因是因为zookeeper与hbase连接出现问题，hbase需要zookeeper来协调管理元数据的存储以及数据的管理。解决方法是将两者冲突的文件删除掉，这些文件一般我们自己配置的文件。重新启动集群环境。