FATAL org.apache.hadoop.hbase.regionserver.HRegionServer
RegionServer abort: loaded coprocessors are: []
5:16:18.598 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server tracing026,60020,1420362977338 has been rejected; Reported time is too far out of sync with master. Time difference of 55987ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:344)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:237)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1295)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5:16:18.598 PM INFO org.apache.hadoop.ipc.RpcServer
Stopping server on 60020
5:16:18.598 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
Stopping infoServer
5:16:18.599 PM INFO org.mortbay.log
Stopped HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60030
5:16:18.700 PM INFO org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager
Stopping RegionServerSnapshotManager abruptly.
5:16:18.700 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
aborting server null
5:16:18.701 PM INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
Closing zookeeper sessionid=0x34ab42bf22f006c
5:16:18.703 PM INFO org.apache.zookeeper.ZooKeeper
Session: 0x34ab42bf22f006c closed
5:16:18.703 PM INFO org.apache.zookeeper.ClientCnxn
EventThread shut down
5:16:18.705 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
stopping server null; all regions closed.
5:16:18.805 PM INFO org.apache.hadoop.hbase.regionserver.Leases
regionserver60020 closing leases
5:16:18.806 PM INFO org.apache.hadoop.hbase.regionserver.Leases
regionserver60020 closed leases
5:16:18.806 PM INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread
Waiting for Split Thread to finish...
5:16:18.806 PM INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread
Waiting for Merge Thread to finish...
5:16:18.806 PM INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread
Waiting for Large Compaction Thread to finish...
5:16:18.806 PM INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread
Waiting for Small Compaction Thread to finish...
5:16:18.809 PM WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
Node /hbase/rs/0.0.0.0,60020,1420362977338 already deleted, retry=false
5:16:18.809 PM WARN org.apache.hadoop.hbase.regionserver.HRegionServer
Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/0.0.0.0,60020,1420362977338
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:177)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1273)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1262)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1334)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1048)
at java.lang.Thread.run(Thread.java:745)
5:16:18.812 PM INFO org.apache.zookeeper.ZooKeeper
Session: 0x34ab42bf22f006a closed
5:16:18.812 PM INFO org.apache.zookeeper.ClientCnxn
EventThread shut down
5:16:18.812 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
stopping server null; zookeeper connection closed.
5:16:18.813 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
regionserver60020 exiting
5:16:18.813 PM ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine
Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2467)
解决方法:
同步各节点的时间,建议使用ntpd server