【问题解决】hbase 集群启动失败

异常信息如下：

ABORTING region server slave1,60020,1557348082471: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,60020,1557348082471 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799855ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

Cause:
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,60020,1557348082471 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799855ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2297)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,60020,1557348082471 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799855ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2295)
    ... 2 more

2019-05-08 05:41:26,545 ERROR [RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=60000] master.MasterRpcServices: Region server slave,60020,1557348082478 reported a fatal error:
ABORTING region server slave,60020,1557348082478: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave,60020,1557348082478 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799992ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

Cause:
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave,60020,1557348082478 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799992ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2297)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave,60020,1557348082478 has been rejected; Reported time is too far out of sync with master.  Time difference of 28799992ms > max allowed of 120000ms
    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2295)
    ... 2 more

 

2  问题解决  查阅相关资料说是因为网路延迟导致的链接超时，首先增加配置  在hbase-site.xml中配置  ：

<property>
        <name>hbase.master.maxclockskew</name>
        <value>120000</value>
 </property>

目的是增大链接超时时间，但是我在重新启动后还是启动不了集群， 于是想到矫正集群服务器的时间  ：

采取使用网略时间校正，保证集群节点时间一致（误差很小）

使用时间服务器 解决步骤如下  ：

1     yum -y install ntpdate   |  ubuntu 系统 使用apt  

2    ntpdate ntp1.aliyun.com   (同步阿里云的时间) 

几个节点依次执行上述命令，再次启动后hbase正常启动。