HBase错误日志：集群时钟不同步导致HBase集群无法启动

最新推荐文章于 2021-06-26 14:38:40 发布
最新推荐文章于 2021-06-26 14:38:40 发布
阅读量355 收藏
点赞数
文章标签： 大数据 java python
原文链接：https://my.oschina.net/xhhuang/blog/894505
版权

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

今天在启动HBase的时候出bug了，启动不了，查看master端的日志之后发现是因为之前配置集群的时候没有做时间同步，时间久了之后集群中不同节点的时间相差太大，导致HBase无法启动。

下面是master端的log日志记录和slave端的log日志记录。

master端部分日志记录：

2017-05-07 21:32:24,173 INFO  [master:16000.activeMasterManager-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/59.71.229.189:2181, sessionid = 0x15be31b7a810000, negotiated timeout = 40000
2017-05-07 21:32:24,191 INFO  [master:16000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-05-07 21:32:24,448 WARN  [B.defaultRpcServer.handler=2,queue=2,port=16000] master.ServerManager: Server slave2,16020,1494163845439 has been rejected; Reported time is too far out of sync with master.  Time difference of 97488ms > max allowed of 30000ms
2017-05-07 21:32:24,448 WARN  [B.defaultRpcServer.handler=10,queue=1,port=16000] master.ServerManager: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
2017-05-07 21:32:24,497 ERROR [B.defaultRpcServer.handler=4,queue=1,port=16000] master.MasterRpcServices: Region server slave1,16020,1494163890158 reported a fatal error:
ABORTING region server slave1,16020,1494163890158: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

Cause:
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:329)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2298)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:906)
	at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2296)
	... 2 more

2017-05-07 21:32:24,497 ERROR [B.defaultRpcServer.handler=1,queue=1,port=16000] master.MasterRpcServices: Region server slave2,16020,1494163845439 reported a fatal error:
ABORTING region server slave2,16020,1494163845439: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave2,16020,1494163845439 has been rejected; Reported time is too far out of sync with master.  Time difference of 97488ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

Cause:
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave2,16020,1494163845439 has been rejected; Reported time is too far out of sync with master.  Time difference of 97488ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:329)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2298)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:906)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave2,16020,1494163845439 has been rejected; Reported time is too far out of sync with master.  Time difference of 97488ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2296)
	... 2 more

slave端部分日志记录：

2017-05-07 21:31:31,969 FATAL [regionserver/slave1/59.71.229.190:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:329)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2298)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:906)
	at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2296)
	... 2 more

2017-05-07 21:31:31,971 FATAL [regionserver/slave1/59.71.229.190:16020] regionserver.HRegionServer: ABORTING region server slave1,16020,1494163890158: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:329)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2298)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:906)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2296)
	... 2 more
2017-05-07 21:31:31,972 FATAL [regionserver/slave1/59.71.229.190:16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2017-05-07 21:31:31,986 INFO  [regionserver/slave1/59.71.229.190:16020] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "NonHeapMemoryUsage" : {
      "committed" : 41861120,
      "init" : 2555904,
      "max" : -1,
      "used" : 41492824
    },
    "ObjectPendingFinalizationCount" : 0,
    "HeapMemoryUsage" : {
      "committed" : 251396096,
      "init" : 260046848,
      "max" : 4086824960,
      "used" : 77308128
    },
    "Verbose" : false,
    "ObjectName" : "java.lang:type=Memory"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
    "modelerType" : "RegionServer,sub=IPC",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "slave1"
  } ],
  "beans" : [ ],
  "beans" : [ ]
}
2017-05-07 21:31:32,007 INFO  [regionserver/slave1/59.71.229.190:16020] regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave1,16020,1494163890158 has been rejected; Reported time is too far out of sync with master.  Time difference of 52782ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:409)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:275)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:361)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

 

转载于:https://my.oschina.net/xhhuang/blog/894505

weixin_33845477
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
HBase 集群服务器时间不同步导致HRegionServer无法启动
Veechange的博客
11-01 1928
由于机器重启，所以单独启动HRegionServer，结果发HRegionServer进程刚启动就自动关闭了，查看日期文件，发现报错了： Server centos6,16020,1477966664214 has been rejected; Reported time is too far out of sync with master. Time difference of 43894m
Hbase 各regionServer 时间不一致
zhouleilei的专栏
07-03 226
可能会遇到个regionServer 时间不一致的问题，那如果时间不一致超过一定限度，比如30s hbase默认的容忍间隔。那再regionServerStartup 的时候 就会跑出一个异常 ClockOutOfSyncException 还有一种地方 在regionserver 想master reportForDuty 的过程中会调用regionServerStartup 此时也会检查 时...
hbase奇葩错误（节点时间不一致导致的）
upupfeng的博客
11-22 729
问题描述 如果集群机器间的时间不一致，会导致HMaster可以起来，但是HRegionServer起不来   解决方案 修改时间与其他的机器时间保持一致 yum install -y ntpdate ntpdate -u ntp.api.bz                        ...
Hbase-启动后RegionServer由于时间不同步自动挂掉
知行合一，止于至善
08-23 1069
查看挂掉的节点hdp-node-02日志 [root@hdp-node-02 logs]# cat hbase-root-regionserver-hdp-node-02.log 2017-08-23 17:00:59,154 FATAL [regionserver/hdp-node-02/192.168.33.102:16020] regionserver.HRegionServer: Mas
时间不一致导致hbase无法启动
weixin_34268753的博客
07-10 128
2019独角兽企业重金招聘Python工程师标准>>> ...
hbase-exporter:HBase Prometheus导出器
03-05
解析“ hbase hbck”命令的输出以检查HBase中的不一致情况将Hbase标记为不健康要求满足以下条件之一至少有一个陈旧的区域正在过渡'hbase hbck'命令显示HBase不一致对预定义表的写入不成功无法确定ZooKeeper领导者...
hbase-cluster:现成的 HBase 集群
06-05
在您自己的工作站上启动大数据集群 HBase 0.98.3完全分布式集群，只有 3 个依赖项维基可以在找到该项目的 wiki概述使用这个现成的集群作为您试验大数据技术的起点。 您可以完全自由地修改它以满足您的需求，同时拥有...
月光宝盒双11-HBase集群应用和优化经验
最新发布
10-26
HBase集群应用和优化经验 HBase集群是大数据处理和存储的重要组件，本篇文章将分享月光宝盒双11 HBase集群应用和优化经验，涵盖了月光宝盒项目简介、HBase集群双11相关工作、月光宝盒双11相关工作、总结的话、Q&A等...
Hbase2.2.0集群部署.docx
09-14
Hbase2.2.0集群部署需要配置环境变量、配置Hbase配置文件、启动Hbase集群、使用Hbase Shell和Hbase DDL/DML语言等步骤。通过这些步骤，我们可以成功部署Hbase2.2.0集群，并使用Hbase Shell和Hbase DDL/DML语言来管理...
zookeeper+hbase集群搭建
04-15
1. **使用未编译的Zookeeper**：这可能导致Zookeeper无法启动。确保使用官方编译好的`zookeeper-bin`版本。 2. **Hbase Shell命令报错**：如果Hbase Shell中出现错误，可能是因为HDFS中的旧数据。这时，可以尝试删除...
HBase log中这些日志 说明整个集群正常吗？
shatelang的专栏
05-25 3987
HMaster节点：  hbase-hadoop-master-hadoop-node8.log 2012-05-25 08:54:37,472 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1377f608e3f0006, likely serv
hbase全分布式遇到的问题2--集群时间不同步
beijihukk的博客
03-14 2160
现象：start-abase.sh启动hbase集群后，web UI能访问到masternode:16010，但是过几秒钟后就发现原先的从节点的regionserver进程都自己死掉了，示意如下： 以某个从节点为例，刚开始时 #jps 14343 DataNode 7789   HQuorumPeer 78790 HRegionServer ... 过了几秒钟后 #jps 1
解决hbase时间不同步问题
duzewen的博客
03-27 3472
时间不同步问题:   在每台机子上输入sudo ntpdate time.nist.gov再起hbase集群就行了
大数据集群问题排查：时间不同步导致hbase regionserver只启动了部分节点的进程
学亮编程手记
06-26 425
注意排查时间同步的问题：如果三个节点的时间不一致，会导致regionserver进程启动失败！！！ 配置时间同步以后，集群正常启动：
20131022Hive和Hbase（时间不一致）问题
godspeedlaile9的专栏
10-22 800
Hive问题： 错误提示：FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient FAILED: Execution Error, return code 1 from org.apach
Hbase对时，时差范围的确定
weixin_30593443的博客
10-02 128
Hbase对时具有严格的要求，集群内部所有机器之间的时差默认不能超过30秒，也就是说，一旦某个regionserver节点上的时间与master节点上的时间差值超过30秒，就会导致相应的regionserver无法上线。如果需要重新设置各机器之间的允许时差范围，可以通过在hbase-site.xml添加相应的配置，如下： <property> <na...
配置hadoop分布式集群时解决各节点时间不同步的问题
weixin_42209572的博客
05-04 2141
Hadoop集群用到的工具 版本号 节点名称 hadoop 2.7.1 Master hbase 1.4.7 Slave1 Slave2 Slave3 根据前文的配置Hadoop分布式集群方法： https://blog.csdn.net/weixin_42209572/article/details...
Hbase时间不同步问题解决方案
$firecat利白的代码足迹$
08-21 5026
http://baijiahao.baidu.com/s?id=1582997438282198970&amp;wfr=spider&amp;for=pc Hbase 集群服务器时间不同步导致HRegionServer无法启动 通过start-hbase.sh命令在主节点启动Hbase集群，可以看出主节点node01、node02， 从节点node02、node03、node04均成功启动 ...
HBase因节点时间不同步无法启动regionserver
会飞的大象
03-03 1458
HBase因节点时间不同步无法启动regionserver

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值