原因分析:
线上hbase,在凌晨1点左右,发现某一台regionserver进行了重启(regionserver加了守护线程)
1、查看master日志:
2020-02-27 01:04:57,001 ERROR [RpcServer.FifoRWQ.default.read.handler=26,queue=10,port=16000] master.MasterRpcServices: Region server a3ster,16020,1582342923163reported a fatal error:
ABORTING region server a3ser,16020,1582342923163: Replay of WAL required. Forcing server shutdown
Cause:
org.apache.hadoop.hbase.DroppedSnapshotException: region: T_BL,\x0A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1572576275632.069e4d877a4ff46f9964ac8bcddb09ef.
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2509)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2186)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2148)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2039)
at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1965)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:505)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFl