hbase time out以及大量读取时拖死regionserver

最新推荐文章于 2022-03-07 18:22:13 发布
柔情岁月-万圣节
最新推荐文章于 2022-03-07 18:22:13 发布
阅读量2k
点赞数
分类专栏： hbase 问题
本文链接：https://blog.csdn.net/redhat1986/article/details/103881874
版权
Hbase 各部分报错信息：
2019-12-31 00:05:43,248 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 file(s) in mqContent of ods_app_statusInfo,bfb0833b-9e45-49d0-b4d2-8974c05e0ad0_281682053156_1569725859793,1576221814373.424aae9c61e98a7396a79e8987c4a18e. into 37b551c7a45e45e39fd32e8790702220(size=73.8 M), total size for store is 5.9 G. This selection was in queue for 0sec, and took 1sec to execute.
2019-12-31 00:05:43,248 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed compaction: Request = regionName=ods_app_statusInfo,bfb0833b-9e45-49d0-b4d2-8974c05e0ad0_281682053156_1569725859793,1576221814373.424aae9c61e98a7396a79e8987c4a18e., storeName=mqContent, fileCount=3, fileSize=73.8 M, priority=5, time=46226778348101394; duration=1sec
2019-12-31 00:06:08,459 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for lixianwei6 (auth:SIMPLE)
2019-12-31 00:06:08,459 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 192.168.35.31 port: 31042 with version info: version: "1.2.0-cdh5.7.0" url: "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.7.0-Packaging-HBase-2016-04-17_01-02-25/hbase-1.2.0-cdh5.7.0" revision: "Unknown" user: "jenkins" date: "Sun Apr 17 01:21:04 PDT 2016" src_checksum: "91b52afd1a8dfc556696ed78433f5621"
2019-12-31 00:06:08,663 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2196be1b connecting to ZooKeeper ensemble=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181
2019-12-31 00:06:08,663 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181 sessionTimeout=600000 watcher=hconnection-0x2196be1b0x0, quorum=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181, baseZNode=/hbase
2019-12-31 00:06:08,664 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181. Will not attempt to authenticate using SASL (unknown error)
2019-12-31 00:06:08,665 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.35.11:36981, server: ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181
2019-12-31 00:06:08,666 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181, sessionid = 0x26950f4755ad038, negotiated timeout = 60000
2019-12-31 00:06:08,669 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x26950f4755ad038
2019-12-31 00:06:08,670 INFO org.apache.zookeeper.ZooKeeper: Session: 0x26950f4755ad038 closed
2019-12-31 00:06:08,670 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-31 00:06:08,760 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x18315582 connecting to ZooKeeper ensemble=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181
2019-12-31 00:06:08,760 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181 sessionTimeout=600000 watcher=hconnection-0x183155820x0, quorum=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181, baseZNode=/hbase
2019-12-31 00:06:08,761 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 192.168.1.2/192.168.35.33:2181. Will not attempt to authenticate using SASL (unknown error)
2019-12-31 00:06:08,761 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.35.11:53396, server: 192.168.1.2/192.168.35.33:2181
2019-12-31 00:06:08,762 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 192.168.1.2/192.168.35.33:2181, sessionid = 0x36950f4756bcedd, negotiated timeout = 60000
2019-12-31 00:06:08,764 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x36950f4756bcedd
2019-12-31 00:06:08,764 INFO org.apache.zookeeper.ZooKeeper: Session: 0x36950f4756bcedd closed
2019-12-31 00:06:08,764 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-31 00:06:08,797 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x35596a36 connecting to ZooKeeper ensemble=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181
2019-12-31 00:06:08,798 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181 sessionTimeout=600000 watcher=hconnection-0x35596a360x0, quorum=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181, baseZNode=/hbase
2019-12-31 00:06:08,798 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 192.168.1.2/192.168.35.33:2181. Will not attempt to authenticate using SASL (unknown error)
2019-12-31 00:06:08,798 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.35.11:53397, server: 192.168.1.2/192.168.35.33:2181
2019-12-31 00:06:08,799 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 192.168.1.2/192.168.35.33:2181, sessionid = 0x36950f4756bcede, negotiated timeout = 60000
2019-12-31 00:06:08,801 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x36950f4756bcede
2019-12-31 00:06:08,801 INFO org.apache.zookeeper.ZooKeeper: Session: 0x36950f4756bcede closed
2019-12-31 00:06:08,801 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-31 00:06:08,870 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1574a984 connecting to ZooKeeper ensemble=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181
2019-12-31 00:06:08,870 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181 sessionTimeout=600000 watcher=hconnection-0x1574a9840x0, quorum=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181, baseZNode=/hbase
2019-12-31 00:06:08,870 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181. Will not attempt to authenticate using SASL (unknown error)
2019-12-31 00:06:08,871 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.35.11:36984, server: ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181
2019-12-31 00:06:08,871 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181, sessionid = 0x26950f4755ad039, negotiated timeout = 60000
2019-12-31 00:06:08,873 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x26950f4755ad039
2019-12-31 00:06:08,874 INFO org.apache.zookeeper.ZooKeeper: Session: 0x26950f4755ad039 closed
2019-12-31 00:06:08,874 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-31 00:06:08,931 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7b6a2b5e connecting to ZooKeeper ensemble=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181
2019-12-31 00:06:08,931 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181 sessionTimeout=600000 watcher=hconnection-0x7b6a2b5e0x0, quorum=ee-444-.h.aaa.com.cn:2181,ee-444--35-27.h.aaa.com.cn:2181,192.168.1.2:2181, baseZNode=/hbase
2019-12-31 00:06:08,932 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181. Will not attempt to authenticate using SASL (unknown error)
2019-12-31 00:06:08,932 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.35.11:36985, server: ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181
2019-12-31 00:06:08,933 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ee-444--35-27.h.aaa.com.cn/192.168.35.27:2181, sessionid = 0x26950f4755ad03a, negotiated timeout = 60000
2019-12-31 00:06:08,940 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x26950f4755ad03a
2019-12-31 00:06:08,940 INFO org.apache.zookeeper.ZooKeeper: Session: 0x26950f4755ad03a closed
2019-12-31 00:06:08,940 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-31 00:06:16,287 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: 192.168.1.1,60020,1574306271464-MemstoreFlusherChore requesting flush of ods_app_statusInfo,aa2c0f56-5cdc-4826-b37f-6ef6b13529c4_281682053156_1569290518058,1576221814373.466298f1ab77fe1bb6b475c1ad1610a5. because mqContent has an old edit so flush to free WALs after random delay 294957ms
2019-12-31 00:06:16,287 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: 192.168.1.1,60020,1574306271464-MemstoreFlusherChore requesting flush of gia:sdk_eid_token,4d4ea3fe,1570514674010.ba8e81608183e459f926b7c8fbacfab6. because f has an old edit so flush to free WALs after random delay 3341ms
2019-12-31 00:06:18,869 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1082ms
GC pool 'ParNew' had collection(s): count=1 time=1472ms
2019-12-31 00:06:21,462 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1092ms
GC pool 'ParNew' had collection(s): count=1 time=1524ms
2019-12-31 00:06:24,222 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1259ms
GC pool 'ParNew' had collection(s): count=1 time=1535ms
2019-12-31 00:06:26,801 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1078ms
GC pool 'ParNew' had collection(s): count=1 time=1546ms
2019-12-31 00:06:29,313 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1492ms
GC pool 'ParNew' had collection(s): count=1 time=1514ms
2019-12-31 00:06:31,901 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1087ms
GC pool 'ParNew' had collection(s): count=1 time=1529ms
2019-12-31 00:06:34,501 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms
GC pool 'ParNew' had collection(s): count=1 time=1507ms
019-12-31 00:06:37,065 INFO org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1063ms
GC pool 'ParNew' had collection(s): count=1 time=1520ms
2019-12-31 00:06:37,242 WARN org.apache.hadoop.hbase.ipc.RpcServer: (responseTooSlow): {
   "processingtimems":24085,"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","client":"192.168.35.31:31042","param":"region { type: REGION_NAME value: \"riskCase,20190510141041223NPiBeJD,1569155331884.22e55a437934ba25a45f61a963145725.\" } scan { column { family: \"riskCase\" } attribute { name: \"_ScanActualStartRow\" value: \"20190510141041223NPiBeJD\" } attribute { name: \"_isolationlevel_\" value: \"\\000\" } attribute { name: \"_ondemand_\" value: \"\\001\" } attribute { name: \"_SCAN_REGION_SERVER\" value: \"\\000\\000192.168.1.1,60020,1574306271464\" } attribute { name: \"_Aggs\" value: \"\\000\\002\\032\\001\\024\\003\\001\\004\\217\\310\\002\\000\\003\\005\\200\\000\\000\\000\\006\\003\\003\\005\\200\\000\\000\\001\\006\\003\\033\\001\\024\\003\\001\\004\\217\\310\\002\\000\\003\\005\\200\\000\\000\\000\\006\\003\\003\\005\\200\\000\\000\\001\\006\\003\" } attribute { name: \"_UngroupedAgg\" value: \"\\001\" } start_row: \"20190510141041223NPiBeJD\" stop_row: \"20190517164704595YqC6p4C\" filter { name: \"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter\" serialized_filter: \"\" } time_range { from: 0 to: 1577725572933 } max_versions: 1 cache_blocks: true max_result_size: 2097152 caching: 2147483647 } number_of_rows: 0 close_scanner: false client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false","starttimems":1577725573156,"queuetimems":0,"class":"HRegionServer","responsesize":16,"method":"Scan"}
2019-12-31 00:06:37,589 WARN org.apache.hadoop.hbase.ipc.RpcServer: (responseTooSlow): {
   "processingtimems":24422,"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","client":"192.168.35.31:31042","param":"region { type: REGION_NAME value: \"riskCase,20191017163210193OAfvgl5,1572068092584.d844077bef3a92f418da49ff85f3d7f0.\" } scan { column { family: \"riskCase\" } attribute { name: \"_ScanActualStartRow\" value: \"20191018231103562MH2CRNe\" } attribute { name: \"_isolationlevel_\" value: \"\\000\" } attribute { name: \"_ondemand_\" value: \"\\001\" } attribute { name: \"_SCAN_REGION_SERVER\" value: \"\\000\\000192.168.1.1,60020,1574306271464\" } attribute { name: \"_Aggs\" value: \"\\000\\002\\032\\001\\024\\003\\001\\004\\217\\310\\002\\000\\003\\005\\200\\000\\000\\000\\006\\003\\003\\005\\200\\000\\000\\001\\006\\003\\033\\001\\024\\003\\001\\004\\217\\310\\002\\000\\003\\005\\200\\000\\000\\000\\006\\003\\003\\005\\200\\000\\000\\001\\006\\003\" } attribute { name: \"_UngroupedAgg\" value: \"\\001\" } start_row: \"20191018231103562MH2CRNe\" stop_row: \"20191019123914173yuQrVxj\" filter { name: \"org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter\" serialized_filter: \"\" } time_range { from: 0 to: 1577725572933 } max_versions: 1 cache_blocks: true max_result_size: 2097152 caching: 2147483647 } number_of_rows: 0 close_scanner: false client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false","starttimems":1577725573166,"queuetimems":0,"class":"HRegionServer","responsesize":16,"method":"Scan"}
2019-12-31 00:09:07,725 INFO org.apache.hadoop.hbase.replication.regionserver.Replication: Normal source for cluster 1: Total replicated edits: 6260770, current progress: 
walGroup [192.168.1.1%2C60020%2C1574306271464.default]: currently replicating from: hdfs://thhbase/hbase/WALs/192.168.1.1,60020,1574306271464/192.168.1.1%2C60020%2C1574306271464.default.1577723476526 at position: 65078559


2019-12-31 00:09:07,725 INFO org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=12.25 GB, freeSize=46.39 MB, max=12.29 GB, blockCount=195419, accesses=3280643448, hits=1677046891, hitRatio=51.12%, , cachingAccesses=3222814917, cachingHits=1675401681, cachingHitsRatio=51.99%, evictions=482171, evicted=1547115469, evictedPerRun=3208.64501953125
2019-12-31 00:09:07,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: Flushing 1/1 column families, memstore=23.76 KB
2019-12-31 00:09:07,725 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-719784169-192.168.35.32-1533894083090:blk_1074960654_1219852
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2305)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1093)
2019-12-31 00:09:07,725 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-719784169-192.168.35.32-1533894083090:blk_1074960764_1219962
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2305)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1093)
2019-12-31 00:09:07,736 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-719784169-192.168.35.32-1533894083090:blk_1074960654_1219852 in pipeline DatanodeInfoWithStorage[192.168.35.11:50010,DS-cd69a1f6-dc9e-4dbf-abf2-8ead8eba0327,DISK], DatanodeInfoWithStorage[192.168.35.13:50010,DS-bd7ab223-340c-4521-86f9-3549805abfeb,DISK], DatanodeInfoWithStorage[192.168.35.5:50010,DS-bc76d780-e5c4-4829-b956-b696d0d3eeed,DISK]: bad datanode DatanodeInfoWithStorage[192.168.35.11:50010,DS-cd69a1f6-dc9e-4dbf-abf2-8ead8eba0327,DISK]
2019-12-31 00:09:07,736 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: 192.168.1.1,60020,1574306271464-MemstoreFlusherChore requesting flush of ods_app_statusInfo,aa2c0f56-5cdc-4826-b37f-6ef6b13529c4_281682053156_1569290518058,1576221814373.466298f1ab77fe1bb6b475c1ad1610a5. because mqContent has an old edit so flush to free WALs after random delay 148466ms
2019-12-31 00:09:07,734 INFO org.apache.hadoop.hbase.ScheduledChore: Chore: 192.168.1.1,60020,1574306271464-MemstoreFlusherChore missed its start time
2019-12-31 00:09:07,734 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for hbase (auth:SIMPLE)
2019-12-31 00:0
最低0.47元/天解锁文章
柔情岁月-万圣节
关注
0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hbase time out以及大量读取时拖死regionserver

Hbase 各部分报错信息：2019-12-31 00:05:43,248 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 file(s) in mqContent of ods_app_statusInfo,bfb0833b-9e45-49d0-b4d2-8974c05e0ad0_28168...
复制链接

扫一扫