Hbase常见避免热点问题的方法

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/WYpersist/article/details/79954419

Hbase常见避免热点问题的方法

加盐

一把rowkey前缀,决定了在哪一个分区。

 


 

降低热点问题,但是会造成读的时候,效率下降。

 

哈希

 

反转

 

举例:

 

前缀都是一样,可能都会往一个region里面写数据时,就会出现热点问题。

返回来,把号码倒过来,就会是不同的数字,解决了热点问题。 

时间戳反转

 

HBASE总结

1、尽量减少行和列的大小

 

2列簇尽可能越短越好,最好是一个字符

3、冗长的属性名虽然可读性好,但是更短的属性存储在HBase中会更好

HBase常见的问题(遇到的喊一声)

03-14

hbase(main):001:0> create 'test','c1','c2'rnrnERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row 'test,,00000000000000', but failed after 7 attempts.rnExceptions:rnjava.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues=.META.,,1/info:server/1394769851007/Put/vlen=12, .META.,,1/info:serverstartcode/1394769851007/Put/vlen=8rnrn[b]HMASTER的LOG[/b]rnrn2014-03-14 14:14:53,430 DEBUG org.apache.hadoop.hbase.master.LoadBalancer: Server information: slave2,60020,1394769841706=5, slave1,60020,1394769842452=5rn2014-03-14 14:14:53,430 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=2 regions=10 average=5.0 mostloaded=5 leastloaded=5rn2014-03-14 14:16:35,259 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table test state to enabled but was already enabledrn2014-03-14 14:16:35,344 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000msrn2014-03-14 14:16:35,344 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Enabling new writer for /hbase/test/35c2cba1ede5344f7c6afc97d7ac76fc/.logs/hlog.1394777795344rn2014-03-14 14:16:35,354 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs -- HDFS-200 -- not available, dfs.support.append=falsern2014-03-14 14:16:35,355 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: New hlog /hbase/test/35c2cba1ede5344f7c6afc97d7ac76fc/.logs/hlog.1394777795344rn2014-03-14 14:16:35,355 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: getNumCurrentReplicas--HDFS-826 not available; hdfs_out=org.apache.hadoop.hdfs.DFSClient$DFSOutputStream@b8a595, exception=org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.getNumCurrentReplicas()rn2014-03-14 14:16:35,355 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc.rn2014-03-14 14:16:35,468 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc.; next sequenceid=1rn2014-03-14 14:16:35,474 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Added region test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc. to METArn2014-03-14 14:16:35,474 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc.: disabling compactions & flushesrn2014-03-14 14:16:35,474 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc.rn2014-03-14 14:16:35,474 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed c1rn2014-03-14 14:16:35,475 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed c2rn2014-03-14 14:16:35,475 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc.rn2014-03-14 14:16:35,475 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 9 on 60000.logSyncer interrupted while waiting for sync requestsrn2014-03-14 14:16:35,475 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 9 on 60000.logSyncer exitingrn2014-03-14 14:16:35,475 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://master:9000/hbase/test/35c2cba1ede5344f7c6afc97d7ac76fc/.logsrn2014-03-14 14:16:35,512 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/test/35c2cba1ede5344f7c6afc97d7ac76fc/.oldlogsrn2014-03-14 14:16:35,515 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144bec1bc2c0000-0x144bec1bc2c0000-0x144bec1bc2c0000 Creating (or updating) unassigned node for 35c2cba1ede5344f7c6afc97d7ac76fc with OFFLINE statern2014-03-14 14:16:35,542 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc. so generated a random one; hri=test,,1394777795244.35c2cba1ede5344f7c6afc97d7ac76fc., src=, dest=slave1,60020,1394769842452; 2 (online=2, exclude=null) available serversrnrn[b]slave1的log[/b]rn2014-03-14 14:14:19,674 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=957.34 KB, free=197.79 MB, max=198.73 MB, blocks=5, accesses=306, hits=297, hitRatio=97.05%%, cachingAccesses=302, cachingHits=297, cachingHitsRatio=98.34%%, evictions=0, evicted=0, evictedPerRun=NaNrnslave2的logrn2014-03-14 14:14:20,348 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=975.32 KB, free=197.77 MB, max=198.73 MB, blocks=6, accesses=62, hits=52, hitRatio=83.87%%, cachingAccesses=58, cachingHits=52, cachingHitsRatio=89.65%%, evictions=0, evicted=0, evictedPerRun=NaNrnrnHMASTER的配置rnconfiguration>rnrnhbase.rootdirrnhdfs://master:9000/hbasernrn rnrnhbase.cluster.distributedrntruernrn rnhbase.masterrnhdfs://master:60000rnrnrnhbase.zookeeper.quorumrnmaster,slave1,slave2rnrnrnhbase.zookeeper.property.dataDirrn/home/hadoop/hbase-0.90.5/zookeeperrnrnhbase.hlog.split.skip.errorsrntruernrnrnrnhbase-env.sh的配置rnexport HBASE_MANAGES_ZK=falsernexport JAVA_HOME=/usr/java/jdk1.7.0_25rnexport HBASE_CLASSPATH=/home/hadoop/hadoop-0.20.2/confrn

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试