1、hbase压缩与编码的配置
解决方案:
1)apt-get install liblzo2-dev
2)hadoop-gpl-compression-0.2.0-dev.jar 放入classpath
把libgpl下的共享库文件放入/opt/hbase/hbase/lib/native/Linux-amd64-64/
libgplcompression.a libgplcompression.la libgplcompression.so libgplcompression.so.0 libgplcompression.so.0.0.0
3)配置:
<property>
<name>io.compression.codecs</name>
<value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
4)测试:
hbase org.apache.hadoop.hbase.util.CompressionTest hdfs:///user.dat lzo
hdfs://hbase-hdfs.goso.cn:9000/hbase-weibo/weibo_test 1021877013
hbase@GS-WDE-SEV0151:/opt/hbase/hbase$ hadoop fs -dus /hbase-weibo/weibo_lzo
hdfs://hbase-hdfs.goso.cn:9000/hbase-weibo/weibo_lzo 1179175365
hbase@GS-WDE-SEV0151:/opt/hbase/ops$ hadoop fs -dus /hbase-weibo/weibo_diff
hdfs://hbase-hdfs.goso.cn:9000/hbase-weibo/weibo_diff 2754679243
hdfs://hbase-hdfs.goso.cn:9000/hbase-weibo/weibo-new 5270708315
2、相关测试
个数 | 耗时 | |
weibo_test | 2314054 | 3m49.661s |
weibo-new | 2314054 | 1m55.349s |
weibo_lzo | 2314054 | 3m24.378s |
weibo_diff | 2314054 | 4m41.792s |
个数 | 耗时 | |
weibo_test | 100,0000 | 122min12s, 平均7.3ms/Record |
weibo-new | 100,0000 | 68min40s, 平均3.99ms/Record |
weibo_lzo | 100,0000 | 83m26.539s, 平均5.00ms/Record |
weibo_diff | 100,0000 | 58m5.915s, 平均3.48ms/Record |
个数 | 耗时 | |
weibo_test | 8640447 | 571670ms, 66µs/Put, 6.61ms/batch |
weibo-new | 8640447 | 329694ms,38.12µs/Put, 3.81ms/batch |
weibo_lzo | 8640447 | 295770ms, 34.23µs/Put, 3.42ms/batch |
weibo_diff | 8640447 | 250399ms, 28.97µs/Put,2.90ms/batch |
3、总体结论分析
From Binospace, post HBase实战系列1—压缩与编码技术
文章的脚注信息由WordPress的wp-posturl插件自动生成