详解HBase Compaction

1. Compaction是什么

合并多个HFile生成一个HFile

Compaction有两种：

• Minor Compaction（部分文件合并）
• Major Compaction（完整文件合并）

2.为什么要Compaction？

• 减少HFile文件的个数
• 提高性能
• 清除过期和删除数据

3.配置

修改Hbase配置文件可以控制compaction行为

键值	默认值	意义
hbase.regionserver.thread.splitcompactcheckfrequency	20s	compaction检查周期
hbase.hstore.compactionThreshold	3	最小minor compaction的文件个数
hbase.hstore.blockingStoreFiles	7	Block flush操作的Store个数
hbase.hstore.blockingWaitTime	90s	Block flush操作的等待时间
hbase.hstore.compaction.max	10	最大minor compaction的文件个数
hbase.hregion.majorcompaction	1 day	Major compaction的周期

4.流程

Compaction是一个Async的过程,可以由客户端发起，也可能是服务器端自己检查发起compaction.

1）客户端发起

Client端：

HBaseAdmin::compaction or majorCompaction

==>HMaster modifyTable

==>RegionManager::startAction

==> put into map regionsToCompact and regionsToMajorCompact

==>Send to HRegionServer

Server端：

HRegionServer::run forward the request to CompactionSplitThread

==>CompactionSplitThread handle the request from queue

==>HRegion::compactStores

==>Do compaction preparations, create the compaction folder

==>HStore::compaction

==>Create a HFile.Writer for writing

==>Create a StoreScanner for major compaction

==>Create a MinorCompactionStoreScanner for minor compaction

==>Scan the scanner and write to the hfile

==>Complete the compaction,delete old files and move the file to store folder

2) Server检查发起

Major compaction:

Major compaction由region server定期检查

==>HRegionServer::MajorCompactionChecker

==>Send the request to CompactionSplitThread

Minor compaction:

Minor compaction由Memstore flush到HDFS前检查

==>MemStoreFlusher::flushRegion

==>Send the request to CompactionSplitThread