compact处理流程分析

compact处理流程分析

compact的处理与split相同，由client端与flush时检查发起。

针对compact还有一个在rs生成时生成的CompactionChecker线程定期去检查是否需要做compact操作

线程执行的间隔时间通过hbase.server.thread.wakefrequency配置，默认为10*1000ms

CompactionChecker线程主要作用：

生成通过hbase.server.thread.wakefrequency(10*1000ms)配置的定期检查region是否需要compact的检查线程,

如果需要进行compact,会在此处通过compact的线程触发compcat的请求

此实例中通过hbase.server.thread.wakefrequency(10*1000ms)配置majorcompact的优先级,

如果majorcompact的优先级大过此值,把compact的优先级设置为此值.

Store中通过hbase.server.compactchecker.interval.multiplier配置多少时间需要进行compact检查的间隔

默认为1000ms,

compactionChecker的检查周期为wakefrequency*multiplierms,

也就是默认情况下线程调用1000次执行一次compact检查

a.compaction检查时发起compact的条件是

如果一个store中所有的file个数减去在做(或发起compact请求)的个数，大于或等于

hbase.hstore.compaction.min配置的值,

老版本使用hbase.hstore.compactionThreshold进行配置,默认值为3

b.majorcompact的条件检查

通过hbase.hregion.majorcompaction配置major的检查周期,default=1000*60*60*24

通过hbase.hregion.majorcompaction.jitter配置major的浮动时间,默认为0.2,

也就是major的时间上下浮动4.8小时

b2.检查(当前时间-major配置时间>store最小的文件生成时间)表示需要major,

b2.1>store下是否只有一个文件,同时这个文件已经到了major的时间,

b2.1>检查ttl时间是否达到(intager.max表示没配置),达到ttl时间需要major,否则不做

b2.2>文件个数大于1,到达major的时间,需要major

Client端发起compactRegion的request

Client通过HBaseAdmin.compact发起regionserver的rpc连接，调用regionserver.compactRegion

如果传入的是tablename而不是regionname,会迭代出此table的所有region调用HRegionServer.compactRegion

由client发起，调用HRegionServer.compactRegion

publicCompactRegionResponse compactRegion(finalRpcController controller,

finalCompactRegionRequest request)throwsServiceException {

try{

checkOpen();

requestCount.increment();

从onlineRegions中得到request的Hregion实例

HRegion region= getRegion(request.getRegion());

region.startRegionOperation(Operation.COMPACT_REGION);

LOG.info("Compacting" +region.getRegionNameAsString());

booleanmajor =false;

byte[] family =null;

Storestore =null;

如果client发起的request中传入有columnfamily的值，得到此cf的HStore

if(request.hasFamily()){

family= request.getFamily().toByteArray();

store= region.getStore(family);

if(store ==null){

thrownewServiceException(newIOException("columnfamily " + Bytes.toString(family)+

"does not exist in region " +region.getRegionNameAsString()));

}

}

检查是否是major的compact请求

if(request.hasMajor()){

major= request.getMajor();

}

如果是发起majorcompaction的操作，

if(major) {

if(family !=null){

store.triggerMajorCompaction();

} else{

region.triggerMajorCompaction();

}

}

String familyLogMsg= (family!= null)?"for column family: " +Bytes.toString(family):"";

LOG.trace("User-triggeredcompaction requested for region " +

region.getRegionNameAsString()+ familyLogMsg);

String log= "User-triggered "+ (major ?"major ": "")+ "compaction"+ familyLogMsg;

否则是一般compation的请求，通过compactsplitThread.requestCompaction发起compactrequest

if(family!= null){

compactSplitThread.requestCompaction(region,store, log,

Store.PRIORITY_USER,null);

} else{

compactSplitThread.requestCompaction(region,log,

Store.PRIORITY_USER,null);

}

returnCompactRegionResponse.newBuilder().build();

}catch(IOException ie){

thrownewServiceException(ie);

}

}

非major的compact处理流程

requestCompaction不管是直接传入sotre或者是region的传入，

如果传入的是region,那么会拿到region下的所有store，迭代调用每一个store的compactionrequest操作。

所有的非majorcompaction request最终会通过如下方法发起compactionrequest

privatesynchronized CompactionRequestrequestCompactionInternal(finalHRegion r,

finalStore s,

finalString why,intpriority,CompactionRequest request,booleanselectNow)

针对store的compactionrequest处理流程

如果要对一个HBASE的表禁用掉compaction操作，可以通过createtable时配置COMPACTION_ENABLED属性

privatesynchronized CompactionRequestrequestCompactionInternal(finalHRegion r, finalStore s,

finalString why,intpriority,CompactionRequest request,booleanselectNow)

throwsIOException {

if(this.server.isStopped()

|| (r.getTableDesc()!= null&& !r.getTableDesc().isCompactionEnabled())){

returnnull;

}

CompactionContextcompaction= null;

此时的调用selectNow为true,(如果是系统调用，此时的selectNow为false,)

也就是在发起request到CompactSplitThread.CompactionRunner线程执行时，

如果是系统调用,传入的CompactionContext的实例为null,否则是用户发起的调用在这个地方得到compaction实例

if(selectNow){

通过HStore.requestCompaction得到一个compactionContext,计算要进行compact的storefile

并设置其request.priority为Store.PRIORITY_USER表示用户发起的request

如果是flush时发起的compact，

并设置其request.priority为hbase.hstore.blockingStoreFiles配置的值减去storefile的个数,

表示系统发起的request,

如果hbase.hstore.blockingStoreFiles配置的值减去storefile的个数==PRIORITY_USER

那么priority的值为PRIORITY_USER+1

见生成CompactionRequest实例

compaction= selectCompaction(r,s,priority,request);

if(compaction== null)returnnull;// message logged inside

}

//We assume that most compactionsare small. So, put system compactionsinto small

//pool; we will do selection there, and move to large pool ifnecessary.

longsize =selectNow ?compaction.getRequest().getSize(): 0;

此时好像一直就得不到largeCompactions的实例(在system时通过CompactionRunner线程检查)，

因为selectNow==false时，size的大小为0

不可能大于hbase.regionserver.thread.compaction.throttle配置的值

此配置的默认值是hbase.hstore.compaction.max*2*memstoresize

ThreadPoolExecutor pool= (!selectNow&& s.throttleCompaction(size))

? largeCompactions: smallCompactions;

通过smallCompactions的线程池生成CompactionRunner线程并执行,见执行Compaction的处理线程

pool.execute(newCompactionRunner(s,r,compaction,pool));

if(LOG.isDebugEnabled()){

String type= (pool ==smallCompactions)? "Small ": "Large ";

LOG.debug(type+ "Compaction requested: "+ (selectNow? compaction.toString(): "system")

+ (why!= null&& !why.isEmpty()? "; Because: "+ why : "")+ "; "+ this);

}

returnselectNow ?compaction.getRequest(): null;

}

生成CompactionRequest实例

Hstore.requestcompaction得到要进行compact的storefile,并生成一个CompactionContext

publicCompactionContextrequestCompaction(intpriority, CompactionRequest baseRequest)

throwsIOException {

//don't even select for compaction if writes are disabled

if(!this.areWritesEnabled()){

returnnull;

}

生成一个DefaultStoreEngine.DefaultCompactionContext实例(如果storeEngine是默认的配置)

CompactionContextcompaction= storeEngine.createCompaction();

this.lock.readLock().lock();

try{

synchronized(filesCompacting){

//First, see if coprocessorwould want to override selection.

if(this.getCoprocessorHost()!= null){

List<StoreFile>candidatesForCoproc= compaction.preSelect(this.filesCompacting);

booleanoverride =this.getCoprocessorHost().preCompactSelection(

this,candidatesForCoproc,baseRequest);

if(override){

//Coprocessoris overriding normal file selection.

compaction.forceSelect(newCompactionRequest(candidatesForCoproc));

}

}

//Normal case - coprocessoris not overriding file selection.

if(!compaction.hasSelection()){

如果是client端发起的compact,此时的值为true,如果是flush时发起的compact，此时的值为false

booleanisUserCompaction= priority== Store.PRIORITY_USER;

offPeakHours的值说明：

1.通过hbase.offpeak.start.hour配置major的启动开始小时，如配置为1

2.通过hbase.offpeak.end.hour配置major的启动结束小时，如配置为2

如果启动时间是1与2配置的小时时间内,