UserScan的处理流程分析

UserScan的处理流程分析

前置说明

Userscan是通过client或cp中发起的scanner操作。

在Scan中通过caching属性来返回可以返回多少条数据，每次进行next时。

通过batch属性来设置每次在rs端每次nextkv时，可读取多少个kv，(在同一行的情况下)

在生成Scan实例时，最好是把family与column都设置上，这样能保证查询的最高效.

client端通过生成Scan实例，通过HTable下的如下方法得到ClientScanner实例

publicResultScannergetScanner(finalScan scan)

在生成的ClientScanner实例中的callable属性的值为生成的一个ScannerCallable实例。

并通过callable.prepare(tries!= 0);方法得到此scan的startkey所在的region的location.在meta表中。

把startkey对应的location中得到此location的HRegionInfo信息。

并设置ClientScanner.currentRegion的值为当前的region.也就是startkey所在的region.

通过ClientScanner.next向rs发起rpc调用操作。调用HRegionServer.scan

publicScanResponse scan(finalRpcControllercontroller,finalScanRequest request)

ClientScanner.next时，首先是发起openScanner操作，得到一个ScannerId

通过ScannerCallable.call方法：

if(scannerId== -1L) {

this.scannerId= openScanner();

} else{

openScanner方法:中发起一个scan操作，通过rpc调用rs.scan

ScanRequest request=

RequestConverter.buildScanRequest(

getLocation().getRegionInfo().getRegionName(),

this.scan,0, false);

try{

ScanResponse response= getStub().scan(null,request);

longid =response.getScannerId();

if(logScannerActivity){

LOG.info("Openscanner=" + id+ " for scan="+ scan.toString()

+ "on region " +getLocation().toString());

}

returnid;

HregionServer.scan中对openScanner的处理：

publicScanResponse scan(finalRpcControllercontroller,finalScanRequest request)

throwsServiceException {

Leases.Lease lease= null;

String scannerName= null;

........................................很多代码没有显示

requestCount.increment();

intttl = 0;

HRegion region= null;

RegionScannerscanner =null;

RegionScannerHolder rsh= null;

booleanmoreResults= true;

booleancloseScanner= false;

ScanResponse.Builder builder= ScanResponse.newBuilder();

if(request.hasCloseScanner()){

closeScanner= request.getCloseScanner();

}

introws = 1;

if(request.hasNumberOfRows()){

rows= request.getNumberOfRows();

}

if(request.hasScannerId()){

.................................很多代码没有显示

} else{

得到请求的HRegion实例,也就是startkey所在的HRegion

region= getRegion(request.getRegion());

ClientProtos.Scan protoScan= request.getScan();

booleanisLoadingCfsOnDemandSet= protoScan.hasLoadColumnFamiliesOnDemand();

Scan scan= ProtobufUtil.toScan(protoScan);

//if the request doesn't set this, get the default region setting.

if(!isLoadingCfsOnDemandSet){

scan.setLoadColumnFamiliesOnDemand(region.isLoadingCfsOnDemandDefault());

}

scan.getAttribute(Scan.SCAN_ATTRIBUTES_METRICS_ENABLE);

如果scan没有设置family,把region中所有的family当成scan的family

region.prepareScanner(scan);

if(region.getCoprocessorHost()!= null){

scanner= region.getCoprocessorHost().preScannerOpen(scan);

}

if(scanner ==null){

执行HRegion.getScanner方法。生成HRegion.RegionScannerImpl方法

scanner= region.getScanner(scan);

}

if(region.getCoprocessorHost()!= null){

scanner= region.getCoprocessorHost().postScannerOpen(scan,scanner);

}

把生成的RegionScanner添加到scanners集合容器中。并设置scannerid(一个随机的值),

scannername是scannerid的string版本。添加过期监控处理，

通过hbase.client.scanner.timeout.period配置过期时间，默认值为60000ms

老版本通过hbase.regionserver.lease.period配置。

过期检查线程通过Leases完成。对scanner的过期处理通过一个

HregionServer.ScannerListener.leaseExpired实例来完成。

scannerId= addScanner(scanner,region);

scannerName= String.valueOf(scannerId);

ttl= this.scannerLeaseTimeoutPeriod;

}

............................................很多代码没有显示

Hregion.getScanner方法生成RegionScanner实例流程

publicRegionScannergetScanner(Scanscan)throwsIOException {

returngetScanner(scan,null);

}

层次的调用,此时传入的kvscannerlist为null

protectedRegionScannergetScanner(Scanscan,

List<KeyValueScanner>additionalScanners)throwsIOException {

startRegionOperation(Operation.SCAN);

try{

//Verify families are all valid

prepareScanner(scan);

if(scan.hasFamilies()){

for(byte[] family :scan.getFamilyMap().keySet()){

checkFamily(family);

}

}

returninstantiateRegionScanner(scan,additionalScanners);

}finally{

closeRegionOperation();

}

}

最终生成一个HRegion.RegionScannerImpl实例

protectedRegionScannerinstantiateRegionScanner(Scanscan,

List<KeyValueScanner>additionalScanners)throwsIOException {

returnnewRegionScannerImpl(scan,additionalScanners,this);

}

RegionScanner实例的生成构造方法：

RegionScannerImpl(Scanscan,List<KeyValueScanner>additionalScanners,HRegion region)

throwsIOException {

this.region= region;

this.maxResultSize= scan.getMaxResultSize();

if(scan.hasFilter()){

this.filter= newFilterWrapper(scan.getFilter());

} else{

this.filter= null;

}

this.batch= scan.getBatch();

if(Bytes.equals(scan.getStopRow(),HConstants.EMPTY_END_ROW)&& !scan.isGetScan()){

this.stopRow= null;

} else{

this.stopRow= scan.getStopRow();

}

//If we are doing a get, we want to be [startRow,endRow] normally

//it is [startRow,endRow) and if startRow=endRow we get nothing.

this.isScan= scan.isGetScan()? -1 : 0;

//synchronize on scannerReadPoints so that nobody calculates

//getSmallestReadPoint, before scannerReadPoints is updated.

IsolationLevelisolationLevel= scan.getIsolationLevel();

synchronized(scannerReadPoints){

if(isolationLevel== IsolationLevel.READ_UNCOMMITTED){

//This scan can read even uncommitted transactions

this.readPt= Long.MAX_VALUE;

MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);

} else{

this.readPt= MultiVersionConsistencyControl.resetThreadReadPoint(mvcc);

}

scannerReadPoints.put(this,this.readPt);

}

//Here we separate all scanners into two lists - scanner that providedata required

//by the filter to operate (scanners list) and all others(joinedScanners list).

List<KeyValueScanner>scanners =newArrayList<KeyValueScanner>();

List<KeyValueScanner>joinedScanners= newArrayList<KeyValueScanner>();

if(additionalScanners!= null){

scanners.addAll(additionalScanners);

}

迭代每一个要进行scan的store。生成具体的StoreScanner实例。通常情况下joinedHead的值为null

for(Map.Entry<byte[],NavigableSet<byte[]>>entry :

scan.getFamilyMap().entrySet()){

Storestore =stores.get(entry.getKey());

生成StoreScanner实例。通过HStore.getScanner(scan,columns);

KeyValueScannerscanner =