Lucene学习总结之七：Lucene搜索过程解析(4)

最新推荐文章于 2019-11-24 18:56:02 发布

chenhe0410

最新推荐文章于 2019-11-24 18:56:02 发布

阅读量217

点赞数

原文链接：https://my.oschina.net/ruanjun/blog/657726

版权

【转载 http://blog.csdn.net/forfuture1978/article/details/5449943】

2.4、搜索查询对象

2.4.1.2、创建Weight对象树

BooleanQuery.createWeight(Searcher) 最终返回return new BooleanWeight(searcher)，BooleanWeight构造函数的具体实现如下：

public BooleanWeight(Searcher searcher) {

this.similarity = getSimilarity(searcher);

weights = new ArrayList(clauses.size());

//也是一个递归的过程，沿着新的Query对象树一直到叶子节点

for (int i = 0 ; i < clauses.size(); i++) {

weights.add(clauses.get(i).getQuery().createWeight(searcher));

}

对于TermQuery的叶子节点，其TermQuery.createWeight(Searcher) 返回return new TermWeight(searcher)对象，TermWeight构造函数如下：

public TermWeight(Searcher searcher) {

this.similarity = getSimilarity(searcher);

//此处计算了idf

idfExp = similarity.idfExplain(term, searcher);

idf = idfExp.getIdf();

}

//idf的计算完全符合文档中的公式：

public IDFExplanation idfExplain(final Term term, final Searcher searcher) {

final int df = searcher.docFreq(term);

final int max = searcher.maxDoc();

final float idf = idf(df, max);

return new IDFExplanation() {

public float getIdf() {

return idf;

}};

}

public float idf(int docFreq, int numDocs) {

return (float)(Math.log(numDocs/(double)(docFreq+1)) + 1.0);

}

而ConstantScoreQuery.createWeight(Searcher) 除了创建ConstantScoreQuery.ConstantWeight(searcher)对象外，没有计算idf。

由此创建的Weight对象树如下：

              |                          //contents:cat^0.33333325
              |                          this$0    TermQuery (id=151)
              |                          value    0.0
              |                  modCount    2
              |                  size    2
              |------[1]    TermQuery$TermWeight (id=218)
                            idf    2.0986123
                            idfExp    Similarity$1 (id=233)
                            queryNorm    0.0
                            queryWeight    0.0
                            similarity    DefaultSimilarity (id=177)

//contents:foods
this$0    TermQuery (id=154)
                            value    0.0
                    modCount    2
                    size    2
        modCount    3
        size    3

2.4.1.3、计算Term Weight分数

(1) 首先计算sumOfSquaredWeights

按照公式：

代码如下：

float sum = weight.sumOfSquaredWeights();

//可以看出，也是一个递归的过程

public float sumOfSquaredWeights() throws IOException {

float sum = 0.0f;

for (int i = 0 ; i < weights.size(); i++) {

float s = weights.get(i).sumOfSquaredWeights();

if (!clauses.get(i).isProhibited())

sum += s;

}

sum *= getBoost() * getBoost(); //乘以query boost

return sum ;

}

对于叶子节点TermWeight来讲，其TermQuery$TermWeight.sumOfSquaredWeights()实现如下：

public float sumOfSquaredWeights() {

//计算一部分打分，idf*t.getBoost()，将来还会用到。

queryWeight = idf * getBoost();

//计算(idf*t.getBoost())^2

return queryWeight * queryWeight;

}

对于叶子节点ConstantWeight来讲，其ConstantScoreQuery$ConstantWeight.sumOfSquaredWeights() 如下：

public float sumOfSquaredWeights() {

//除了用户指定的boost以外，其他都不计算在打分内

queryWeight = getBoost();

return queryWeight * queryWeight;

}

(2) 计算queryNorm

其公式如下：

其代码如下：

public float queryNorm(float sumOfSquaredWeights) {

return (float)(1.0 / Math.sqrt(sumOfSquaredWeights));

}

(3) 将queryNorm算入打分

代码为：

weight.normalize(norm);

//又是一个递归的过程

public void normalize(float norm) {

norm *= getBoost();

for (Weight w : weights) {

w.normalize(norm);

}

其叶子节点TermWeight来讲，其TermQuery$TermWeight.normalize(float) 代码如下：

public void normalize(float queryNorm) {

this.queryNorm = queryNorm;

//原来queryWeight为idf*t.getBoost()，现在为queryNorm*idf*t.getBoost()。

queryWeight *= queryNorm;

//打分到此计算了queryNorm*idf*t.getBoost()*idf = queryNorm*idf^2*t.getBoost()部分。

value = queryWeight * idf;

}

我们知道，Lucene的打分公式整体如下，到此计算了图中，红色的部分：

2.4.2、创建Scorer及SumScorer对象树

当创建完Weight对象树的时候，调用IndexSearcher.search(Weight, Filter, int)，代码如下：

//(a)创建文档号收集器

TopScoreDocCollector collector = TopScoreDocCollector.create(nDocs, !weight.scoresDocsOutOfOrder());

search(weight, filter, collector);

//(b)返回搜索结果

return collector.topDocs();

public void search(Weight weight, Filter filter, Collector collector)

throws IOException {

if (filter == null) {

for (int i = 0; i < subReaders.length; i++) {

collector.setNextReader(subReaders[i], docStarts[i]);

//(c)创建Scorer对象树，以及SumScorer树用来合并倒排表

Scorer scorer = weight.scorer(subReaders[i], !collector.acceptsDocsOutOfOrder(), true);

if (scorer != null) {

//(d)合并倒排表，(e)收集文档号

scorer.score(collector);

}

} else {

for (int i = 0; i < subReaders.length; i++) {

collector.setNextReader(subReaders[i], docStarts[i]);

searchWithFilter(subReaders[i], weight, filter, collector);

}

在本节中，重点分析(c)创建Scorer对象树，以及SumScorer树用来合并倒排表，在2.4.3节中，分析 (d)合并倒排表，在2.4.4节中，分析文档结果收集器的创建(a)，结果文档的收集(e)，以及文档的返回(b)。

BooleanQuery$BooleanWeight.scorer(IndexReader, boolean, boolean) 代码如下：

public Scorer scorer(IndexReader reader, boolean scoreDocsInOrder, boolean topScorer){

//存放对应于MUST语句的Scorer

List required = new ArrayList();

//存放对应于MUST_NOT语句的Scorer

List prohibited = new ArrayList();

//存放对应于SHOULD语句的Scorer

List optional = new ArrayList();

//遍历每一个子语句，生成子Scorer对象，并加入相应的集合，这是一个递归的过程。

Iterator cIter = clauses.iterator();

for (Weight w : weights) {

BooleanClause c = cIter.next();

Scorer subScorer = w.scorer(reader, true, false);

if (subScorer == null) {

if (c.isRequired()) {

return null;

}

} else if (c.isRequired()) {

required.add(subScorer);

} else if (c.isProhibited()) {

prohibited.add(subScorer);

} else {

optional.add(subScorer);

}

//此处在有关BooleanScorer及scoreDocsInOrder一节会详细描述

if (!scoreDocsInOrder && topScorer && required.size() == 0 && prohibited.size() < 32) {
return new BooleanScorer(similarity, minNrShouldMatch, optional, prohibited);
}

//生成Scorer对象树，同时生成SumScorer对象树

return new BooleanScorer2(similarity, minNrShouldMatch, required, prohibited, optional);

}

对其叶子节点TermWeight来说，TermQuery$TermWeight.scorer(IndexReader, boolean, boolean) 代码如下：

public Scorer scorer(IndexReader reader, boolean scoreDocsInOrder, boolean topScorer) throws IOException {

//此Term的倒排表

TermDocs termDocs = reader.termDocs(term);

if (termDocs == null)

return null;

return new TermScorer(this, termDocs, similarity, reader.norms(term.field()));

}

TermScorer(Weight weight, TermDocs td, Similarity similarity, byte[] norms) {

super(similarity);

this.weight = weight;

this.termDocs = td;

//得到标准化因子

this.norms = norms;

//得到原来计算得的打分：queryNorm*idf^2*t.getBoost()

this.weightValue = weight.getValue();

for (int i = 0; i < SCORE_CACHE_SIZE; i++)

scoreCache[i] = getSimilarity().tf(i) * weightValue;

}

对其叶子节点ConstantWeight来说，ConstantScoreQuery$ConstantWeight.scorer(IndexReader, boolean, boolean) 代码如下：

public ConstantScorer(Similarity similarity, IndexReader reader, Weight w) {

super(similarity);

theScore = w.getValue();

//得到所有的文档号，形成统一的倒排表，参与倒排表合并。

DocIdSet docIdSet = filter.getDocIdSet(reader);

DocIdSetIterator docIdSetIterator = docIdSet.iterator();

}

对于BooleanWeight，最后要产生的是BooleanScorer2，其构造函数代码如下：

public BooleanScorer2(Similarity similarity, int minNrShouldMatch,

List required, List prohibited, List optional) {

super(similarity);

//为了计算打分公式中的coord项做统计

coordinator = new Coordinator();

this.minNrShouldMatch = minNrShouldMatch;

//SHOULD的部分

optionalScorers = optional;

coordinator.maxCoord += optional.size();

//MUST的部分

requiredScorers = required;

coordinator.maxCoord += required.size();

//MUST_NOT的部分

prohibitedScorers = prohibited;

//事先计算好各种情况的coord值

coordinator.init();

//创建SumScorer为倒排表合并做准备

countingSumScorer = makeCountingSumScorer();

}

Coordinator.init() {

coordFactors = new float[maxCoord + 1];

Similarity sim = getSimilarity();

for (int i = 0; i <= maxCoord; i++) {

//计算总的子语句的个数和一个文档满足的子语句的个数之间的关系，自然是一篇文档满足的子语句个个数越多，打分越高。

coordFactors[i] = sim.coord(i, maxCoord);

}

在生成Scorer对象树之外，还会生成SumScorer对象树，来表示各个语句之间的关系，为合并倒排表做准备。

在解析BooleanScorer2.makeCountingSumScorer() 之前，我们先来看不同的语句之间都存在什么样的关系，又将如何影响倒排表合并呢？

语句主要分三类：MUST，SHOULD，MUST_NOT

语句之间的组合主要有以下几种情况：

多个MUST，如"(+apple +boy +dog)"，则会生成ConjunctionScorer(Conjunction 交集)，也即倒排表取交集
MUST和SHOULD，如"(+apple boy)"，则会生成ReqOptSumScorer(required optional)，也即MUST的倒排表返回，如果文档包括SHOULD的部分，则增加打分。
MUST和MUST_NOT，如"(+apple –boy)"，则会生成ReqExclScorer(required exclusive)，也即返回MUST的倒排表，但扣除MUST_NOT的倒排表中的文档。
多个SHOULD，如"(apple boy dog)"，则会生成DisjunctionSumScorer(Disjunction 并集)，也即倒排表去并集
SHOULD和MUST_NOT，如"(apple –boy)"，则SHOULD被认为成MUST，会生成ReqExclScorer
MUST，SHOULD，MUST_NOT同时出现，则MUST首先和MUST_NOT组合成ReqExclScorer，SHOULD单独成为SingleMatchScorer，然后两者组合成ReqOptSumScorer。

下面分析生成SumScorer的过程：

BooleanScorer2.makeCountingSumScorer() 分两种情况：

当有MUST的语句的时候，则调用makeCountingSumScorerSomeReq()
当没有MUST的语句的时候，则调用makeCountingSumScorerNoReq()

首先来看makeCountingSumScorerSomeReq代码如下：

private Scorer makeCountingSumScorerSomeReq() {

if (optionalScorers.size() == minNrShouldMatch) {

//如果optional的语句个数恰好等于最少需满足的optional的个数，则所有的optional都变成required。于是首先所有的optional生成ConjunctionScorer(交集)，然后再通过addProhibitedScorers将prohibited加入，生成ReqExclScorer(required exclusive)

ArrayList allReq = new ArrayList(requiredScorers);

allReq.addAll(optionalScorers);

return addProhibitedScorers(countingConjunctionSumScorer(allReq));

} else {

//首先所有的required的语句生成ConjunctionScorer(交集)

Scorer requiredCountingSumScorer =

requiredScorers.size() == 1

? new SingleMatchScorer(requiredScorers.get(0))

: countingConjunctionSumScorer(requiredScorers);

if (minNrShouldMatch > 0) {

//如果最少需满足的optional的个数有一定的限制，则意味着optional中有一部分要相当于required，会影响倒排表的合并。因而required生成的ConjunctionScorer(交集)和optional生成的DisjunctionSumScorer(并集)共同组合成一个ConjunctionScorer(交集)，然后再加入prohibited，生成ReqExclScorer

return addProhibitedScorers(

dualConjunctionSumScorer(

requiredCountingSumScorer,

countingDisjunctionSumScorer(

optionalScorers,

minNrShouldMatch)));

} else { // minNrShouldMatch == 0

//如果最少需满足的optional的个数没有一定的限制，则optional并不影响倒排表的合并，仅仅在文档包含optional部分的时候增加打分。所以required和prohibited首先生成ReqExclScorer，然后再加入optional，生成ReqOptSumScorer(required optional)

return new ReqOptSumScorer(

addProhibitedScorers(requiredCountingSumScorer),

optionalScorers.size() == 1

? new SingleMatchScorer(optionalScorers.get(0))

: countingDisjunctionSumScorer(optionalScorers, 1));

}

然后我们来看makeCountingSumScorerNoReq代码如下：

private Scorer makeCountingSumScorerNoReq() {

// minNrShouldMatch optional scorers are required, but at least 1

int nrOptRequired = (minNrShouldMatch < 1) ? 1 : minNrShouldMatch;

Scorer requiredCountingSumScorer;

if (optionalScorers.size() > nrOptRequired)

//如果optional的语句个数多于最少需满足的optional的个数，则optional中一部分相当required，影响倒排表的合并，所以生成DisjunctionSumScorer

requiredCountingSumScorer = countingDisjunctionSumScorer(optionalScorers, nrOptRequired);

else if (optionalScorers.size() == 1)

//如果optional的语句只有一个，则返回SingleMatchScorer，不存在倒排表合并的问题。

requiredCountingSumScorer = new SingleMatchScorer(optionalScorers.get(0));

else

//如果optional的语句个数少于等于最少需满足的optional的个数，则所有的optional都算required，所以生成ConjunctionScorer

requiredCountingSumScorer = countingConjunctionSumScorer(optionalScorers);

//将prohibited加入，生成ReqExclScorer

return addProhibitedScorers(requiredCountingSumScorer);

}

经过此步骤，生成的Scorer对象树如下：

                  |                //weight(contents:boy)
                  |                weight    TermQuery$TermWeight (id=201)
                  |                weightValue    2.107161
                  |         size    1
                  |---requiredScorers    ArrayList (id=187)
                          |   elementData    Object[10] (id=193)
                          |---[0]    ConstantScoreQuery$ConstantScorer (id=203)
                                  docIdSetIterator    OpenBitSetIterator (id=206)
                                  similarity    DefaultSimilarity (id=64)
                                  theScore    0.47844642

//ConstantScore(contents:apple*)
this$0    ConstantScoreQuery (id=207)
                        size    1
                similarity    DefaultSimilarity (id=64)
        size    1
    similarity    DefaultSimilarity (id=64)

生成的SumScorer对象树如下：

                                                    |        //weight(contents:boy)
                                                    |        weight    TermQuery$TermWeight (id=201)
                                                    |        weightValue    2.107161
                                                    |---reqScorer    BooleanScorer2$2(ConjunctionScorer) (id=281)
                                                             |     coord    1.0
                                                             |     doc    -1
                                                             |     lastDoc    -1
                                                             |     lastDocScore    NaN
                                                             |     lastScoredDoc    -1
                                                             |---scorers    Scorer[1] (id=283)
                                                                      |---[0]    ConstantScoreQuery$ConstantScorer (id=203)
                                                                                doc    -1
                                                                                doc    -1
                                                                                docIdSetIterator    OpenBitSetIterator (id=206)
                                                                                similarity    DefaultSimilarity (id=64)
                                                                                theScore    0.47844642

//ConstantScore(contents:apple*)
this$0    ConstantScoreQuery (id=207)
                                                                 similarity    DefaultSimilarity (id=64)
                                                                 this$0    BooleanScorer2 (id=82)
                                                                 val$requiredNrMatchers    1
                                                           similarity    null
                                                minNrShouldMatch    0
                                                optionalScorers    ArrayList (id=185)
                                                prohibitedScorers    ArrayList (id=186)
                                                requiredScorers    ArrayList (id=187)
                                                similarity    DefaultSimilarity (id=64)
                                     similarity    DefaultSimilarity (id=64)
                                     this$0    BooleanScorer2 (id=50)
                          similarity    null
                 similarity    null
       minNrShouldMatch    0
       optionalScorers    ArrayList (id=55)
       prohibitedScorers    ArrayList (id=60)
       requiredScorers    ArrayList (id=63)
       similarity    DefaultSimilarity (id=64)

转载于:https://my.oschina.net/ruanjun/blog/657726

chenhe0410

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Lucene学习总结之七：Lucene搜索过程解析(4)

【转载 http://blog.csdn.net/forfuture1978/article/details/5449943】 2.4、搜索查询对象 2.4.1.2、创建Weight对象树 BooleanQuery.createWeight(Searcher) 最终返回return ne...
复制链接

扫一扫