lucene源码分析---7

lucene源码分析—QueryParser的parse函数

本章主要分析QueryParser类的parse函数，定义在其父类QueryParserBase中，
QueryParserBase::parse

  public Query parse(String query) throws ParseException {
    ReInit(new FastCharStream(new StringReader(query)));
    try {
      Query res = TopLevelQuery(field);
      return res!=null ? res : newBooleanQuery().build();
    } catch (ParseException | TokenMgrError tme) {

    } catch (BooleanQuery.TooManyClauses tmc) {

    }
  }

parse首先将需要搜索的字符串query封装成FastCharStream，FastCharStream实现了Java的CharStream接口，内部使用了一个缓存，并且可以方便读取并且改变读写指针。然后调用ReInit进行初始化，ReInit以及整个QueryParser都是由JavaCC根据org.apache.lucene.queryparse.classic.QueryParser.jj文件自动生成，设计到的JavaCC的知识可以从网上或者别的书上查找，本博文不会重点分析这块内容。
parse最重要的函数是TopLevelQuery，即返回顶层Query，TopLevelQuery会根据用来搜索的字符串query创建一个树形的Query结构，传入的参数field在QueryParserBase的构造函数中赋值，用来标识对哪个域进行搜索。
QueryParserBase::parse->QueryParser::TopLevelQuery

  final public Query TopLevelQuery(String field) throws ParseException {
    Query q;
    q = Query(field);
    jj_consume_token(0);
    {
      if (true) return q;
    }
    throw new Error();
  }

TopLevelQuery函数中最关键的是Query函数，由于QueryParser由JavaCC生成，这里只看QueryParser.jj文件。

QueryParser.jj::Query

Query Query(String field) :
{
  List<BooleanClause> clauses = new ArrayList<BooleanClause>();
  Query q, firstQuery=null;
  int conj, mods;
}
{
  mods=Modifiers() q=Clause(field)
  {
    addClause(clauses, CONJ_NONE, mods, q);
    if (mods == MOD_NONE)
        firstQuery=q;
  }
  (
    conj=Conjunction() mods=Modifiers() q=Clause(field)
    { addClause(clauses, conj, mods, q); }
  )*
    {
      if (clauses.size() == 1 && firstQuery != null)
        return firstQuery;
      else {
  return getBooleanQuery(clauses);
      }
    }
}

Modifiers返回搜索字符串中的”+”或”-“，Conjunction返回连接字符串。Query首先通过Clause函数返回一个子查询，然后调用addClause函数添加该子查询，

QueryParserBase::addClause

  protected void addClause(List<BooleanClause> clauses, int conj, int mods, Query q) {
    boolean required, prohibited;

    ...

    if (required && !prohibited)
      clauses.add(newBooleanClause(q, BooleanClause.Occur.MUST));
    else if (!required && !prohibited)
      clauses.add(newBooleanClause(q, BooleanClause.Occur.SHOULD));
    else if (!required && prohibited)
      clauses.add(newBooleanClause(q, BooleanClause.Occur.MUST_NOT));
    else
      throw new RuntimeException("Clause cannot be both required and prohibited");
  }

addClause函数中省略的部分是根据参数连接符conj和mods计算required和prohibited的值，然后将Query封装成BooleanClause并添加到clauses列表中。

回到Query函数中，如果子查询clauses列表只有一个子查询，就直接返回，否则通过getBooleanQuery函数封装所有的子查询并最终返回一个BooleanClause。

下面来看Clause函数，即创建一个子查询，
QueryParser.jj::Clause

Query Clause(String field) : {
  Query q;
  Token fieldToken=null, boost=null;
}
{
  [
    LOOKAHEAD(2)
    (
    fieldToken=<TERM> <COLON> {field=discardEscapeChar(fieldToken.image);}
    | <STAR> <COLON> {field="*";}
    )
  ]

  (
   q=Term(field)
   | <LPAREN> q=Query(field) <RPAREN> (<CARAT> boost=<NUMBER>)?

  )
    {  return handleBoost(q, boost); }
}

LOOKAHEAD(2)表示要看两个符号，如果是Field，则要重新调整搜索的域。Clause函数最重要的是Term函数，该函数返回最终的Query，当然Clause函数也可以嵌套调用Query函数生成子查询。

QueryParser.jj::Term

Query Term(String field) : {
  Token term, boost=null, fuzzySlop=null, goop1, goop2;
  boolean prefix = false;
  boolean wildcard = false;
  boolean fuzzy = false;
  boolean regexp = false;
  boolean startInc=false;
  boolean endInc=false;
  Query q;
}
{
  (
     (
       term=<TERM>
       | term=<STAR> { wildcard=true; }
       | term=<PREFIXTERM> { prefix=true; }
       | term=<WILDTERM> { wildcard=true; }
       | term=<REGEXPTERM> { regexp=true; }
       | term=<NUMBER>
       | term=<BAREOPER> { term.image = term.image.substring(0,1); }
     )
     [ fuzzySlop=<FUZZY_SLOP> { fuzzy=true; } ]
     [ <CARAT> boost=<NUMBER> [ fuzzySlop=<FUZZY_SLOP> { fuzzy=true; } ] ]
     {
       q = handleBareTokenQuery(field, term, fuzzySlop, prefix, wildcard, fuzzy, regexp);
     }
     | ( ( <RANGEIN_START> {startInc=true;} | <RANGEEX_START> )
         ( goop1=<RANGE_GOOP>|goop1=<RANGE_QUOTED> )
         [ <RANGE_TO> ]
         ( goop2=<RANGE_GOOP>|goop2=<RANGE_QUOTED> )
         ( <RANGEIN_END> {endInc=true;} | <RANGEEX_END>))
       [ <CARAT> boost=<NUMBER> ]
        {
          boolean startOpen=false;
          boolean endOpen=false;
          if (goop1.kind == RANGE_QUOTED) {
            goop1.image = goop1.image.substring(1, goop1.image.length()-1);
          } else if ("*".equals(goop1.image)) {
            startOpen=true;
          }
          if (goop2.kind == RANGE_QUOTED) {
            goop2.image = goop2.image.substring(1, goop2.image.length()-1);
          } else if ("*".equals(goop2.image)) {
            endOpen=true;
          }
          q = getRangeQuery(field, startOpen ? null : discardEscapeChar(goop1.image), endOpen ? null : discardEscapeChar(goop2.image), startInc, endInc);
        }
     | term=<QUOTED>
       [ fuzzySlop=<FUZZY_SLOP> ]
       [ <CARAT> boost=<NUMBER> ]
       { q = handleQuotedTerm(field, term, fuzzySlop); }
  )
  { return handleBoost(q, boost); }
}

如果一个查询不包括引号(QUOTED)，边界符号(RANGE，例如小括号、中括号等)，大部分情况下最终会通过handleBareTokenQuery函数生成一个Term，代表一个词，然后被封装成一个子查询Clause，最后被封装成一个Query，Clause和Query互相嵌套，即一个Query里可以包含多个Clause，一个Clause里又可以从一个Query开始，最终的叶子节点就是Term对应的Query。

QueryParserBase::handleBareTokenQuery

  Query handleBareTokenQuery(String qfield, Token term, Token fuzzySlop, boolean prefix, boolean wildcard, boolean fuzzy, boolean regexp) throws ParseException {
    Query q;

    String termImage=discardEscapeChar(term.image);
    if (wildcard) {
      q = getWildcardQuery(qfield, term.image);
    } else if (prefix) {
      q = getPrefixQuery(qfield,
          discardEscapeChar(term.image.substring
              (0, term.image.length()-1)));
    } else if (regexp) {
      q = getRegexpQuery(qfield, term.image.substring(1, term.image.length()-1));
    } else if (fuzzy) {
      q = handleBareFuzzy(qfield, fuzzySlop, termImage);
    } else {
      q = getFieldQuery(qfield, termImage, false);
    }
    return q;
  }

举例来说，查询字符串AAA*代表prefix查询，此时参数prefix为真，A*A代表wildcard查询，此时参数wildcard为真，AA~代表fuzzy模糊查询，此时参数fuzzy为真。这里假设三个都不为真，就是一串平常的单词，最后会通过getFieldQuery生成一个Query，本文重点分析该函数。

QueryParserBase::handleBareTokenQuery->getFieldQuery

  protected Query getFieldQuery(String field, String queryText, boolean quoted) throws ParseException {
    return newFieldQuery(getAnalyzer(), field, queryText, quoted);
  }

  protected Query newFieldQuery(Analyzer analyzer, String field, String queryText, boolean quoted)  throws ParseException {
    BooleanClause.Occur occur = operator == Operator.AND ? BooleanClause.Occur.MUST : BooleanClause.Occur.SHOULD;
    return createFieldQuery(analyzer, occur, field, queryText, quoted || autoGeneratePhraseQueries, phraseSlop);
  }

getAnalyzer返回QueryParserBase的init函数中设置的分词器，这里为了方便分析，假设为SimpleAnalyzer。quoted以及autoGeneratePhraseQueries表示是否创建PhraseQuery，phraseSlop为位置因子，只有PhraseQuery用得到，这里不管它。下面来看createFieldQuery函数。
QueryParserBase::handleBareTokenQuery->getFieldQuery->newFieldQuery->QueryBuilder::createFieldQuery

  protected final Query createFieldQuery(Analyzer analyzer, BooleanClause.Occur operator, String field, String queryText, boolean quoted, int phraseSlop) {

    try (TokenStream source = analyzer.tokenStream(field, queryText);
         CachingTokenFilter stream = new CachingTokenFilter(source)) {

      TermToBytesRefAttribute termAtt = stream.getAttribute(TermToBytesRefAttribute.class);
      PositionIncrementAttribute posIncAtt = stream.addAttribute(PositionIncrementAttribute.class);

      int numTokens = 0;
      int positionCount = 0;
      boolean hasSynonyms = false;

      stream.reset();
      while (stream.incrementToken()) {
        numTokens++;
        int positionIncrement = posIncAtt.getPositionIncrement();
        if (positionIncrement != 0) {
          positionCount += positionIncrement;
        } else {
          hasSynonyms = true;
        }
      }

      if (numTokens == 0) {
        return null;
      } else if (numTokens == 1) {
        return analyzeTerm(field, stream);
      } else if (quoted && positionCount > 1) {
        ...
      } else {
        if (positionCount == 1) {
          return analyzeBoolean(field, stream);
        } else {
          return analyzeMultiBoolean(field, stream, operator);
        }
      }
    } catch (IOException e) {

    }
  }

关于分词器的tokenStream以及incrementToken函数在《lucene源码分析—4》中分析过了。直接看最后的结果，假设numTokens==1，则分词器的输出结果只有一个词，则使用analyzeTerm创建最终的Query；
假设positionCount == 1，则表示结果中多个词出现在同一个位置，此时使用analyzeBoolean创建Query；剩下情况表示有多个词，至少两个词出现在不同位置，使用analyzeMultiBoolean创建Query。本文只分析analyzeMultiBoolean函数，
QueryParserBase::handleBareTokenQuery->getFieldQuery->newFieldQuery->QueryBuilder::createFieldQuery->analyzeMultiBoolean

  private Query analyzeMultiBoolean(String field, TokenStream stream, BooleanClause.Occur operator) throws IOException {
    BooleanQuery.Builder q = newBooleanQuery();
    List<Term> currentQuery = new ArrayList<>();

    TermToBytesRefAttribute termAtt = stream.getAttribute(TermToBytesRefAttribute.class);
    PositionIncrementAttribute posIncrAtt = stream.getAttribute(PositionIncrementAttribute.class);

    stream.reset();
    while (stream.incrementToken()) {
      if (posIncrAtt.getPositionIncrement() != 0) {
        add(q, currentQuery, operator);
        currentQuery.clear();
      }
      currentQuery.add(new Term(field, termAtt.getBytesRef()));
    }
    add(q, currentQuery, operator);

    return q.build();
  }

  private void add(BooleanQuery.Builder q, List<Term> current, BooleanClause.Occur operator) {
    if (current.isEmpty()) {
      return;
    }
    if (current.size() == 1) {
      q.add(newTermQuery(current.get(0)), operator);
    } else {
      q.add(newSynonymQuery(current.toArray(new Term[current.size()])), operator);
    }
  }

  public Builder add(Query query, Occur occur) {
    clauses.add(new BooleanClause(query, occur));
    return this;
  }

分词器的输出结果保存在TermToBytesRefAttribute中，analyzeMultiBoolean函数将同一个起始位置不同的Term添加到列表currentQuery中，如果同一个位置只有一个Term，则将其封装成TermQuery，如果有多个Term，就封装成SynonymQuery，TermQuery和SynonymQuery最后被封装成BooleanClause，添加到BooleanQuery.Builder中的一个BooleanClause列表中。最后通过BooleanQuery.Builder的build函数根据内置的BooleanClause列表创建一个最终的BooleanClause。