java 中定义solr字段,java-自定义Solr TokenFilter lemmatizer

我正在尝试编写用于字段类型的简单Solr lemmatizer,但是我似乎找不到有关编写TokenFilter的任何信息,因此我有点迷失了.这是我到目前为止的代码.

import java.io.IOException;

import java.util.List;

import org.apache.lucene.analysis.TokenFilter;

import org.apache.lucene.analysis.TokenStream;

import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

class FooFilter extends TokenFilter {

private static final Logger log = LoggerFactory.getLogger(FooFilter.class);

private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

private final PositionIncrementAttribute posAtt = addAttribute(PositionIncrementAttribute.class);

public FooFilter(TokenStream input) {

super(input);

}

@Override

public boolean incrementToken() throws IOException {

if (!input.incrementToken()) {

return false;

}

char termBuffer[] = termAtt.buffer();

List allForms = Lemmatize.getAllForms(new String(termBuffer));

if (allForms.size() > 0) {

for (String word : allForms) {

// Now what?

}

}

return true;

}

}

最佳答案

接下来,您要用单词替换或附加当前令牌termAtt.

样本替换语义

termAtt.setEmpty();

termAtt.copyBuffer(word.toCharArray(), 0, word.length());

添加新令牌的示例语义

对于要添加的每个令牌,必须设置CharTermAttribute属性,并设置递增令牌例程以返回true.

private List extraTokens = ...

public boolean incrementToken() {

if (input.incrementToken()){

// ...

return true;

} else if (!extraTokens.isEmtpy()) {

// set the added token and return true

termAtt.setTerm(extraTokens.remove(0));

return true;

}

return false;

}