n gram java_Java Lucene NGramTokenizer

我不认为你会发现你正在寻找寻找方法返回String.你需要处理

Attributes.

应该工作的东西像：

Reader reader = new StringReader("This is a test string");

NGramTokenizer gramTokenizer = new NGramTokenizer(reader,3);

CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);

gramTokenizer.reset();

while (gramTokenizer.incrementToken()) {

String token = charTermAttribute.toString();

//Do something

}

gramTokenizer.end();

gramTokenizer.close();

一定要重置()令牌,如果它需要重新使用后,虽然.

每个意见令牌组合,而不是字符：

Reader reader = new StringReader("This is a test string");

TokenStream tokenizer = new StandardTokenizer(Version.LUCENE_36,reader);

tokenizer = new ShingleFilter(tokenizer,3);

CharTermAttribute charTermAttribute = tokenizer.addAttribute(CharTermAttribute.class);

while (tokenizer.incrementToken()) {

String token = charTermAttribute.toString();

//Do something

}