IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing

异常信息如下:

java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
	at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:111)
	at java.io.Reader.read(Reader.java:140)
	at org.wltea.analyzer.core.AnalyzeContext.fillBuffer(AnalyzeContext.java:124)
	at org.wltea.analyzer.core.IKSegmenter.next(IKSegmenter.java:122)
	at org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:78)
	at unit.test.IKAnalyzerTest.test01(IKAnalyzerTest.java:29)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at junit.framework.TestCase.runTest(TestCase.java:168)
	at junit.framework.TestCase.runBare(TestCase.java:134)
	at junit.framework.TestResult$1.protect(TestResult.java:110)
	at junit.framework.TestResult.runProtected(TestResult.java:128)
	at junit.framework.TestResult.run(TestResult.java:113)
	at junit.framework.TestCase.run(TestCase.java:124)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

代码:

public void test01() throws IOException {
<span style="white-space:pre">	</span>String text="基于java语言开发的轻量级的中文分词工具包,一个轻量级框架!"; 
		
        // 创建分词对象  
        Analyzer analyzer = new IKAnalyzer(true);       
        StringReader reader = new StringReader(text);  
        
        // 分词  
        TokenStream ts = analyzer.tokenStream("", reader);  
        CharTermAttribute term = ts.getAttribute(CharTermAttribute.class);  
        
        // 遍历分词数据  
        while(ts.incrementToken()){  
            System.out.print(term.toString()+" | ");  
        }  
        
        reader.close();  
        System.out.println();  
}

上面的代码为旧的分词步骤,按照新的API,调用TokenStream的流程如下:

1、Instantiation of TokenStream/TokenFilters which add/get attributes to/ the AttributeSource.
2、The consumer calls reset().
3、The consumer retrieves attributes the stream and stores local references to all attributes it wants to access.
4、The consumer calls incrementToken() until it returns false consuming the attributes after each call.
5、The consumer calls end() so that any end-of-stream operations can be performed.
6、The consumer calls close() to release any resource when finished using the TokenStream.


所以在调用incrementToken()之前需要调用一次reset(),如下面的第10行代码:

public void test01() throws IOException {
		String text="基于java语言开发的轻量级的中文分词工具包,一个轻量级框架!"; 
		
        // 创建分词对象  
        Analyzer analyzer = new IKAnalyzer(true);       
        StringReader reader = new StringReader(text);  
        
        // 分词  
        TokenStream ts = analyzer.tokenStream("", reader); 
        ts.reset();
        CharTermAttribute term = ts.getAttribute(CharTermAttribute.class);  
        
        // 遍历分词数据  
        while(ts.incrementToken()){  
            System.out.print(term.toString()+" | ");  
        }  
        
        reader.close();  
        System.out.println();  
	}



  • 3
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值