Lucene之庖丁分词器及性能分析

首先简单介绍一下中文分词器，lucene默认的中文分词器有：单字分词StandardAnalyzer 、 二分法分词 CJKAnalyzer。另外就是外部的词典分词了，最简单的是极易分词MMAnalyzer、庖丁分词PaodingAnalyzer。

单字分词就是把一句中文一个字一个字的分开，二分法分词就是相邻的二个字是一个关键词，基本上这两种分词方法用的很少了，用法也很简单。

MMAnalyzer分词基本也不用什么设置，直接导入jar包然后 new MMAnalyzer();就行了。

下面具体介绍庖丁分词。首先下载jar包： http://code.google.com/p/paoding/downloads/list

得到paoding-analysis-2.0.4-beta.zip，然后解压开把paoding-analysis.jar加到classpath路径下，建立如下测试文件：

String enStr = "What are you believe.txt";
 String chStr = "我们是中国人！";
 
 Analyzer en1 = new StandardAnalyzer();
 Analyzer en2 = new SimpleAnalyzer();
 
 Analyzer ch1 = new CJKAnalyzer();
 Analyzer ch2 = new MMAnalyzer();
 Analyzer ch3 = new PaodingAnalyzer();
 
 
 @Test
 public void test() throws Exception{
//  analyzer(en1,enStr);
//  analyzer(en2,enStr);
//  analyzer(en1, chStr);
//  analyzer(ch1, chStr);
  Long time1 = System.currentTimeMillis();
  analyzer(ch2, chStr);
  Long time2 = System.currentTimeMillis();
  System.out.println("MMAnalyzer用时： "+(time2 - time1));
  analyzer(ch3, chStr);
  Long time3 = System.currentTimeMillis();
  System.out.println("PaodingAnalyzer用时： "+(time3 - time2));
 }

 private void analyzer(Analyzer analyzer, String text) throws IOException {
  System.out.println("分词器---> " + analyzer);
  TokenStream tokenStream = analyzer.tokenStream("content", new StringReader(text));
  for (Token token = new Token(); (token = tokenStream.next(token)) != null;) {
   System.out.println(token);
  }
 }

报错：

net.paoding.analysis.exception.PaodingAnalysisException: please set a system env PAODING_DIC_HOME or Config paoding.dic.home in paoding-dic-home.properties point to the dictionaries!
 at net.paoding.analysis.knife.PaodingMaker.setDicHomeProperties(PaodingMaker.java:320)
 at net.paoding.analysis.knife.PaodingMaker.getDicHome(PaodingMaker.java:261)
 at net.paoding.analysis.knife.PaodingMaker.loadProperties(PaodingMaker.java:189)
 at net.paoding.analysis.knife.PaodingMaker.loadProperties(PaodingMaker.java:228)
 at net.paoding.analysis.knife.PaodingMaker.loadProperties(PaodingMaker.java:228)
 at net.paoding.analysis.knife.PaodingMaker.getProperties(PaodingMaker.java:130)
 at net.paoding.analysis.analyzer.PaodingAnalyzer.init(PaodingAnalyzer.java:70)
 at net.paoding.analysis.analyzer.PaodingAnalyzer.<init>(PaodingAnalyzer.java:59)
 at net.paoding.analysis.analyzer.PaodingAnalyzer.<init>(PaodingAnalyzer.java:52)
 at com.itcast.lucene.AnalyzerTest.<init>(AnalyzerTest.java:27)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:202)
 at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:251)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:248)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38)
 at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)

大概意思是说没有索引字典 ，建立系统环境变量  PAODING_DIC_HOME  或者 在paoding-dic-home.properties  中配置。

于是建立目录：D:\java\paoding\dic，此处最好为英文目录。否则可能乱码导致找不到。 打开jar包，找到  paoding-dic-home.properties 把 paoding.dic.home，去掉注释，修改为你的dic路径即可： paoding.dic.home=D:\\java\\paoding\\dic （注意要用转义的双反斜杠）。

再运行，报错：

org.apache.jasper.JasperException: not found the dic home dirctory! D:\java\paoding\dic
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:346)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:810)

没有找到字典从 D:\java\paoding\dic目录中。

解决方案：把zip解压包中的dic文件夹中的内容拷贝到上述文件目录。运行一切正常。

下面是性能分析：输出结果：

分词器---> jeasy.analysis.MMAnalyzer@1372a1a
(我们,0,2)
(中国人,3,6)
MMAnalyzer用时： 929
分词器---> net.paoding.analysis.analyzer.PaodingAnalyzer@15ff48b
(我们,0,2)
(中国,3,5)
(国人,4,6)
PaodingAnalyzer用时： 7

MMAnalyzer@1372a1a  用900多毫秒，  PaodingAnalyzer  用7秒，性能高低显而易见。