也公布一段精简的分词代码

 公司要在产品中增加一个知识库检索功能，需要用到从一段文字中分析出关键字来，即将一句话中的中文文字分成中文词语或词组，忽略半角符号。从网上Google一下，现成的分词代码有好几个版本，甚至都能直接拿来使用，但我发现占用资料、分词速度还是不尽如人意，后来试着直接使用泛型的List，发现代码即精简，效率也不错。

    /// <summary>
    /// 从一句话中识别出关键字，供查询知识库。
    /// 目前的关键字库为从网上下载，并没有按知识库优化，建议最终的关键字是按知识库优化之后的
    /// </summary>
   public  class CFindKeyWord
    {
       private static List<string> keyWords = null;

       /// <summary>
       /// 从一句话中识别出关键字，供查询知识库。
       /// </summary>
       /// <param name="strText"></param>
       /// <returns></returns>
       public static string MyFindKeyWord(string strText)
       {
           if (keyWords == null)
           {
               keyWords = new List<string>();
               //string path = System.Environment.CurrentDirectory + "//KeyWords.txt";
               //StreamReader Reader = new StreamReader(path, Encoding.Default);
               StringReader Reader = new StringReader(MSPConsole.Properties.Resources.KeyWords);

               while (Reader.Peek() != -1)
               {
                   keyWords.Add(Reader.ReadLine());

               }
               Reader.Close();
               Reader.Dispose();

               keyWords.Sort();
           }
           StringBuilder strBuilder = new StringBuilder();
           int iCount = 0;
           int banJiaoCount = 0;
           while (iCount < strText.Length)
           {
               if (strText[iCount] < (char)0x7f)
               {
                   strBuilder.Append(strText[iCount]);
                   banJiaoCount++;
                   iCount++;
                   continue;
               }
               else if (banJiaoCount > 0)
               {
                   strBuilder.Append(" ");
                   banJiaoCount = 0;
               }
               int len = strText.Length - iCount;
               while (len > 0)
               {
                   string tmpS = strText.Substring(iCount, len);
                   int index = keyWords.BinarySearch(tmpS);
                   if (index > -1)
                   {
                       strBuilder.Append(tmpS);
                       strBuilder.Append(" ");
                       //iCount += index;
                       break;
                   }
                   len--;
               }
               iCount++;
           }

           return strBuilder.ToString().TrimEnd();
       }
    }

 

 

注：程序中的关键字词库，网上到处有下载，这贴这段代码的目的，不是为了介绍分词的方便，只是觉得C#的功能真的很强大。